People’s trust in AI is taking a dangerous new turn. Researchers say AI-generated photos and summaries are successful in distorting people’s memories.
MIT scientist Pattie Maes spoke to the Washington Post about her team’s research. Participants viewed a series of photographs and, after a pause, some participants saw the exact same photographs while others saw versions modified by the AI. When later presented with a memory test, people who viewed the AI-altered images were not only more likely to misremember the original image, but they were also overconfident in their false memories.
Human memory is unstable, unreliable and very open to suggestion. This topic has been widely addressed in psychological experiments dating back to the 19th century. This is what makes misinformation particularly dangerous, with some studies showing that it can continue to influence decision-making, including policy decisions, even after corrections are issued.
The introduction of artificial intelligence only makes this dynamic more precarious. AI-generated slops are becoming increasingly convincing and widespread online, and they have enabled and fueled a new wave of fraud and misinformation.
Numerous studies, besides Maes’s, have shown that these AI-generated pieces of misinformation successfully implanted false memories into participants’ brains, thereby changing what participants know to be true. A study last year found that deepfakes and AI-generated texts effectively plant false memories of completely made-up political scandals, going so far as to reduce voting intentions.
Even when AI is not used for malicious purposes, it can have a potentially catastrophic influence on the truth. This is because generative AI tends to make errors in its results, called hallucinations. As government agencies, banks, the military, hospitals, and an ever-growing list of organizations race to deploy AI, you’ll likely hear about the importance of “humans in the loop”—that is, a human, often a senior executive with subject matter expertise, who audits the results of the AI system. The idea is that widespread deployment of AI under human oversight will speed up everything from military strikes to record-keeping, while ensuring that any AI-generated errors will be detected before they can translate into action and cause real harm.
But research shows that users aren’t particularly good at spotting these hallucinations. And with our spotty memories and tendency to believe AI systems, even when something an AI system posts contradicts information that the human expert in the know knows to be true, the expert is vulnerable to distortion of their judgment.
There may be ways to resolve this issue. Currently, the average person’s ability to correctly identify AI-generated content is essentially a coin toss, studies show, but many researchers are working to find interventions and training methods to help humans better spot deepfakes.
But there is still no foolproof method to remedy this erroneous trend. As experts struggle to find one, the AI industry is racing to sideline humans with agentic AI systems that they hope to make completely autonomous.
This gap has already led to catastrophic consequences. A US military strike on an elementary school in Iran killed more than 150 people, including at least 123 children, earlier this year. An internal Defense Department investigation, reported by Bloomberg last month, concluded that the strike was partly due to an over-reliance on an AI system developed by Palantir, which recommended the elementary school as a strike target from day one due to outdated data.
Gn tech
