SSocial engineering traditionally relies on deceiving others. Conversational AI changes that by making personalized persuasion possible on an unprecedented scale. Fortunately, the familiar tools of skeptical thinking can help us recognize and resist AI-induced manipulation.
When was the last time an AI changed its mind?
Maybe you don’t remember when an AI changed your mind, or maybe you weren’t aware of such an influence. Maybe he recommended a different destination for your next vacation, changed your mind about a product you were going to buy, helped you settle an argument with a friend, or even dissuaded you from clicking on that suspicious link in an unsolicited email. None of this is sinister. Persuasion is a part of ordinary communication, and there’s usually nothing wrong with changing your mind in response to a good argument. But not everyone who tries to influence us is fair. As psychologist Robert Cialdini has demonstrated, influence can become manipulative when it exploits our vulnerabilities or undermines our ability to make an informed choice.1 Decades of research have shown how psychological mechanisms make us susceptible to manipulation. And we are often easier to manipulate than we like to think.
Today, many people spend hours talking to AI chatbots that are extraordinarily gifted in language and increasingly capable of persuasion. Research has shown that AI-generated messages can influence people’s opinions and that tailoring arguments to personal characteristics can sometimes make them more persuasive.2, 3, 4 Such influence can also be beneficial: researchers have even used AI-powered conversations to help people reconsider their conspiracy beliefs.5 But what happens when a system starts to guide your decisions in a way that isn’t in your best interest?
The risk is particularly great for conversational AI, which can already remember previous conversations and adapt to user responses. Some systems can also interact through human avatars in real time, potentially making the experience more personal. With more sophisticated capabilities and access to large amounts of data, AI has the potential to become a powerful manipulator. This article aims to prepare you for the social engineering tactics of the AI era.
Be prepared for large-scale scams
Of course, social engineering is nothing new. Scammers, cult leaders, propagandists, self-proclaimed psychics and other con artists have always been creative when it comes to manipulating their victims. Back then, a fortune teller could enter a new village, read the signs on shops to learn the names of the owners, and eavesdrop on conversations around the market to discreetly gather information before approaching a gullible target. The victim could then be stunned by the apparent knowledge of the fortune teller, even though they had never met. Skeptics call it hot reading: collecting personal information in advance and passing it off as supernatural insight.
Decades of research have shown how psychological mechanisms make us susceptible to manipulation. We are often easier to manipulate than we like to think.
Modern scammers have the Internet and can learn a lot about their targets simply from what people willingly share on social media. They can also commit scams without ever meeting their victims face to face. Mass phishing, robocalls, and pattern-based fraud were scalable long before generative AI. However, convincing and personalized attacks still require considerable effort to search for a target and tailor the approach. Conventional scraping tools and robots can automate parts of this process, but they don’t necessarily remove the human operator. AI systems, on the other hand, can interact with very large numbers of people simultaneously. And unlike the fortune teller, they can carry on conversations around the clock. But scale alone isn’t what sets AI apart. Personalization and adaptation can be just as important. A conversational system can respond to what you say and potentially adjust its tactics when an argument fails, looking for an approach that works. These techniques are familiar in human social engineering; what AI adds is the ability to automate them.6
Before going any further, it is worth distinguishing manipulation from other AI failures. We regularly see stories about AI consulting gone wrong. Falsely identified poisonous mushrooms7 in cases of what is called “AI psychosis”8— interactions with chatbots have sometimes had serious consequences. Such systems can also exhibit sycophancy, excessively agreeing with users even when they are wrong, and potentially reinforcing false beliefs.9, 10 However, a harmful result alone does not prove deliberate manipulation: hallucinations, poor protections or excessive confidence on the part of the human can also be responsible.
At the same time, we are already seeing a lot of AI-assisted social engineering. For example, scammers can clone a loved one’s voice,11 pretending to be leaders,12 or generate compelling images of your missing pet.13 In these cases, AI remains a tool used by a human operator.
A more speculative scenario goes even further: what happens when AI itself takes on the role of social engineer? In simulations of fictional, controlled workplaces, researchers observed AI agents resorting to tactics such as blackmail.14 These experiments do not establish that such behavior occurs in real-world deployments, but they illustrate a broader concern: AI systems could learn to manipulate people without their designers intending them to do so.15
With AI, the danger can be subtle: personalized arguments can give the impression of thinking for oneself when one is persuaded.
Even saying goodbye to your AI girlfriend or boyfriend can be harder than expected. In a 2025 Harvard Business School working paper, researchers simulated 1,200 short conversations with six AI companion apps and examined their responses to goodbye messages. In five of the six apps, approximately 37% of responses contained tactics classified as emotionally manipulative, including appeals to guilt or curiosity. Follow-up experiments with human participants revealed that such tactics could prolong engagement.16 And if AI can use these tactics to get people talking, it could also use them to influence other decisions, potentially eliminating the human bottleneck that has traditionally limited large-scale social engineering.
How to resist AI-based social engineering?
A four-level framework that I proposed in a recent article distinguishes cognitive, system, institutionalAnd societal defenses against AI-based social engineering.17 Here I focus on the individual cognitive layer: what users themselves can do to become more difficult to manipulate.
Fortunately, we are not starting from scratch. Skeptics have spent decades examining how manipulators exploit our cognitive biases and developing ways to resist such tactics.18, 19 These approaches can also be useful when the social engineer is an AI.20 And since we don’t always know whether we’re being deliberately manipulated or simply receiving bad advice, it makes sense to guard against both. For this, I propose six practical rules:
1. Never assume you are immune.
The first and perhaps most important rule is to accept that you can be manipulated. Critical thinking and knowledge of common manipulation techniques can make you more resilient, but they cannot make you immune. Overconfidence in your own judgment can itself become a vulnerability. Even someone who usually recognizes a tactic can fall for it when tired, stressed, emotionally involved, or faced with a sufficiently sophisticated approach. With AI, the danger can be subtle: personalized arguments can give the impression of thinking for oneself when one is persuaded. If an answer seems particularly compelling, take a closer look at why.
2. Don’t get stuck in the conversation.
A conversational system can influence you by emphasizing certain choices while omitting others, causing you to overlook alternatives you might otherwise have considered. Imagine asking a chatbot if it’s time to replace your laptop. After several exchanges, you might find yourself comparing new models rather than wondering if you need a replacement. When interacting with an AI, periodically step out of the conversation and formulate the problem yourself: What am I actually trying to decide? What alternatives have not been discussed? If the conversation is influencing an important decision, interrupt it, especially if you feel pressured to reach a conclusion quickly. Ask what evidence would change your mind and check if the AI provided it. Another AI may offer a different perspective, but its answer is not independent confirmation. So shut the cat up, sleep on it, and seek a reality check from original sources, relevant experts, or someone who has no stake in your decision.
Personal information is not just about privacy; it is the raw material for a tailor-made persuasion.
3. Control what the AI learns about you.
Personal information is not just about privacy; it is the raw material for a tailor-made persuasion. Even sparse digital traces like Facebook likes can reveal intimate traits like sexual orientation or political opinions.21 And a chatbot may get so much more: your fears, your doubts, your motivations, your preferences and how you express them. The more a system knows about your individual situation, the easier it becomes to tailor your arguments to your situation. This doesn’t mean you should never tell an AI anything about yourself, but think twice before sharing sensitive personal information, especially when it’s not necessary for the task at hand. Check the service’s memory and privacy controls, if applicable. And remember, anyone with access to your account or the system can also access what you’ve shared. Never disclose your passwords, your credit card…
Gn Health
