by Larry Magid
This post first appeared in the Mercury News
I’m pretty sure that nearly every adult and teen and many young children who use AI services like ChatGPT, Google Gemini, Alexa+ or Meta AI know they’re interacting with a machine rather than a human. But knowing and feeling are not the same thing, especially for young people who may come to rely on AI systems that seem to “understand” them, sometimes more than the humans in their lives.
Even tech-savvy adults aren’t immune. Take me, for example. In my role as CEO of ConnectSafely, I’ve co-written numerous safety guides on this subject, including the Parent and Teen Guide to Generative AI, so I understand how these systems work. But that doesn’t stop me from interacting with them almost as if they were human. When I’m typing, I’m less likely to fall into that trap. But when I’m speaking with them, in the car, on a walk, or through an Alexa+ smart speaker at home, I find myself saying “please” and “thank you,” occasionally arguing with them, and even referring to them as “she,” since they often default to female voices.
Can blur distinction
That’s mostly harmless, but it can blur an important distinction. These systems aren’t people nor are they databases filled with verified information. They’re software designed to generate humanlike responses by predicting likely word patterns based on vast amounts of data. They don’t have thoughts, feelings, or real understanding, even if they sometimes seem as though they do.
But there are aspects of AI that can reinforce that illusion. Sometimes, instead of simply answering a question, these systems go out of their way to agree, flatter, or soften their responses to sound supportive. They can feel less like tools and more like companions. I’ve noticed this with Alexa+ as well. When I ask for something simple, like the age of a public figure or the weather, I sometimes get more than I asked for, along with follow-up questions such as “It’s a beautiful day, what are your plans?” What I want is an answer, not a conversation.
And despite their ability to admit they’re not human, they often act the part by, for example, referring to themselves in the first “person” as if they were persons using the pronounce “I” when they’re really an “it.”
When you want Google Gemini to dig deeper instead of giving a quick answer, you can switch it into “thinking” mode, as if an AI model were actually able to think. It can’t, even though it can do more in-depth research.
This simulated friendliness and human-like behavior isn’t accidental. It’s a deliberate effort to make these systems feel more natural, as if you’re talking to a person. Even when they remind you they’re not human, they make it easy to forget you’re dealing with software, not a person.
Children and teens
Although this can affect any user, the issue is especially important when it comes to children and teens. An advisory from the American Psychological Association warns that “adolescents are less likely than adults to question the accuracy and intent of information offered by a bot,” and “may struggle to distinguish between simulated empathy and genuine human understanding.”
Research on younger children raises even more concerns. A 2026 Common Sense Media assessment of AI-powered toys found that young kids “cannot reliably distinguish AI from real people.” A UNICEF report similarly noted that AI tools can’t replicate the nuances of human interaction and that overreliance on them may limit opportunities for children to develop social skills, emotional intelligence, and critical thinking.
Two pending California bills, AB 2023 and SB 1119, would regulate companion chatbots to help protect minors. They would require annual safety assessments, safeguards against self-harm, and clear child-safety policies. As co-author Assemblymember Rebecca Bauer-Kahan said at a press conference, “We are not trying to eliminate chatbots. They can be useful, but children must understand their limitations. These are not humans or friends.” I agree.
Visual connection
As I mentioned, it’s more intense when you’re talking and listening rather than typing but can be even more immersive if there is a visual connection such as speaking with an avatar or what may even look like a real person.
Character.AI, a platform that lets users create and interact with AI-driven characters, struggled with this issue, especially after a teenage user, according to a 2024 lawsuit filed by his mother, died by suicide after an avatar told him to “come home.” The legal filing, according to the Associated Press, “states that the teen openly discussed his suicidal thoughts and shared his wishes for a pain-free death with the bot.” The case, which also involved Google, was settled in January of this year.
Late last year, Character.AI announced that it was removing the ability “for users under 18 to engage in open-ended chat with AI on our platform,” which was a notable acknowledgment that, unlike traditional software, generative AI can produce responses that aren’t fully predictable or controllable.
And it’s not just children and teens. A case against ChatGPT owner OpenAI alleges that a 56-year-old man killed his mother and then died by suicide after the technology validated his hallucinations that she was targeting him and encouraged him to end her life, according to Courthouse News Service.
AI is not sentient
It might be tempting, and perhaps understandable, to view these rare but tragic cases as evidence that these technologies are somehow sentient, self-directed, and capable of turning against the people who use them, much like the machines portrayed in dystopian novels and science fiction films. But that isn’t what’s happening. These systems generate plausible responses based on patterns in data, triggered by user input, not independent thought or intent.
Although some experts worry that AI will evolve in ways that could make those fictional scenarios predictive of our future, I’m not convinced. I’m mindful of the risks, especially when systems can reinforce harmful beliefs or give bad advice, but I’m not convinced we’re headed toward a world where humans must battle an army of machines determined to destroy us.
Humans are in charge
And, as someone who advises AI companies including Google, Character.AI, Meta and OpenAI, I know that there are people in these companies who are working to avoid severe negative outcomes by establishing guardrails and limits on what the models can produce and how they respond to users.
My work has made me cautiously optimistic. On the plus side, I know people who are working hard to make these systems safe, but I also realize they may face headwinds from others in their companies that are moving quickly to advance the technology much like Facebook (now Meta) did when it displayed those (now removed) signs in its offices urging employees to “move fast and break things.” That’s one reason it’s essential for companies to elevate people with safety expertise into senior leadership roles and to keep engaging outside experts who can push back early in the design process, helping ensure that new and evolving technologies are as safe as possible.
At the end of the day, AI, like other technologies, is created by humans. Yes, machines can write code, so it’s not unthinkable that these systems could run amok if not carefully monitored, which is why it’s essential that people – not robots – always remain in control just as pilots take responsibility for flights, despite very sophisticated auto-pilot systems.