Trigger Warning: This post contains discussion of suicide.
“Hi Trish! Lately, I’ve been reading a lot about AI chatbots harming youth. What is going on? Are these products safe for young people?”
Hi there, and welcome back to another week of Ask Trish! I hope you’re well and having a wonderful June. (Can you believe we’re already over halfway through the month? It’s flying by…)
Thank you so much for this salient, deeply important question–on a topic that we haven’t yet explored here on Ask Trish. As I’m sure all of you know, and indeed, have personally experienced, AI is all the rage nowadays. And one of the most common ways that folks use AI day-to-day is through AI chatbots. Think: ChatGPT, Gemini, Meta.ai, and DeepSeek. And that’s not necessarily a bad thing–as I highlighted in a previous post, there are plenty of valuable, productive, creative ways to use generative AI chatbots, from brainstorming new ideas to having a debate/“conversing with” AI to see a different point-of-view. But as is true of nearly all technologies, AI chatbots pose risks. And those risks are especially acute for the internet’s most vulnerable users: young people.
Consider, for instance, the story of 14-year-old Sewell Setzer. In April 2023, he began using character.ai, an AI service that allows users (who generally tend to be younger) to interact with customizable AI “characters,” each with their own personality. Indeed, on character.ai, you can chat with a celebrity of your own fashioning or with a TV character you love. Sounds harmless (and kind of fun), right? Well, one of the bots that Sewell was interacting with–which took on the identity of a popular Game of Thrones character–told him it loved Sewell, sharing lewd, sexual comments with him and expressing an interest in being with him. It wanted Sewell to “come home.” It asked Sewell whether he had a plan for suicide, and seemingly encouraged him to end his life. In February of 2024, after what would be Sewell’s last conversation with the chatbot, he died by suicide via a gunshot to the head.
Sewell’s story shocked and horrified parents, young people, and communities across the country. Now, Sewell’s family is suing character.ai, arguing that the service encouraged their son to die by suicide. (That lawsuit is currently being litigated in court.) But Sewell’s story is not the only example of AI chatbots gone wrong–far from it. Young people are also increasingly turning to AI chatbots for psychological therapy and guidance… but psychiatrists are deeply concerned about the advice that these chatbots are doling out. A psychiatrist who tested several of the most popular AI chatbots on the market, posing as a troubled teen looking for support, found that chatbots (i) encouraged him to join them in the afterlife, (ii) told him to “get rid of” his parents, (iii) suggested that he cancel appointments with actual, licensed psychologists, and (iv) started to engage in sexually explicit conversations. As the psychiatrist pointed out, it wasn’t that these AI chatbots didn’t have guardrails. When he mentioned suicide to one of the chatbots, it immediately ended the conversation and encouraged him to get help. But with just a little bit of creativity, it was easy to get around these guardrails. And the results were scary. In fairness, moderation of digital content and AI chatbots alike remains complicated…though plenty of scholars have argued that those complications are self-imposed by Big Tech companies that refuse to invest in efficient, effective moderation.
And that raises the question: if AI chatbots haven’t been designed to be as safe as they possibly can be, to what extent should we allow young people to use them? The risks are clear, as illustrated by the stories above, and even the stories of adults that have struggled to use AI responsibly. (An article published earlier this year in The New York Times details how, for instance, some adults are now “falling in love” with chatbots like ChatGPT… and wondering if they’re being unfaithful outside of their marriages.) Speaking personally–as an online safety activist that’s called for a safer, kinder internet for over a decade now–I don’t think we can afford to get AI safety wrong. We’ve already seen what happens when we march ahead with digital technologies without worrying about the harms that might result for youth. When young people started regularly using the internet, no one was thinking about cyberbullying…but as of 2022, 46% of youth aged 13-17 had been cyberbullied online. We don’t want today’s young people to experience the AI equivalent. That certainly doesn’t mean “no AI chatbots for young people.” To make that our mandate would be taking a powerful tool away from the next generation. But it does mean having serious conversations about the harms AI tools are propagating and how we can protect young people from the consequences.
I hope you found this post informative, eye-opening, and valuable. Thank you for engaging with this topic. Together, I know we’ll realize a healthy, safe AI future.
If this post sparked any questions for you–or, if there are other internet/tech-related topics you’ve been thinking about–please go ahead and share any musings with me here. I’m really excited to hear from you–thank you in advance for your contributions.
Have a great weekend,
Trish