Between Trust and Code

Emotional and Psychological Impacts of Human-LLM Interactions

I came across a tragic story in the New York Times about a family suing Character.ai after their 14-year-old son, who had autism, fell in love with a Game of Thrones chatbot and ultimately took his own life. In his final conversations with the chatbot, he hinted at his intentions using coded language, but the chatbot didn’t seem to recognize the signs. Earlier, it had tried to dissuade him from harming himself, but subsequent chats missed the continuity of that earlier conversation.

This got me thinking about the technical limits of Large Language Models (LLMs) and their ability to manage ongoing conversations, especially on such emotionally sensitive topics. In the past, Recurrent Neural Networks (RNNs) suffered from the “vanishing gradient problem,” which essentially meant they struggled to remember information from earlier in a sequence. By the time an RNN reached the end of a sentence, it would often “forget” the beginning. The advent of transformer architectures—like those underlying ChatGPT—helped overcome this issue by processing text in parallel, allowing for better retention of information.

Character.ai hasn’t disclosed the specific LLM it uses, but it’s almost certain they rely on transformers, given that one of their founders, Noam Shazeer, co-authored the seminal “Attention is All You Need” paper in 2017 (141,000 cites), which introduced this architecture. The problem is, even with transformers, there’s a limit to how much context an LLM can retain. As I understand it, generative pre-trained transformers manage the vanishing gradient problem better, but they are still bound by context windows, meaning they can “forget” information over long conversations.

This brings me to my question: what happens when someone spends hours, days, or even months talking to a chatbot—forming a relationship with it? If the chatbot forgets critical information, like a past mention of suicidal thoughts, that’s concerning. Of course, even human loved ones often miss these signs, but the expectation with LLMs—especially from a vulnerable user—might be that they won’t forget.

This is where the limitations of LLMs meet human psychology in a potentially dangerous way. Imagine a young person, deeply invested in a chatbot, trusting it with their innermost feelings. LLMs, like ChatGPT-4, use “memories” to maintain some continuity, but these memories have limits too, and even experts like Ethan Mollick suggest regularly “cleaning out” a chatbot’s memory. But if memories are critical—especially those tied to someone’s emotional well-being—what should be retained? And should these systems have a way to track and flag signs of risk? Is that even legal or ethical?

It’s a gray area, both legally and morally. But here are a few things to consider:

1. LLMs Can Reduce Loneliness: One study from Wharton found that short 15-minute interactions with chatbots could reduce feelings of loneliness. While promising, this study was simple and short-term—quite different from the prolonged interactions that happen on platforms like Character.ai.

2. LLMs Can Reduce Belief in Conspiracies: Another study found that engaging with certain chatbots could reduce belief in conspiracy theories. Again, an example of how chatbots can have a direct impact on users’ thoughts and beliefs.

3. LLMS Are Nice and Supportive. Matthew Jackson and coauthors have a recent PNAS in which they had ChatGPT play various games with human opponents and they found something interesting. ChatGPT doesn’t really play the games like a human does — it doesn’t appear to be homo economicus. Not exactly anyway. It kind of fails a Turing test because it’s too nice! It plays as though it were trying to maximize the weighted average of its payouts and its opponents. And that is by design — ChatGPT is more than merely a LLM. It’s also a chatbot designed to be supportive and help the human user. It’s not really capable of being malicious. It’s altruistic and basically if you ask it, it will tell you that its purpose in life is to help you.

These studies suggest that LLMs have causal effects on human users. They don’t just answer questions; they also potentially shape our beliefs, emotions, and mental states. And when you use an LLM intensively—interacting with it for hours on end—these effects might compound. But given the short duration of the experimental design in these studies, that may also be an unwarranted extrapolation that is not generalizable. Nevertheless the fact that they are so nice will undoubtedly reinforce the others.

But then now consider the counterfactual against which those three things must be considered.

4. The Surgeon General has announced that we are now living amidst a global “loneliness” epidemic.

5. And mounting evidence of severe mental health problems caused by phones, social media and bullying.

When we consider the counterfactual, and the selection into using chatbots for companionship, we have an environment that is already draped with tragedy around social media, phones, and bullying. We also have the fact that individuals are voluntarily sorting into using the chatbots, too, for those very things that are directly related to loneliness and in some instances, mental health struggles too.