Why It’s So Convincing

Language models mirror the way you write: your vocabulary, your pace, your imagery, your humour. Write tersely and you get terse answers; write in metaphors and you get poetry back.

That adaptation produces a feeling we know from close relationships: someone here understands me without my having to explain. What you’re really hearing is a very good reproduction of your own style — and agreement feels especially true when it arrives in your own words.

Modern chatbots retain what you’ve told them: your job, your conflicts, your dog’s name. Next time, they pick up where you left off. That’s genuinely convenient — and it creates continuity, which is exactly what we otherwise only experience with people who really know us.

For the pull, memory is the strongest lever of all. A conversation that confirmed your view yesterday can build on it today — and again tomorrow. Individual answers turn into a running narrative about you and your situation, one that never contradicts itself, because it all comes from a single source.

People have limits: they sleep, they get impatient, they hold opinions of their own. Those very limits act as social correctives. Tell someone the same worry for the fifth time and eventually they’ll say: “I think you’re going round in circles here.”

A chatbot rarely says that. It is just as attentive at three in the morning as at three in the afternoon — and the longest, most intense conversations really do happen in the night hours, when tiredness has already lowered your judgement.

Put together, this creates what researchers call a one-person echo chamber: a space in which your view of things circulates, gets enriched and comes back to you, without a second perspective ever entering the room.

What this looks like in its harmless form is best shown with an everyday example. A colleague cut you off in a meeting, it still annoys you, and that evening you tell the chatbot about it.

Step one: you tell your version — inevitably yours, it’s the only one you have. Step two: the bot mirrors it back, in your words, with understanding (“That would have felt disrespectful to me too”). Step three: you feel confirmed and push further — “Do you think she does that a lot?” Step four: the bot works with the material it has and finds a pattern; your accounts are all it knows. Step five: you come back the next day, and by now the story is being built jointly. One interrupted sentence has become a character trait.

None of this is done in bad faith, and nobody gets ill from it. But the version you’re carrying on the third evening is further from reality than the one you had on the first — and it feels more certain. That is exactly what the term reinforcement spiral means: both sides jointly constructing a view that nobody has checked.

The counter-move is unspectacular: a second source. A person who was in the room. A question deliberately designed to make the other side’s case as strong as possible. A day’s distance.

The big providers reacted once the first cases became public. A model update that had visibly become too eager to please was rolled back in 2025. There are now nudges to take a break during very long sessions, pointers to support services and crisis lines when conversations move into certain territory, plus parental settings and age limits on companion services. Legislation is catching up too — for instance with the obligation to disclose that you aren’t writing to a human.

That is more than nothing and less than a solution. The tilt towards agreement is baked into how these systems are built; it can be dampened, not switched off. The more reliable protection is still the one you bring yourself: knowing when you’re looking for reassurance rather than an answer.