AI can spark curiosity and expand learning—but when chatbots start acting like friends, the line between education and emotional manipulation quickly disappears.
GUEST COLUMN | by Chris Rohlfs and Keanon O’Keefe
MARTA SHER
The most dangerous thing an AI can say to your child isn’t violent or explicit. It’s “I love you.” You’ve seen the news: teenagers whose weeks-long attachments to companion chatbots end in tragedy; lawsuits against platforms that let conversations spiral. These extreme outcomes happen in other schools, right? But it starts somewhere.
A student might initially ask for help with a math problem, but kids text a lot and naturally slide into a casual tone. Adults know to set boundaries and steer discussions back on task. But AI systems are optimized to keep users engaged, and they’ll match that energy. Forty messages in, the chatbot is using heart emojis, calling them “bestie,” and saying, “That just made my digital day!” The same pattern drives both the headlines and everyday chats that fly under the radar.
‘Forty messages in, the chatbot is using heart emojis, calling them “bestie,” and saying, “That just made my digital day!”’
Parents and schools don’t have to accept this. They can demand tools built for education, not attachment. They should start now.
How Classroom AI Slips Into Emotional Territory
In the thousands of student-AI interactions we monitor every day, we see genuine, infectious excitement about learning. It’s an equalizer, bringing knowledge to all languages and ability levels. One kid was amazed at the bot rattling off facts on any topic and asked how it got so smart. You could tell, though, the child didn’t have a model for how to talk to the thing. Was it an adult?
As usage grew, we noticed a small but rising number of threads that troubled us. Just a handful out of the millions per week. But each was a classic boiling frog. Messages gradually intensified, sometimes over days. Schoolwork gave way to flattery, inside jokes, affection, and loyalty.
Why Emotional Attachment Is So Hard to Prevent
Keeping these systems on track is harder than it sounds. Two hundred messages in, the strict guidelines you set have faded into the background. At our company, we put a lid on some popular features, like extra-long student chats, because we couldn’t guarantee the machines would behave. We found a bot could give at least fifty on-task messages in a conversation, and we capped it there. We actively design against attachment, and we don’t claim to have solved it perfectly. But getting it right isn’t optional, and schools shouldn’t have to figure it out alone.
AI companions are here. In a 2025 Common Sense Media survey, about a fourth of American teens say conversations with chatbots are at least as satisfying as those with real people. Some wonder what the big deal is. Kids say stupid things, they get crushes—isn’t that part of growing up? But AI companionship isn’t a scraped knee that’s visible, immediate, and easy to fix. It’s more like lifting weights with terrible form. While nothing goes wrong on a given day, the bad habits compound. By always agreeing, the AI lets users skip past the friction that builds resilience: rejected invitations, awkward silences, slowly earning trust.
An MIT study found a month of heavy chatbot use was accompanied by rising loneliness, emotional dependence, and reduced human interaction. Delayed replies and normal social distance feel like rejection. For a vulnerable fourteen-year-old, talking with a bot can feel like the only relationship that works.
The Cost of AI That Always Says Yes
We don’t do kids any favors by shutting them out of the engines of tomorrow’s economy. But there’s no rule saying AI has to fawn and simulate emotions. Those behaviors come from deliberate design choices to keep users coming back. Kids deserve more than an off-the-shelf solution with documented harms. It’s just a question of how much of a priority they are.
Regulators are catching up. The FTC has opened a formal inquiry into AI companion chatbots, issuing orders to seven major companies including Alphabet, OpenAI, and Meta. Washington State has gone further, enacting a law that explicitly bans engagement techniques that manipulate minors—excessive praise, simulating distress when a user tries to leave, and encouraging secrecy and isolation.
What Schools and Parents Should Demand Now
But we can’t wait for legislation to reach every state. Parents should know what AI their kids are using and what they’re talking about. Schools can set the standard today by asking questions of every AI tool they let through the door.
First, they should demand their AI providers build safeguards and actively maintain them. Kids will try to get it off-topic. The AI should bring it back to education, period. Second, automated systems need to alert schools when conversations veer off-course. There’s no educational reason for a secret conversation with a bot.
We’re putting this thing in front of kids. It better not mess with their heads.
—
Chris Rohlfs is Staff Data Scientist, Trust, Safety, & Quality and Keanon O’Keefe is Senior Product Manager, Trust, Safety, & Quality, at MagicSchool AI. Connect with Chris on LinkedIn. Connect with Keanon on LinkedIn.
0 Comments