On the AI features that now arrive switched on, and the right to delete that stops working the moment a model learns.
GUEST COLUMN | by Daniel Kilback
Most schools don’t think hard about what happens to student data once a new tool is in the building. They’ve never had to. Whatever goes in can come back out. Close the account, delete the files, ask the vendor to remove what they hold, and the data leaves more or less when the relationship does. That assumption has held for so long that nobody says it out loud. It sits underneath every decision to try something new.
‘That assumption has held for so long that nobody says it out loud. It sits underneath every decision to try something new.’
I watched a version of this recently, consulting for a school whose families came largely from the diplomatic and military world. The kind who move every few years. Some of them started asking a question most schools never hear. What had the school collected from their child, what had it generated about them, and could any of it be cut back before the family’s next posting? They weren’t worried about a breach. They wanted the trail minimized before they moved on. It turned out to be one of the harder things the school had taken on, and not because anyone resisted it. The systems were built to gather and keep. Nobody had been asked to run them the other way. Minimization wasn’t a habit there. It was barely a category.
When Deletion Stops Meaning Deletion
And that school was at least working with data it could still find. A model takes away even that. When a system learns from a piece of data, the data no longer sits in a row you can locate and delete. It dissolves into the model’s parameters, in no single place you can point to. You can delete the file you uploaded. What the model learned from it stays.
The regulators say this plainly. The UK’s Information Commissioner’s Office has held for years that AI models can contain personal data, and that erasing it may not be possible without retraining the model or deleting it outright. CNIL, in guidance updated this January, reaches the same conclusion: to remove data from the heart of a model, you retrain it; filtering is the weaker fallback when you can’t. The European Data Protection Board frames erasure as reversing what the model memorised, which means taking out both the original data and the mark it left behind. So as things stand in 2026, getting one child back out of a trained model means retraining the whole thing or scrapping it. There’s no reliable way to lift out a single student. The regulators are honest that this could change as the techniques improve. It hasn’t yet.
What Regulators are Saying About AI Erasure
In the United States this isn’t theoretical. When the Federal Trade Commission decides a company trained on data it shouldn’t have, the remedy hasn’t only been to delete the data. It has ordered the models destroyed along with it, a step it calls algorithmic disgorgement, and it has reached for it in cases involving children. The classroom platform Edmodo was ordered to delete the models it built on student data it had collected without proper consent. In the Alexa case the agency was blunt about why. A company can’t keep children’s data forever, and certainly not to train its algorithms.
The AI Features Arriving Switched On
Now look at where that lands. The writing platform with the assistant built in. The reading app that listens to a child sound out words. The tutoring system that adapts to one student across a year. Every one of them is adding AI, and when the vendor switches it on, it tends to arrive already on. Student data is feeding models right now, and the right you were relying on to undo that has quietly stopped reaching it.
No breach in any of this. Nobody broke in. The data did exactly what the contract allowed. The gap isn’t the teacher who clicked accept, or the administrator who signed off on the tool. It’s that the question which would have caught it was never anyone’s job to ask.
The Procurement Question Nobody Owns
It belongs in the contract, and the ICO says as much. When you bring in an AI service, you have to choose one that lets you honor people’s rights over their data, or you can’t meet your own obligations as the school holding it. Staff training and an acceptable-use policy come afterward. By then the decision that mattered has already been made, in a procurement nobody treated as a data decision.
So the thing to ask isn’t where the data is stored, or whether it’s encrypted on the way there. Both matter, and both assume the data can still be reached. The deeper question sits underneath them. Which student data feeds the model, and can any of it come back out? Most vetting never gets that far. A vendor who can answer it, who can tell you what trains the model, what doesn’t, and what a deletion request actually does, has told you something useful before you sign anything. A vendor who can’t has also told you something.
This is hard to do well, and that’s worth saying out loud. The honest answer might be that the vendor doesn’t fully know yet, and some of it can’t be settled by a clause. But it’s this data, more than almost any other, where the cost of guessing is highest. A student record isn’t thin, and it doesn’t expire the day they leave. It’s built when a person is too young to agree to any of it, by a child who doesn’t know it’s happening and won’t find out. If harm comes, it comes years later, to an adult who can’t trace it back to a feature a school turned on when they were nine. We lean on “we can always delete it later.” For a child’s data, that’s the promise that has already quietly expired.
I want the people signing these contracts to know what they’re agreeing to, while the choice is still theirs to make. The next time a tool offers to switch its AI on, someone should be able to answer a plain question before anyone clicks yes: which student data trains this model, and can it ever come back out? If no one can answer, that’s your answer.
—
Daniel Kilback is Director of Technology & Innovation and Data Protection Officer at the American Community School Amman. He writes about technology, privacy, and the long-term implications of digital systems in education. Connect with Daniel on LinkedIn.
0 Comments