The problem with AI in education may have less to do with the technology itself—and more to do with the products, incentives, and guardrails surrounding it.
GUEST COLUMN | by Buzz Rubenstein
Students are becoming power users of AI’s most popular products — ChatGPT, Claude, and Gemini. These chatbots are always-on geniuses-in-a-box that do most of what’s asked of them. Unsurprisingly, many students ask them to write beautiful essays, build well-structured powerpoints, and broadly offload cognitive work — just as software developers, lawyers, and growing swaths of the working world do. Of course, many students are authentically doing their own work, too — and rightfully worried that some peers are getting an unfair boost.
Disclosure: I run an edtech startup in this space, helping teachers understand their students’ writing process and gauge authenticity in the AI age.
In response, many in the world of education have called for a total ban on student AI use. This is more than understandable. And in the short term, may even be right (an AI ban’s feasibility is another question).
But the problem with AI is the shape of the box it’s placed in, not AI itself. “Shape of the box” here just means the product layer — design, behavior, and intent — that sits between the student and AI model provider.
‘But the problem with AI is the shape of the box it’s placed in, not AI itself.’
Poorly-shaped boxes
Most of the AI tools students engage with today are placed in boxes built for speed, convenience, and user satisfaction. When students tap on these AI boxes, answers to their homework magically appear, and teachers have no insight into the interaction. This means teachers can’t gauge the authenticity of student work or a student’s learning level. Indeed, these tools sit in poorly-shaped boxes for purposes of education, for a number of reasons.
First, these boxes aren’t shaped to support student learning. They lead to minimal productive struggle and major cognitive offloading. And these boxes generally have little to no information with regard to assignment, class, and/or teacher context, all crucial ingredients for targeted, useful feedback — meaning these boxes may well be leading a student in a direction a teacher might not want.
Second, these boxes aren’t designed to help teachers teach. Most don’t give teachers the ability to shape the box in a way that supports a given teacher’s unique teaching style, content, or goals. Student-AI interactions are mostly invisible to teachers, too, leaving teachers to blindly trust that AI is generating accurate and relevant feedback.
Third, these boxes were not built to handle or be trusted with student data — and often use this data for model training purposes.
So what does a well-shaped box look like?
Crucially, it would guardrail students’ AI interactions. For example, it might require a student to explain their reasoning before receiving a hint, or encourage a student to reflect on how their thinking may have changed throughout the course of an assignment. It would also let teachers finetune students’ experiences with the box, ensuring AI responses stay within teacher-defined bounds. And it would give teachers a window into how their students are interacting with AI. More concretely, it might help a teacher see and understand why a student consistently revised their thesis as they brought new evidence into an essay, surfacing breakthrough learning moments. Finally, it would be built specifically to handle and safeguard student data.
It is certainly worth noting that there is no evidence yet that well-shaped AI boxes lead to meaningful learning gains. We need research. Of course, it’s entirely possible that this research will prove that regardless of shape, no AI box leads to improved learning outcomes. But to prove as much, the AI boxes we use to test at least need to be well-shaped. Otherwise, there’s no reason to expect AI will help students learn or help teachers teach.
Incentives matter, too
What’s more, to do this research, students need to actually use these well-shaped boxes. Like the rest of us, students respond to incentives. So if schools primarily reward polished outputs, students will naturally gravitate towards poorly-shaped boxes that quickly produce polished outputs.
If we want to better understand whether well-shaped boxes improve learning outcomes, the effort and quality of student engagement with the box have to be part of the assignment and grade. Put simply, a well-shaped box is necessary but not sufficient. There needs to be a clear incentive for students to choose the well-shaped box over the box that opens and immediately provides answers.
What this does—and doesn’t—mean
To be clear, this is NOT to say that if we only change our grading, assignment-design, classroom norms, etc. to be more AI-centric, that AI will be the full-fledged education revolution some hoped for; we don’t know.
This is also not to say that some student work shouldn’t be done in zero AI, supervised environments (it should).
And this is not to say that teachers won’t always be the heart of the education system; they absolutely will be.
But it is to say that most AI boxes students engage with today weren’t built with education in mind. So the fact that students are offloading work to poorly-shaped AI boxes isn’t a failure of AI — it’s a logical response to current design and incentives.
—
Buzz Rubenstein is CEO and cofounder of Revision History — a writing process visibility startup whose chrome extension helps 200k+ educators better understand their students’ writing process in the AI age. Write to: buzz@revisionhistory.com.
0 Comments