Back to the catalogue

AI helps complete the task. What remains when it is removed?

Two trials of AI tutoring help distinguish performance during a task from learning that students can later demonstrate independently.

A staircase on a book combines scaffolding and independent supports.

Cover image generated with AI for Ed-IA.

The question missing from the headline

One headline says AI improves results. Another says it harms learning. Before choosing a headline, look at what students did, what support they received and when they were assessed. An answer completed with support and a later independent answer provide different information. A reading intended to guide school decisions needs to make this distinction visible.

Mathematics: assisted practice, independent assessment

Bastani and colleagues study nearly a thousand secondary students in Turkey. A basic GPT-4 chat improves assisted practice but reduces subsequent unassisted exam performance. A tutor with hints and teacher-prepared information mitigates that harm; it does not establish an exam improvement over the control. [1]

Physics: a tutor designed for teaching

Kestin and colleagues study 194 Harvard undergraduates across two physics lessons. They compare a purpose-designed tutor with active learning classes, reversing conditions the next week. The tutor achieves better immediate test results. This design does not establish outcomes a term later. [2]

QuestionWhat to check
Which tool was evaluated?General chat or tutor with prepared teaching content
What was the control?Usual practice, active learning or another intervention
When was learning measured?During support, immediately afterwards or later
Who took part?Age, subject, prior knowledge and setting

Why these results cannot simply cancel each other out

Comparing them helps identify questions, but cannot produce a combined effect: age, subject, intervention and comparison all change. Nor should every difference be attributed to AI as an isolated ingredient. A tutor contains decisions about sequence, hints and feedback. The useful question is which combination helps particular students subsequently do something they could not previously do.

A possible school decision

I suggest testing a short activity with an identifiable objective. Start with an individual answer. Follow it with support that asks for attempts and explanations. Finish with a new unassisted task and a delayed task. Record preparation time, errors and students for whom the support is unhelpful. This is a limited local check, rather than a replication of either trial.

An improvement in a small group might justify a more carefully specified trial. It could also reflect extra time, additional attention or an easier topic. Recording what happened and what changed helps avoid convenient conclusions. Collecting only the finished work produced while AI was available leaves an essential part of the question unanswered.

What remains open

We need to observe retention, transfer to different problems and sustained use with varied groups. Preparing and maintaining good content also has a cost. These studies help refine the trial we want to run; turning them into a general policy requires a broader evidence base.

Ed-IA / César Bosch