�鶹ԭ��

AI tutors, with a little human help, offer ‘reliable’ instruction, study finds

An AI-powered tutor, paired with a human helper and individual-level data on a student’s proficiency, can outperform a human alone, with near-flawless results, a new study suggests.

The results could open a new front in the evolving discussion over how to use in schools — and how closely humans must watch it when it’s interacting with kids, reports.

In a involving 165 British secondary school students, ages 13-15, the ed-tech startup put a small group of expert human tutors in charge of a , or LLM, offered by Google’s . As it tutored students on math problems via Eedi’s platform, it drafted replies when students needed help. Before the messages went out, the human tutors got a chance to revise each one to the point where they’d feel comfortable sending it themselves.

Students didn’t know whether they were talking to a human or a chatbot, but they had longer conversations, on average, with the “supervised” AI/human combination than simply with a human tutor, said Bibi Groot, Eedi’s chief impact officer.

In the end, students using the supervised AI tutor performed slightly better than those who chatted online via text with human tutors — they were able to solve new kinds of problems on subsequent topics successfully 66.2% of the time, compared to 60.7% with human tutors.

The AI, researchers concluded, was “a reliable source” of instruction. Human tutors approved about three out of four drafted messages with few to no edits.

Image

Eedi Labs

Students who got both human and AI tutoring were able to correct misconceptions and offer correct answers over 90% of the time, compared to just 65% of the time when they got a “static, pre-written” response to their questions.

And the AI only “hallucinated,” or offered factual errors, 0.1% of the time — in 3,617 messages, that amounted to just five hallucinations. It didn’t produce any messages that gave the tutors pause over safety.

The results suggest that “pedagogically fine-tuned” AI could play a role in delivering effective, individualized tutoring at scale, researchers said. Interestingly, students who received support from the AI were more likely to solve new kinds of problems on subsequent topics.

The key to the AI’s success, said Groot, was that researchers gave it access to detailed, “extremely personalized” information about what topics students had covered over the previous 20 weeks. That included the topics they’d struggled with and those they’d mastered.

“We know what topics they’re covering in the next 20 weeks — we know the curriculum. We know the other students in the classroom. We know whether they’re putting effort into their questions. We know whether they’re watching videos or not — we know so much about the student without passing any personally identifiable information to the AI.”

That guided the AI’s strategy about whether students needed an extra push or just more support — something an “out-of-the-box, vanilla LLM” can’t do, she said.

“They don’t know anything about what the teacher is teaching in the classroom,” Groot said. “They don’t know what misconceptions or what topics the students are struggling with and what they’ve already mastered, so they’re not able to dynamically change how they address the topic, as a human tutor would.”

Human tutors, she said, generally have “a really good sense of where the student struggles, because they have some sort of ongoing relation with a student most of the time. An LLM tutor generally doesn’t.”

All the same, even master tutors typically don’t go into a session knowing a student’s comprehensive history in a course, including their misconceptions about the material. “All of that is too much information for a human tutor to read up on and deal with while they’re having one conversation” with a student, Groot said.

And they’re under pressure to respond quickly, “so that the student is not left waiting. And that’s quite an intensive experience for tutors that leads to a bit of cognitive overload,” she said. The AI doesn’t suffer from that. It needs less than a millisecond to read all of those contexts and come up with that first question.”

Even with their personal connection to students, human tutors can’t be available 24/7. Groot said Eedi employs about 25 tutors across several time zones who are available to students from 9 a.m. to 10 p.m. every day, but to give students broader access would require hiring “an army of tutors,” she said.

The new findings could encourage schools to use AI as a kind of “front line” tutor, with humans intervening when a student is “derailing the conversation, or they have such a persistent misconception that the AI can’t deal with it,” said Groot. “We think that would be an interesting way to collaborate between the AI and the human, because there is still a really important role for a human tutor. But our human tutors just cannot have conversations with thousands of students at once.”

The new study, published on Nov. 25 on Eedi’s site and scheduled to appear in a peer-reviewed journal next year, differed in one important way from recent studies that looked at AI tutoring. Researchers at in October 2024 examined AI-assisted human tutoring, in which tutors primarily drove the conversation. But in that case, the AI acted as a kind of assistant, providing suggestions behind the scenes. In the Eedi study, it was the other way around, with AI driving the conversation and humans overseeing it.

Robin Lake, director of the at Arizona State University, said the study is important in and of itself, but also in the context of broader findings elsewhere suggesting that, with proper training and guidance, “AI can be an incredibly powerful tool — and certainly has a potential to take tutoring to scale in ways that we’ve never seen before.”

Under controlled circumstances, she said, it’s also “outperforming humans — that’s really important.”

Lake noted a from Harvard researchers that examined results from 194 undergraduates in a large physics class. They presented identical material in class and via an AI tutor and found that students learned “significantly more in less time” using the tutor. They also felt more engaged and motivated about the material.

Liz Cohen, vice president of policy for 50CAN and author of the recent book “,” said the study provides “valuable evidence” about new kinds of tutoring.

But one of its limitations, she said, is that it relied on 13-to-15-year-olds. “So immediately I have a lot of questions about if the findings are applicable for younger students, especially using a chat-based model,” which may not be a good one for such students.

She also noted that there are many questions around student persistence with AI tutors, including what happens when students get frustrated or aren’t sufficiently engaged in the work?

“I still mostly think that entirely AI tutoring programs are biased towards students who want to do the work or are interested in learning,” Cohen said, “and it’s pretty easy to see that students who aren’t bought in or are frustrated are going to give up more readily with an AI tutor.”

She noted that her 12-year-old daughter has experienced problems persisting in an AI-powered math tutoring program. “She gets frustrated if she can’t get the answer and then she doesn’t want to do it anymore, so I think we need to figure out that piece of it.”

was produced by and reviewed and distributed by �鶹ԭ��.