What does good online maths tutoring look like?
A 2017 paper with the UCL Knowledge Lab, from my time at Third Space Learning: a first step towards checking the quality of online tutoring automatically.
Cukurova, M., Mavrikis, M., Luckin, R., Clark, J. and Crawford, C. (2017). “Interaction analysis in online maths human tutoring: the case of Third Space Learning.” In Artificial Intelligence in Education (AIED 2017), Lecture Notes in Computer Science 10331, Springer, pp. 636–643. doi:10.1007/978-3-319-61425-0_80
Free accepted version at UCL Discovery
Third Space Learning ran one-to-one online maths lessons for primary school children. Every week, human evaluators watched and listened to a sample of each tutor’s sessions to check quality. The long-term aim was to automate that. This “industry paper” reports the first step: a tool for marking up what happens in a session, built on what the research says good teaching looks like.
We grounded the tool in three areas: cognitive (what the child knows and can do), metacognitive (what the child knows about their own learning) and affective (how they feel). Ten teachers each used the first version to tag two hour-long sessions, and told us what worked and what didn’t.
What we learned
- It was quick. Most found real-time tagging much faster than the up-to-three-hour written evaluation it might replace, and less open to an evaluator’s bias.
- It asked too much at once. Nobody could watch the tutor and the child, across all three areas, at the same time.
- Some tags were unclear, such as “specific praise” or “recall”, even after training.
- Feelings were hard to judge from a voice alone. Happy and frustrated were easy. Confused and bored were not.
The second version split tutor actions (question, feedback, guidance) from student actions (question, answer, reflection, self-explanation), moved feelings and general teaching qualities out to separate steps, and defined every tag with examples.
The paper’s wider point: to build an AI that judges teaching, you first need a model of good teaching, and you can only build that by having people do the job the AI will eventually do. What confuses those people is also a clue to what will confuse teachers reading the AI’s feedback.