Personalized learning with HRM
Can a small reasoning model plan what a student should learn next better than an LLM?
Summary
We built a learning platform that recommends math videos from what a student has practiced, with two engines side by side: a prompted LLM and a Hierarchical Reasoning Model (HRM). A re-check after the project showed that the HRM had been trained on the wrong target, so its reported accuracy only matched a model that always answers "correct": a lesson in testing what a number really measures.
Neither engine beat its simple baseline: the LLM matched a one-line rule, and the HRM, re-checked after the project, did not beat always answering "correct".
Key results
- 36% vs 35%the LLM's next-skill accuracy vs a one-line rule (the student's most-practiced skill)
- ~1 min → instanttime per recommendation, LLM vs HRM
- 70%share of correct answers in the data: what the HRM's reported 69.5% really measured
The problem
Most recommenders optimize for clicks and watch time, but a learning platform should recommend what helps a student learn next. LLMs are flexible but slow, costly and can invent answers. Hierarchical Reasoning Models are small brain-inspired networks with a slow “planning” module and a fast “detail” module, and they had never been tried for personalized learning. In this research project (TER), supervised by Bertrand Delezoide, our team of four compared the two for recommending the next math skill to practice.
What we did
- Used ASSISTments 2012–2013, a public dataset of 2.6 million student answers to math exercises over 198 skills, cut into windows of 20 interactions.
- Trained an HRM (about 27 million parameters, 2 high-level and 2 low-level cycles per pass) on these windows.
- Built an LLM engine: qwen2.5 (1.5B) through Ollama gets the student’s history, a summary of what they have mastered and the list of skills, and must answer with valid skill names; invented skills are filtered out.
- Built the platform: a chatbot (Groq API) that finds which skill a student wants to work on, a Supabase database for users and their history, and a Gradio app on Hugging Face Spaces that shows a YouTube video for each recommended skill.
- Handled new users without an entrance quiz: the system starts from the most similar existing student’s history and adds the current session.
- Compared both engines on accuracy and in an A/B test with a simulated student.
- After the project, Youssef El Otmani re-ran the team’s HRM checkpoint on held-out data and compared both engines with simple baselines.
Results
The LLM was slow and only as good as a simple rule. It named the right next skill for 31 to 36% of test students and took up to a minute per answer. Simple rules on the same students do about as well: “the same skill as last time” reaches 28.8% and “the student’s most-practiced skill” 34.7%. The HRM answered instantly, and our report gave it 59.3% after 10 epochs and 69.5% after 150.
The re-check showed that the HRM numbers did not measure next-skill prediction. The data script labeled each step with whether the next answer was correct, not with the next skill, and every 20-row window mixed several students. The 69.5% equals the share of correct answers in the data, which is what a model that always says “correct” scores. On 512 held-out sequences, the HRM reached 66.5% against 70.2% for always answering “correct”, with an AUC of 0.52, close to chance, and its skill suggestions were close to random.
| On held-out data | Result |
|---|---|
| LLM, next skill (best run) | about 36% |
| Rule: the student’s most-practiced skill | 34.7% |
| Rule: same skill as last time | 28.8% |
| HRM, next answer correct or not | 66.5% (AUC 0.52) |
| Always answer “correct” | 70.2% |
What I learned
- Always compare against a trivial baseline. “Always correct” and “the most-practiced skill” would have shown immediately that neither number meant what we thought.
- Check what the label actually is. One line in the data script turned a next-skill model into a correct-or-not model, and every later number inherited the mistake.
- Speed is part of the user experience. A one-minute wait per video is not usable, however good the answer.
The re-check was done by Youssef El Otmani after the project; its scripts and results are in the team repository.