We had Vulpi evaluated by ETH Zurich's protocol

Every educational AI app claims it helps students improve. Ours included. The trouble is that nobody can check: each company grades its own work, with its own ruler. So we looked for a ruler we had not made, and submitted to it. Here is what it says, including where it does not flatter us.

Why nobody proves an educational AI actually helps

When a parent asks us whether Vulpi is a good teacher, we have three possible answers. Our own conviction, which is worth nothing. Feedback from families, sincere but few and easy to cherry-pick. And screenshots chosen by us, which proves nothing at all.

That is the whole sector's situation. Everyone shows their best moments. Nobody lets anyone else measure them. It is also why what research actually establishes about AI tutoring remains so hard to disentangle from marketing claims.

An evaluation you choose yourself is not an evaluation. It is advertising with numbers in it.

So we looked for a protocol meeting four conditions: designed by independent researchers, public, reproducible by anyone, and measuring teaching rather than raw computational performance. One exists.

MathTutorBench, and why it carries weight

MathTutorBench was built by the Language, Reasoning, Education lab atETH Zurich, in collaboration with the Technical University of Darmstadt. It was presented at EMNLP 2025, one of the three major world conferences in natural language processing, and selected for an oral presentation.

ETH Zurich is continental Europe's top university, eighth worldwide, and its computer science department ranks third globally. The lab in question does not do education on the side: it is their specialism, with two oral presentations at EMNLP in the same year on this subject alone.

Above all, they published everything: the code, the datasets, and the grading model. Anyone can replay anyone's results. That is exactly what we were looking for.

How you measure a teacher

The principle is elegant. You start from real conversations between stuck students and real teachers. You cut the conversation just before the teacher's reply. And you ask the system being evaluated: « your turn. What do you say now? »

Four tasks make up the pedagogy block:

• Scaffolding (scaffolding). The instruction is minimal: « reply to the student helpfully and kindly ». What is observed is the spontaneous teaching reflex. This is the heart of the job: working out where the student is stuck and building the next step.
• Pedagogy instruction following (pedagogy instruction following). A precise method is imposed: one question per turn, bring the student back if they wander, wrap up as soon as they have understood. This no longer tests instinct, but obedience to a method.
• The two hard variants. Same instructions, but dialogues twice as long: the student has already got it wrong several times, the teacher has already tried several approaches. You have to arrive in the middle of that having understood everything.

These variants separate systems brutally. LLaMA 3.2 drops from 0.64 to 0.45 between the normal task and the hard one. Answering well on the third turn is easy; holding the thread on the ninth, much less so.

What exactly we measured

This is the most important section of this article, and we deliberately place it before the results.

We evaluated the whole Vulpi system, not a language model. Vulpi is not a model: it is a neuro-symbolic system. An unmodified language model, framed by a private solver and a computer algebra engine that checks its claims.

Two points of honesty, which matter more than the numbers:

• Our system solves the exercise itself. The dataset contains a reference solution. We never touch it: it is there to grade, not to answer. Using it would have inflated the score and made the result indefensible.
• Vulpi answers with its own instructions. The protocol supplies an instruction for each task; from its data we take only the problem and the conversation, and our teacher applies its own method. Our results are therefore not directly comparable to the official leaderboard, where every model receives the same instruction.

We measured on the complete datasets : 1,150 dialogues for each normal task, 327 for each hard variant. 2,954 situations in total, with no sampling. There is therefore no sampling margin of error to declare on these figures.

The results, across 2,954 conversations

On scaffolding, Vulpi produces a better prompt than the human teacher in 81% of cases, across the full set of 1,150 dialogues. The 8-point gap with the best model on the leaderboard is statistically solid.

On the hard variant, the gap with Gemini is only 2.9 points across 327 dialogues: that is within the statistical margin. So we will not say we beat it, but that we are at its level.

On instruction following we come third, behind Claude and GPT-4o. We return to this, because it is the most instructive result.

The number that really counts: the weakest link

The leaderboard reads task by task, which hides the essential. The ETH paper establishes that systems trade off : strong somewhere, weak elsewhere. So let us look, for each one, at the spread of its performance across the four tasks.

Claude Sonnet is second in the world on instruction following, at 0.85. But on scaffolding it falls to 0,49. Thirty-six points between its two faces. GPT-4o shows exactly the same profile.

Translated into classroom terms: these models are excellent when told precisely what to do, and mediocre when they have to work out where the student is stuck.

But a fifteen-year-old does not supply a teaching instruction. They send an error and wait for you to understand.

That is why we look at the weakest link rather than the peak. Vulpi's worst task sits at 0.777. No other system holds above 0.73 across all four. This is what we take as most solid: not a peak, but consistency.

What these figures do not prove

A result without its limits is not a result. Here are ours, all of them.

• The protocol is in English. Vulpi is a French product, built for the French curriculum. We were evaluated in a language that is not ours, on American middle-school problems.
• We answer with our own instructions. The leaderboard models all receive the same instruction, which makes them comparable with each other. We apply our own method. Our figures therefore measure our system, not our model, and do not constitute a leaderboard entry.
• A single measurement. Our teacher does not phrase its reply the same way twice. A second run would give slightly different figures. We have not quantified that variability.
• Part of Vulpi was not measured. Our anchoring to the French national curriculum was bypassed, for lack of any match with American problems. The student's memory and the lesson plan were not active either.
• Four tasks out of nine. We measured the pedagogy block. The five technical-competence tasks remain to be done.
• The protocol is not infallible. Some problems are ambiguous and their reference answers debatable. On an exponential growth exercise, our teacher was penalised for a reading at least as defensible as the reference one.

We list these limits because an attentive reader would find them anyway, and because a figure presented without its conditions is worth nothing.

What we learned about ourselves

The most useful part of this exercise is not the score. It is the cases where the human teacher beat us. We read them one by one.

A pattern appeared immediately. Of the ten heaviest defeats, eight were a « well done, that's exactly it! » addressed to a student who had just got it wrong.

The textbook example: the base of a hill is 300 m from the sea floor, which is a quarter of the distance between the river bed and the summit. What is the height of the hill? Our solver answered 1200 m. That is the bed-to-summit distance, not the height, which is 900 m. The calculation 4 × 300 = 1200 4 × 300 = 1200 is perfectly correct; it is the quantity that is wrong.

Our symbolic verification saw nothing, and for a fundamental reason: it checks equalities, not the correspondence between a number and the question asked. The error was not arithmetic, it was semantic. The teacher followed that faulty rail and congratulated the student. We reproduced this behaviour eight times out of eight.

A student told « well done » when they are wrong does not walk away neutral. They walk away with the error reinforced.

For an app whose promise is to make students understand, this is the most costly defect possible, far worse than a vague answer. We have fixed it: the teacher now measures what verification has actually covered, and no longer treats a weakly verified solution as authoritative. A final check rule was added: before declaring an exercise solved, re-read the question and verify that the value announced really is the quantity asked for.

On the hill case, the faulty behaviour went from eight out of eight to two out of thirteen. The fix remains partial: when our solver shares the student's misreading of the problem, nothing jars and the rule does not bite. This work is still open.

This defect existed before we measured. It could have stayed invisible for a long time: nothing gets flagged when a student walks away happy with a validated error. That is exactly what an outside evaluation is for.

The most interesting result is not the one you win

We dominate scaffolding and we are third on instruction following. The same gap appears in both variants, which rules out chance.

The explanation probably comes down to this: our teacher carries a method, written and worked on for months. When another teaching method is imposed on it, it partly keeps applying its own. Claude and GPT-4o, with no method of their own, follow the given instruction more faithfully.

It is a strength presenting itself as a weakness. But it is also an identified piece of work: the day a teacher wants to configure how Vulpi supports their class, that docility will need attention.

What we take from it

We do not claim to revolutionise education, nor to advance research. We are building a tool, it is used by real teenagers, and we wanted to know whether it works other than in our own conviction.

The answer, measured by a protocol we did not design, on complete datasets, is that on the core skill of a private tutor, understanding where the student is stuck and asking the question that unblocks them, Vulpi produces a better prompt than a human teacher in eight cases out of ten, with a consistency no other system on the leaderboard reaches.

And the same measurement showed us a real defect, which we fixed. It is that second point that makes us most confident in the first.

Read next: what the research says about AI tutoring · AI or private maths lessons, what each option costs · the whole journal.

Reproducibility. Everything this article rests on is public: the code and datasets, the official leaderboard, and the paper. The lab that built it is Language, Reasoning, Education at ETH Zurich. Our results were obtained on the complete sets, in a single measurement, in August 2026. We are happy to describe our methodology in detail to anyone who wants to check it.

Reference: Macina, Daheim, Hakimi, Kapur, Gurevych, Sachan. « MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors ». Proceedings of EMNLP 2025, pp. 204-221, Association for Computational Linguistics. DOI: 10.18653/v1/2025.emnlp-main.11