Anatomy of an AI Math Tutor

I find it hard to come across a meaningful discussion about AI tutoring, because the conversation usually falls into one of three camps. The first makes broad claims, with poor evidence, that AI tutors are already better than teachers. The second insists, with equally weak support, that AI tutors will never work, usually by vaguely appealing to motivation or the social dimensions of learning. The third camp consists of tech startup enthusiasts who have successfully solved one piece of the tutoring puzzle–such as a learning progression model or natural speech-to-text–and then prematurely declare the entire problem solved.

My view is that AI tutors do not get close to human ones–yet. What I would love to talk about is exactly why. What are the missing building blocks? Is it possible to develop them with current technology? If not, what needs to be researched?

There are many heated debates in pedagogy about explicit instruction, knowledge, skills, scaffolding, and countless other topics. They are mostly unproductive because people are arguing over the same term while imagining different things. Even so, I would love to be able to reliably guide AI tutors at this higher level, for example by asking them to vary the timing and scope of scaffolding and then observing how that affects student outcomes. In other words, I want AI tutors that can undergo professional development.

What follows is my attempt to outline the building blocks of such an AI tutor. As the CEO of Levebee, a math learning app for elementary students, I am focusing this text specifically on elementary math tutoring, so some of it may not fully apply to other subjects or higher grades. Similarly, our background is in individual math interventions: intensive one-on-one sessions designed to reteach whatever is needed so that students can again enjoy learning alongside their classmates ASAP. As you read on, picture a student working one-on-one with an AI tutor for a limited period, not an entire classroom using tablets all day. This may be a narrow use case, but it requires relatively little change to how schools operate. It already reconciles shared learning goals with personalized learning, overcoming a common barrier to class-wide implementation. These interventions already take place in schools, but there are not enough trained teachers to provide them to every student who needs one. If we are going to implement AI tutors, this is in my opinion the best place to start.

The thoughts below are based on video recordings of these one-on-one human math intervention sessions, paired with voiceovers by the same tutors retrospectively explaining their thought processes. They also draw on 15 years of experience developing a learning app we can confidently stand behind through outcome-based contracts. Some of these building blocks have already been developed at Levebee, some are currently in development, some are in the queue, and some we still have no idea how to build yet (if you do, I would love to hear from you).

I will occasionally mention other EdTech tools and speculate about why they may not achieve their intended outcomes. This criticism is not driven by hostility. I admire the people who have decided to work in EdTech, enjoy talking with them, and hope to help accelerate the field’s progress. But productivity depends on minimizing the amount of work that must ultimately be discarded, and in my opinion, too much investment is currently flowing in the wrong directions.

I invite you to point out any gaps or flaws in the text below as well. I genuinely hope it encourages more nuanced discussion and constructive collaboration. In any case, this is where our thinking has led us so far.

The Core Architecture

At a high level, an AI tutor does not seem all that complicated.

First, you need a Learning Progression Model: a map of the subject’s underlying skill dependencies and instructional sequence. These are two different things, as I will explain later. Together, they provide both the map and the planned route through it.

Next, you need Learning and Assessment Activities that help the student learn each subskill and determine whether they have mastered it.

The resulting data then goes to a Pedagogical Decision Engine, which evaluates the student’s interactions and defines the pedagogical intent for the next step.

Finally, a Response Generator turns that intent into a concrete action, such as saying something or manipulating an object on the screen.

As always, the devil is in the details.

Learning Progression Model

Learning math happens through small, gradual steps. Even seemingly simple concepts like comparison cannot be mastered all at once. A child needs to understand the words more, fewer, and equal (and later, n more than and n fewer than), navigate a number range from 0 to 5 (and later 0 to 10), and also grasp the ideas of conservation (if I move a picture from the red box to the blue box, it will no longer be in the red box) and reversibility (more in the red box means fewer in the blue box). A good math tutor needs to assess a student's abilities at this level of granularity in order to effectively personalize activities. Recognizing that a student struggles due to issues with comparison skills, but not specifically identifying comparison vocabulary as the root cause, can still lead to selecting an inappropriate learning activity. A near miss is still a miss.

I think of this as a question of map resolution. To navigate well, you need a map with enough detail—whether you are lost in a city or lost in math.

Skill Breakdown

Curricula are not written at this level of detail. In practice, it is the job of teachers to break broad topics down into skills that can be mapped to specific learning activities. Doing this well is rarely taught explicitly; it is usually part of the tacit knowledge educators build over years of experience. Because this knowledge is not documented, it is also absent from training data, which explains why LLMs are currently so bad at this. This is why I am skeptical that AI tutors built on curriculum-level data from for example Learning Commons will work. In our experience, you need at least 10 times more granularity than a typical curriculum provides.

The decomposition is difficult to do even for classroom teachers. To get good at this, you need to closely follow the thought process of each individual child—something that is extremely hard when you are working with 20 students at once. Moreover, this is not a general ability. It is highly specific to each math skill. What has worked best for us is hiring experienced math interventionists who work one-on-one with students and have observed thousands of children move through a specific part of the curriculum.

Some EdTech companies, such as IXL, recognized the demand for granularity but took a shortcut: they labeled each existing activity as a "skill," even when students were literally doing the same thing, only with different pictures. To develop a usable learning progression model, skills must be genuinely distinct. You need to be able to find a student who can do A but not B. This was the first thing we tested in our research, using a correlation matrix like the one below.

In contrast, what is often treated as a single skill actually consists of several separate ones: typically vocabulary, modeling, abstract calculation, and fluency. Many apps focus on only one of these types, with vocabulary omitted entirely. A good tutor needs to address all of them.

Even within K–3 math, Levebee recognizes a thousand distinct skills.

It seems that Math Academy found a similar order of magnitude in the higher grades as well.

Skill Dependencies

Listing the skills is only the beginning. An AI tutor must also understand how those skills depend on one another in order to accurately assess what a student truly knows and choose the right next activity. The challenge is that students often have gaps that reach back years. We have seen eighth-grade students still struggle to write eight hundred seventy-six as a numeral because they do not yet understand place value. And this is far from anecdotal: more than a third of 15-year-olds nearing the end of secondary school perform below basic PISA Level 2 in math¹², a level that, for example, includes solving problems involving simple ratios.

Single-topic math learning apps are therefore in principle ineffective for struggling students. An app focused solely on fractions cannot effectively teach all students fractions unless it also covers multiplication, division, addition, subtraction, and, for some children, even early numeracy. You need to cover it all starting from pre-school to be always able to go back.

The dependencies can be researched empirically by assessing students on each skill without any adaptive algorithm, then measuring the probability of passing or failing one skill based on their performance in others. It's expensive, time-consuming, and a bit torturous for students, because it tests them on skills they are likely to fail, so it cannot be a regular part of the in-app experience. As a result, we had to work with sample sizes in the hundreds rather than the tens of thousands we typically use in our other research projects.

Nevertheless, this research allows us to make the diagnostic assessment adaptive while inferring far more in 20 minutes than would otherwise be possible. Assessment time matters. At first, longer assessments provide more information, but the returns quickly diminish. Beyond a certain point, a longer assessment is increasingly likely to measure perseverance rather than actual skill. It also carries a higher opportunity cost: that time could be spent teaching the student instead. Experienced math interventionists know this. They keep assessments as short as possible and stop as soon as they have enough information to make a decision about next step.

Instructional Sequence

As promising as it sounds, this skill dependency map alone is not enough to build an AI math tutor. In practice, you also need instructional sequence. There are two reasons for this.

First, some skills simply do not have strict dependencies. A student’s mastery of the 3 times table does not necessarily imply mastery of the 2 times table, even if the 2 times table is usually taught first. In cases like this, the algorithm cannot confidently label one skill as a mastered prerequisite for the other. Still, instruction has to move forward at some point, and that means beginning to teach the 3 times table when student masters 2 times table.

The second reason, why a separate learning progression model is needed, is learning debt. As software developers, we are all too familiar with the concept of technical debt³. Implementing features in a quick and dirty way can help you meet deadlines with a functional solution, but it creates a debt that must be repaid with interest later when you want to build additional features on top of it. If you never address this debt, messy code will make adding new features increasingly difficult, until eventually the only solution is to throw everything away and start from scratch.

Learning math works the same way. Some skills are not strictly necessary to get the right answer in the moment, but failing to master them creates learning debt that makes later topics harder to learn. For example, a student can correctly solve a six-digit addition problem without really understanding place value. In the extreme case below, they only need to know that 1 + 1 = 2.

But this shortcut does not generalize well. It would be far better for the student to build a solid understanding of place value earlier, when working on simpler problems within 20.

To determine the order in which skills should be taught, we need instructional sequence. It uses the same set of skills as skill dependencies map, but they differ in how those skills are connected. One model shows which prerequisite skills we can confidently mark as mastered without directly testing them, and which skills we do not need to test because the student would almost certainly not be able to do them yet. The other model shows which new skills a student should begin learning after mastering a given skill.

Here is what the second model looks like for K–3 math in Levebee. The same skills, but with different connections between them.

Studying this model scientifically is much harder, because its effects take time to appear. AI cannot speed up this research much; you simply have to wait for students to continue learning.

Summary

Overall, I believe learning progression models, as building blocks of an AI tutor, are no longer a major technical bottleneck. With current technology, we know how to build them into software. What they need is time, sufficient investment, and rigorous, honest research to determine which models produce the best learning outcomes.

They are, however, only one of many required building blocks.

Learning and Assessment Activities

Once we have mapped the skills, each one needs a corresponding activity for both learning and assessment. Sometimes the same activity can serve both purposes, but not always. To improve the reliability and validity of the assessment, we may simplify an activity into a set of multiple-choice questions. However, multiple-choice questions are a poor learning environment, because they quickly become repetitive and students tend to invest less and less effort when answering them⁴. Videos are not much better: students often lose attention or skip them altogether⁵. What I find useful is to imagine a human tutor doing the same thing as your app—for example, silently handing students one worksheet after another or disengaging and telling them to watch a video instead. Does it look like good teaching when you picture that? What good tutors actually do is work on something alongside the student, discussing it as they go.

Interactive Components

An AI tutor needs hundreds of interactive components at its disposal—picture boxes, grids, number lines, blocks, graphical organizers, diagrams, whole-part models, drawing and writing canvases, speech-to-text modules, and many more—often customized or combined to teach a specific skill. AI tutor must be able to receive a stream of data about how the student interacts with these components, and it also needs to manipulate the objects themselves.

This is where recent progress in generative AI has allowed us to make a breakthrough. Not by using AI to program these components for us but by implementing software architecture that wouldn't be maintainable without AI. We can now generate and manage a vast number of learning activities as state machine configurations, with interactive components serving as states and pedagogical decisions as transitions between them. The state machines can be adjusted by the AI in real time, allowing us to finally implement the "I do, we do" part from "I do, we do, you do" instructional model. They can also be generated using higher level instructions, close to the scaffolding example mentioned at the beginning.

Skill-Dependent Tasks

As excited as I am about this, the hardest part is still designing learning activities that students can complete only by using the specific skill from the learning progression model. Engagement does not equal learning. Most students tend to take the path of least resistance and if the activity can be completed using less effort using some kind of shortcut, it will. This is where current technology reaches its limits. We don't know how to verify this programmatically. We found only one effective way to test this: physically observing students as they work through the activity. I personally observed more than 500 students using Levebee over several months. We made notes on where kids hesitated, got confused, asked questions or tried something unexpected and then checked whether our adjustments worked. This process helped us improve dramatically, but it does not scale well beyond creating our own intuition for designing these activities in the future.

Here comes a recurring theme in this text. A human who is physically present with a student can gather orders of magnitude more context than current apps can. To even begin approximating that level of data, you would need cameras, eye-tracking, audio recording, and many other sensors paired with long-term memory. It is unclear in which situations this level of surveillance would be justified. Even if these technical and ethical concerns were solved, simply collecting more data would still not be enough. What AI still lacks is a human-like theory of mind: the ability to infer what someone else is thinking⁶. Human tutors construct and continuously update a model of the student's thought process. The student might appear completely still while thinking deeply, or they may seem very active but are actually just clicking mindlessly. Tutors respond not directly to what the student is doing or saying, but to discrepancies between the inferred thought process of the student and the one they want to teach them.

Engagement & Accessibility

Suppose you do figure out how to ensure that the student is learning what you intend. How do you then prevent them from getting bored with it? As game developers know, it is difficult to make something stay fun over a long period of time. Which video games have you played for years? If any, they are probably online multiplayer games, not single-player experiences, even personalized ones. And those games are not constrained by the requirement that every activity must teach a very specific skill.

You can still try to make your learning activities "more fun" but this often leads to seductive details effect⁷. Unrelated stories or animations are engaging but distracting, kids remember the fun story but not the concept they are supposed to learn. Gamification can increase engagement or time spent in the app, but it often encourages fast guessing (Kahoot)⁸, simply glues unrelated fun activities on top of learning (Prodigy), or relies on loss aversion to emotionally blackmail students (Duolingo streaks).

Trying to "make it fun” is not the only way to distract children from learning. A less discussed but equally important source of distraction is poor usability and accessibility. Technical friction can pull a child’s attention away from the educational task and toward figuring out how to use the app itself—especially for children with special education needs and English language learners. Some of the children we work with have never used a tablet before. They may not know that an X usually closes something, or that green usually means confirm while red means delete⁹. On top of that, many schools still rely on 10-year-old iPads and poor Wi-Fi connections, which makes solving these problems even more expensive.

At Levebee, we created the diagram below as a reminder of how narrow the learning zone really is, and why getting hundreds of small, often invisible, details right is essential.

One way human tutors make learning sessions engaging is the speed of interaction. In our research on one-on-one math intervention video recordings, we observed an average of 26.15 verbal exchanges per minute, or one interaction every 2.29 seconds. These exchanges included short confirmations, single-word questions, finishing each other’s sentences, and more. What this number doesn't include is nonverbal communication, which is much faster than speech. On top of speech, students and tutors constantly respond to each other’s gestures and facial expressions, making the interaction even more dynamic. For example, a human tutor and student can easily synchronize nonverbally during “say it with me” activities. We currently have no way to replicate this reliably in an in-app environment.

The best we can do is make sure any verbal feedback arrives as quickly as possible. After many optimizations, we can now start playing audio for 90% of AI responses within 2 seconds of a student interaction. That is not ideal, but even young children can usually tolerate it. Any longer, and they start to disengage. That is why I am skeptical about the success of apps with feedback cycles that take minutes, such as Snorkl.

Summary

Generative AI helps us create much richer interactions at a much larger scale, though they are not as fast as human-to-human interactions. But beyond physical testing, we are still not much closer to verifying, with human-like accuracy, whether those interactions actually teach what we intend. I would call this an evolution rather than a revolution.

Pedagogical Decision System

At its core, tutoring is about making thousands of pedagogical decisions. Some are big, like whether to jump back two years in the curriculum. Others are small, like what to do, ask, or say in the moment.

Most of the time, these decisions are grounded in learning progression models and guided by the learning activity. But not always. In our video recordings, tutors sometimes deliberately diverged from the learning progression. For example, when a student had very low self-esteem, the tutor chose an activity that was not at the edge of the student’s learning zone, but a little below it. In other cases, even when the tutor should have jumped back to teach place value within 20, the student resisted working on second-grade content because he was a fourth grader. As a result, the tutor had to stick with a less effective approach, teaching place value through addition and subtraction within 100 while quietly sneaking in problems within 20.

Again, it comes down to a lack of data about many social and emotional states involved, as well as a lack of theory of mind. Dan Mayer from Amplify is skeptical about solving this, noting that "year after year, you will learn new lessons about the human dimensions of teaching and struggle to turn those lessons into software".¹⁰ But why not to learn these lessons about human dimensions of teaching and at least map out what would be needed to turn them into software? Why not focus research and development on these real bottlenecks instead of continuing to invest in flashy distractions, as most current EdTech companies do?

The question is whether AI tutor has to work exactly the same way as a human one. In the end, what matters is deciding correctly what to do next. In principle, an AI could reach that decision in a different way, and that would be fine as long as the decision is a good one. A useful place to begin is with the outcome: identify the distinct decisions that need to be made.

Based on the tutors’ voiceovers explaining their thought processes in our research videos, here is a rough classification of student responses that can help group decisions about what to do next. This list is likely incomplete.

First, the tutor needs to evaluate whether the student's answer is correct. This is harder than it sounds in subjects such as writing, language learning, and reading, where evaluation requires qualitative judgment and partially correct answers may still be accepted to support continued learning. In math, computers can usually distinguish reliably among correct answers, incorrect answers, and no answer. However, it gets more complicated from here.

The next important question is whether the student's answer should be treated as evidence for updating the student's position in the learning progression model. In many cases, it shouldn't. Identifying these cases is where all current AI tutoring systems lag furthest behind humans. Let me give you practical examples.

Correct answer

A correct answer may seem like straightforward proof that a student has learned something. Well, only if we can rule out other reasons for answering correctly.

Misconception

When you ask a student which fraction is greater, 3/4 or 7/8, they may correctly choose 7/8 only because both numbers are larger in the second fraction. A correct answer like this provides little evidence that the student understands fractions. The same explanation does not apply, for example, to 1/2 and 3/10. Questions that allow students to arrive at the right answer for the wrong reason distort the data about their progress. Some tools try to address this by asking students to explain their reasoning verbally, but evaluating self-explanations reliably is difficult and asking every time disrupts the learning¹¹. A better solution is to design the learning progression model around strong diagnostic (hinge) questions that avoid these situations altogether¹². Unfortunately, current LLMs are not very good at designing such questions. We still rely on humans to identify and list these misconceptions before asking a computer to generate the questions. This is where AI tutors that operate as overlays on someone else’s app, such as Recess, reach their limits. It is impossible to build a great AI tutor on top of poorly designed learning tasks.

Guessing

Sometimes students simply click at random to race through the tasks and get lucky. To reduce the influence of guessing, you can look for streaks of correct answers rather than relying on a single response. Better yet, avoid formats that make guessing easy, such as multiple-choice questions. Virtual manipulatives with many degrees of freedom are particularly effective for this purpose. There are other ways to detect guessed correct answer, if you have control of the UI and have a detailed log of all intermediate interactions. Eventually, the student's luck will run out, resulting in incorrect answers that are easier to identify as guesses, as described later.

Another person answering

This might sound like a minor issue because human tutors obviously know who they are tutoring. Apps do not. I discuss cheating separately later, but there are many other reasons someone might answer correctly on a student's behalf. A teacher might provide the correct answer simply to wrap up a lesson. An older sister might help her brother hoping to teach him. Speech-to-text might also mistakenly capture someone else's voice in the room. I am not aware of any good solution to this problem. There is just not enough data to detect this.

Remembering correct answer

Especially during assessments, you need a large enough sample of students answering the same question to calculate the test’s basic psychometric properties. However, if a student takes the test repeatedly and encounters the same question, they may simply remember the correct answer without understanding the underlying concept. One solution we found is to use isomorphic questions that share the same deep structure but differ in surface features, such as the color of the boxes or the objects being counted. This was possible before generative AI, but it is now much easier to create isomorphic versions of complex tasks, such as word problems. In this respect, computers have surpassed teachers in their ability to quickly generate virtually unlimited practice and assessment tasks from known validated examples.

Cheating

Students who want to game the app, whether to earn more points or skip ahead, can ask a friend or a general-purpose chatbot for the correct answer before responding to the AI tutor. Again, there is no good way to detect this. A student can always place the tablet face down and look up the answer beyond the app's reach. The only possible clue, and an unreliable one, is how long the student takes to answer.

Incorrect answer

Similarly, an incorrect answer does not necessarily prove that a student lacks a particular skill. This mistaken assumption, for example, is a major source of students’ frustration with IXL’s strict SmartScore.

Careless mistake

Even adults sometimes make mistakes in elementary math, not because they lack the skill, but because they do not give the problem enough attention in that moment. These occasional mistakes should have little effect on the student's learning progression. Human tutors can often recognize them based on their experience with the child. An AI tutor might approximate this judgment by analyzing the student's performance history or simply by allowing occasional errors rather than requiring 100% accuracy for mastery. When the AI tutor suspects a careless mistake, it should encourage the student to slow down and focus.

Fatigue

Although errors caused by fatigue may look almost identical to careless mistakes, they require a different response: the student should take a break or stop for the day. Unlike humans, AI tutors do not have non-verbal data that could help them recognize fatigue. We have not figured out how to do this yet, but it may be possible to detect fatigue by tracking recurring streaks of careless mistakes and increasingly slow response times.

Input error

This brings us back to accessibility. Even if they know the correct answer, children may tap the wrong spot, drop objects while dragging them, or accidentally submit an answer by tapping the screen to count objects. Preventing as many input errors as possible requires extensive experience designing apps for children and rigorous testing. For example, we have found that delaying evaluation after the student clicks the Check button works well. This gives the student time to review and correct the answer before submitting it. Still, input errors cannot be eliminated entirely. Someone could always bump into the student at the wrong moment. This is another reason not to require 100% accuracy.

Misunderstanding the task

Sometimes students would know the correct answer but become confused by the task design itself. This is a flaw in the learning activity and should be prevented whenever possible. In practice, however, it is impossible to prevent every source of confusion because students bring assumptions from other apps or pen-and-paper tasks. The challenge is even greater with young children. They may have no experience with similar tasks and need more detailed instructions, yet they may struggle to process long instructions and stop listening. The best solution we have found is to keep the initial instructions brief, identify potential misunderstandings during testing, and teach the AI tutor to offer further explanation only when it detects a specific misunderstanding. In fact, human tutors do the same.

Task drift

Some student responses are completely unrelated to the question. A student might simply say, for example, that they like hamsters. This should obviously not affect the learning progression model and can be detected fairly reliably, but it is difficult to decide how the AI tutor should respond. It is tempting to acknowledge the comment by saying, “That’s great. Hamsters are cute and fluffy!”. This may feel like personalized learning, but it puts the tutor on thin ice. Once children realize that the tutor engages in unrelated conversation, they may repeatedly try to steer the interaction away from learning or pressure it to reveal the correct answer. Current LLMs almost always give in if students persist.

Worse, these exchanges can begin to create an emotional bond. However, because current AI lacks a theory of mind, that bond rests on a fake and often incoherent relationship. Now imagine the child follows up by saying that their hamster died yesterday. What to say next? AI toys faced similar issues. Early, unrestricted versions could gradually drift into disturbing, intimate conversations with children. Later versions, in an attempt to fix that, upset children by responding to “I love you, my teddy” with just “OK.” An AI tutor should not pretend to have a relationship with a child, just as a book does not. It is a learning tool.

Our current solution is to avoid unrestricted open chat and instead classify student verbal responses into pre-generated categories that the tutor can handle appropriately. There can be vast amount of them but anything outside those categories receives a deliberately generic response, such as “Let’s read the instructions again.” This is the only approach we have found to be safe and discourage further steering.

Guessing

Students who rush through a task and click mindlessly will eventually run out of luck and answer incorrectly. Guessing then becomes fairly easy to detect by identifying too many incorrect answers in too little time. These responses should not affect the learning progression model. Instead, the AI tutor should encourage the student to slow down. This is technically straightforward to implement, so I am not sure why most learning apps do not already include it.

Another person answering

It is uncommon for someone else to answer incorrectly on a student’s behalf, but it does happen. A student might accidentally log in to someone else’s account, or a friend might try to help but get the math wrong. More often, the cause is likely technical, such as the speech-to-text module picking up someone else’s voice in the room. In these cases, future improvements in the accuracy and speed of speaker diarization could actually help.

Deliberately wrong answer

Even when students know the correct answer, they may deliberately answer incorrectly to receive easier tasks, get more hints, or eventually prompt the AI tutor to reveal the answer. Sometimes they may simply enjoy a funny animation that appears after an incorrect response, even though it was added in good faith to soften the feeling of failure. This behavior is extremely difficult to detect. Whatever the AI tutor does, students may learn its patterns and exploit them. Thanks to their theory of mind, human tutors are fairly good at recognizing this behavior. Without that capability, AI tutors must rely on historical data and respond somewhat unpredictably. For example, they should not always provide a hint, and they should show the funny animation only occasionally.

Genuine mistake

Only after ruling out other causes of incorrect answers can you begin meaningful error analysis that affects the student’s position in the learning progression model. A genuine mistake may reflect a common misconception, missing vocabulary, or cognitive overload (when a student can complete each required step independently but cannot combine them).

The core question is: What decisions can the AI tutor make based on the learning activity, and how can we map every possible student response to one of those decisions? In practice, this means grouping student responses into distinct mistake types. That is easier said than done.

Consider a learning activity in which the student must find all four multiplication and division equations that represent the situation shown in the picture.

How many possible student answers are there? The input allows 15 character types and up to 33 characters, enough to accommodate incorrect answers such as 10+10+10+10+10+10+10+10+10+10=100. That means 6.93 × 10³⁸ possible answers. And the complexity does not end there. When responding to a mistake, the tutor must also consider which equations the student has already found. For example, the student may have written both multiplication equations but neither division equation. Accounting for 15 possible combinations of completed equations brings the total to 1.04 × 10⁴⁰ cases. Brute force cannot reliably map all of them to meaningful mistake types.

When we asked human tutors to analyze the activity, they identified about 180 types of mistakes that collectively cover all 1.04 × 10⁴⁰ cases. These included invalid equation syntax (vast majority of the covered cases), using the correct numbers with the wrong operation, adding instead of multiplying, finding one multiplication equation but not the other, or finding both multiplication equations but none of the corresponding division equations. Each type requires a different response, such as guiding the student toward the second multiplication equation or transitioning from multiplication to division.

Humans do this by imagining plausible thought processes that could lead to each type of mistake. We do not yet know how to use LLMs to distill 1.04 × 10⁴⁰ possible cases into 180 meaningful categories. But once human tutors have manually identified those categories, generative AI enables another breakthrough. It can translate plain-English descriptions of each mistake type into deterministic rules that classify student interaction logs with 100% accuracy. Once the mistake category is identified and matched with the corresponding pedagogical intent, the AI tutor can generate wording and actions tailored to the current task, including references to specific numbers, objects, colors, and other details.

Most of this analysis does not need to happen in the moment. It can be done in advance, much as human tutors prepare for sessions by anticipating common mistakes and planning how to respond. Repeating the same analysis in real time using reasoning models makes AI tutors unnecessarily slow and inefficient.

That said, human tutors can recognize previously unknown types of mistakes and add them to the mental catalog they draw on in future sessions. We can get close to this process by scanning interaction logs at scale for mistakes the AI tutor did not anticipate. However, human tutors must still identify the underlying causes and teach the AI tutor how to respond.

Sometimes, making a good pedagogical decision requires considering multiple mistakes together and responding to them as a pattern rather than individually. Much as I admire Eedi’s research on misconceptions, from this perspective its practical value is limited. In many cases, different misconceptions are really just symptoms of the same underlying gap and point to the same pedagogical decision: go back and reteach the missing foundational concept. Identifying and addressing the specific misconception itself often does not help much.In these cases, the AI tutor must be able to update the student’s position in the learning progression model and switch to an entirely different learning activity. I discuss this further in the section on the Response generator.

No answer

Students do not answer only correctly or incorrectly. Sometimes they do not answer at all. Human tutors respond to inactivity by trying to determine its cause. AI tutors can examine detailed interaction logs even when the student has not submitted an answer. Unless the student has done literally nothing, these logs provide some basis for classification. However, the limited data available, compared with what human tutors can observe, remains a major challenge.

Still thinking

One entirely legitimate reason for inactivity is that the student is still thinking. Human tutors can usually recognize this, but AI tutors lack the nonverbal cues needed to make that judgment. An AI tutor may know, down to the millisecond, how long the student has been inactive and compare that duration with those of other students working on the same task or with the same student’s performance on similar tasks. Yet this data reveals little with certainty. A particular student may simply need more time for a particular task at a particular moment. This makes it difficult to time responses to all types of inactivity. Respond too soon, and you interrupt the student’s thinking. Respond too late, and the student may become disengaged.

Confusion

Another common reason for inactivity is that the student simply does not know where to start. The task design may be confusing, or the student may lack the skills needed to begin. Both cases indicate a flaw in the app: either the learning activity is poorly designed, or it is inappropriate for the student’s level. Still, a small hint from the AI tutor about where to start can help.

Fear of making a mistake

Some students know the correct answer but take a long time to submit it because they are afraid of making a mistake. If the learning activities are rich enough, the interaction log may reveal correct intermediate steps or even a complete solution that has not yet been submitted. In these cases, the AI tutor can encourage the student to continue working or submit their answer. This approach has drawbacks, which I discuss later in the section on deliberate non-response.

Fatigue

We have already discussed how fatigue can lead to incorrect answers, but it may also result in complete inactivity. This is impossible to detect from a single task. Ideally, the AI tutor would estimate the student’s fatigue level throughout the session and, once it reaches a certain threshold, suggest a break or end the session for the day. Unfortunately, we currently do not know how to measure fatigue reliably.

Avoiding difficult work

Learning requires effort, and some students disengage as soon as a task stops being easy. Helping them persist through these moments is largely a matter of motivation. Even at Alpha School, guides who do not otherwise teach students spend time helping them stay motivated. External incentives can help too. We have had some success with points that children can use to buy animals for a virtual zoo, as well as a flower that continues to bloom and provide nectar for a bee as long as the student learns regularly. The key is not to overdo this and let students spend time on unrelated activities. For example, in Levebee, students cannot interact further with the zoo after spending their points; they simply return to learning.

Ideally, learning becomes a habit that requires little or no motivation. Once brushing your teeth becomes automatic, you no longer need to motivate yourself to do it every day. Schools are good at establishing this kind of routine. Duolingo has achieved it too, but unfortunately by making activities trivial, sacrificing learning quality in the process.

Distraction

There are countless reasons for inactivity that have nothing to do with learning. A loud classmate, a bird outside the window, thirst, or heat can all distract a student. In these cases, AI tutors have few options beyond using sounds or animations to regain the student’s attention. Human tutors can remove the source of the distraction or move around to recapture the student’s attention.

Deliberate non-response

If the tutor predictably provides hints or reassurance after periods of inactivity, students may learn to wait deliberately rather than think through the problem. Even encouraging them not to fear mistakes when they are on the right track can teach them to wait for this sentence before submitting an answer. There is no good solution beyond making the AI tutor somewhat unpredictable. It turns out that human unpredictability is a feature, not a bug.

Summary

The pedagogical decision system remains the least developed building block of an AI tutor. Limited contextual data makes it difficult to reliably classify students’ answers and inactivity. Correct and incorrect answers that do not reflect a student’s actual skill distort the learning progression model, leading the AI tutor to select inappropriate activities or keep the student on the current ones for too long.

One major improvement would be the ability to reliably distinguish answers that should influence the learning progression model from those that should not, even without classifying each response in detail.

That does not mean generative AI has not helped us make progress. We can now reliably detect a vast number of misconceptions and generate responses that align with the pedagogical intent and reference the specific task. For example, if a word problem involves John, Adele, and cookies, the tutor can refer to them throughout the subsequent discussion and related tasks. Previously, this was not possible at scale.

At Levebee, the best way we currently found to account for the uncertainty is to consider a student to have mastered an ultra-granular subskill only when they consistently achieve a success rate above 80% on its isomorphic diagnostic tasks.

The question is how many of the remaining problems must be solved before AI tutors become useful enough for wide implementation, especially when paired with people who excel at the human aspects of tutoring but are not necessarily experts in math instruction. As with previous EdTech, self-motivated students may not need every missing piece to be solved. AI tutors could likely begin helping these students sooner.

Response generator

Once the AI tutor has made a pedagogical decision, it must be able to act on it.

Verbal response

This is the most obvious capability: an AI tutor should be able to speak. Recent advances in text-to-speech (TTS) models have significantly improved speech quality. Earlier voices sounded robotic, which was especially counterproductive when giving feedback on incorrect answers. Modern TTS models can follow prompts for tone and pronunciation, allowing an AI tutor to produce human-like speech in a wider range of situations. For example, it can emphasize important parts of a sentence, slow down when necessary, and use interjections to sound friendlier. The biggest drawback is that these high-quality models cannot yet run locally on a device. As a result, they take longer to generate speech and consume internet bandwidth, which can be problematic in environments with poor Wi-Fi.

Object manipulation

Most existing learning apps only tell students what to do, covering only the “you do” stage of the instructional model “I do, we do, you do.” Human tutors can move among all three stages. They might demonstrate a task, let the student try it, step in when the student gets stuck, complete one step for them, and then ask them to continue.

This kind of interaction has been rare in learning apps because programming it manually was extremely tedious. Generative AI has allowed us to make a breakthrough here as well. As mentioned earlier, Levebee’s learning activities are now defined as state machine configurations. The user interface is simply a declarative layer that follows these configurations, both visually and functionally. This allows the AI to control what happens on the screen by changing the state machine’s textual representation, which the interface automatically responds to. The AI can click, drag objects, highlight text, fill in answers, or perform any other action available to the student.

Complete change of activity

The current paradigm for learning apps is a fixed list of activities that students progress through. We have encountered several drawbacks to this approach. Although our diagnostic assessments place students at the appropriate point in the learning progression model, parents and sometimes teachers skip the assessment or override its placement. They want to assign activities that address the skills students are struggling with in school right now, rather than activities that target the root causes of those struggles. It is difficult to persuade them otherwise. If the class is already working on addition and subtraction with exchange within 100, they want the student to practice exactly that.

With the new state machine approach, we can honor that request while allowing the AI tutor to switch to any other activity in the app whenever needed. For example, if a student is struggling with 54 + 27, the AI tutor can temporarily move to an activity that uses blocks to model 50 + 20 and 4 + 7, then return to the original problem. This is much closer to how human tutors personalize sessions around what each student actually needs.

Non-verbal cues

Digital interfaces can already communicate nonverbally through the arrangement of on-screen elements and the use of animations. However, interpreting these cues requires experience with digital interfaces that young children often lack. They may not yet understand the conventional meanings of colors, icons, and other elements. Some developers have tried to convey nonverbal information through animated characters, such as Microsoft’s Mico avatar. We have not yet tried this approach, and it would be interesting to see research on whether such avatars support learning or just distract students.

Summary

The Response generator is where generative AI has driven the most progress. AI tutors can now produce human-like speech, manipulate objects on the screen, and smoothly switch to another activity and back. They still struggle with nonverbal communication, although several promising approaches are being explored.

Keeping teacher in the loop

Despite the progress made, AI tutors still lag behind human tutors. Until that changes, they will need to work alongside human teachers. They must make teachers’ jobs easier, not harder, and so far that has proven difficult. The reasons for this have been studied before, notably by Martin Tallvid¹³ during an earlier wave of optimism that intelligent tutoring systems would revolutionize education.

One challenge is accountability. When teachers delegate instruction to a computer, school leaders and parents still hold them responsible for student outcomes. If a student continues to struggle with math, the teacher will be asked what they are doing to help. They cannot simply say they do not know because the student was working on a computer. That would be like giving someone a self-driving car, telling them to take their hands off the wheel, and then holding them responsible for every accident.

Teachers must be able to understand and control how an AI tutor works. It needs to act like a colleague who works with the student, clearly explains what happened, and follows the teacher’s instructions. This is why the grade-level and percentile scores used by most EdTech products fall short. Statistical models are rarely explainable at the level of detail teachers need to intervene effectively when a student is still struggling. At any point, teachers need to know what the app is trying to teach and how the student is responding. They cannot delegate their own understanding to AI.

Student insights are one area where we still have plenty of work to do at Levebee. Our new state machine architecture allows us to replay an entire learning session down to every click, much like a video game replay. We already have all the necessary data. The challenge is extracting insights that teachers can act on without overwhelming them with charts and tables. Generative AI might actually help with this by translating these data points into plain-English recommendations similar to those a teaching colleague might provide.

There are other challenges as well. The logistics must be smooth and cannot consume too much valuable instructional time. We have worked hard to streamline the app itself and reduce this friction as much as possible. Even so, schools still need to maintain and charge devices, install updates, and ensure that Wi-Fi works. Compared with pen-and-paper activities, digital tools still carry higher logistical costs. AI tutors must be significantly better than paper to justify that overhead.

The hardest problem identified by decades of research into intelligent tutoring systems may be their effect on classroom relationships. Generative AI has not solved this. When teachers simply ask students to sit in front of computers, students may no longer see them as teachers. I suspect this is one reason Alpha School calls them “guides”.

Conclusion

Can we build human-like AI tutors? Not yet. Can we build far more effective learning apps? Absolutely. I believe it is still worth exploring AI tutors further and keep improving our understanding of their anatomy. When new technology helps us solve some of the remaining problems, we will already know what to build. My best guess is that there will be no silver-bullet technological breakthrough though. We should prepare ourselves to solve hundreds of problems, one at a time.

Even if we never build AI tutors as capable as we hope, working toward that goal can still teach us how to create better learning apps with the technology available today.

About the author

Michal Hudecek is the CEO and co-founder of Levebee, an EdTech company that develops AI-supported diagnostic assessments and personalized mathematics interventions for young learners. He has more than 20 years of experience in building educational tools.
LinkedIn | X