Knowledge tracing explained: BKT, Elo and deep learning

Knowledge tracing is the part of learning software that turns a stream of right and wrong answers into a guess about what a child knows. The three standard ways to do it are a small Bayesian model per skill (BKT), a chess-style rating for each child and each item (Elo), and a neural network over the whole answer history (DKT). Peabody (peabody.kids), a free AI-driven adaptive learning platform for kids, uses none of them today: readable rules plus per-item speed, for reasons this post lays out along with the test that would make it switch. AI writes the stories; code decides the rest. Peabody is free through the end of 2026.

This is a technical explainer. If you want the shorter version of how adaptivity works in Peabody, start with how AI-driven adaptive reading works.

The problem every model is solving

A child taps an answer. Was it knowledge, a guess, or a slip? One answer cannot tell you. A model's job is to turn many answers, over time, into a number: the probability the child knows the skill, or the probability they get the next question right. That number then drives what comes next.

Three things make this hard for young children. Answers are multiple choice, so guessing is common. Data is thin: a five-year-old might answer thirty questions in a sitting. And children forget between sittings, so a number from last week is stale.

Bayesian Knowledge Tracing: four numbers per skill

BKT comes from Corbett and Anderson (1995, User Modeling and User-Adapted Interaction; the equations are summarised at https://en.wikipedia.org/wiki/Bayesian_knowledge_tracing). Each skill is either known or not, and the model keeps a probability P(L) that it is known. Four parameters per skill:

Each answer runs two steps. First the evidence step, Bayes' rule: after a right answer, P(L) becomes P(L)(1 − S) divided by P(L)(1 − S) + (1 − P(L))G. After a wrong one, it becomes P(L)S divided by P(L)S + (1 − P(L))(1 − G). Then the learning step: the new P(L) plus (1 − P(L)) times T, because the child may have learnt something from the attempt. The prediction for the next answer is P(L)(1 − S) + (1 − P(L))G. Mastery is declared at P(L) ≥ 0.95, a convention rather than a derivation.

One worked update

Take a sight word with P(L0) = 0.3, T = 0.2, S = 0.1, and three answer boxes, so a blind guess is right one time in three: G = 0.33.

The child taps the right box. Evidence: 0.3 × 0.9 = 0.27 on top; 0.27 + 0.7 × 0.33 = 0.501 underneath; P(L) = 0.539. Learning step: 0.539 + 0.461 × 0.2 = 0.631. One right answer moved the belief from 30% to 63%.

Now run the same answer with G = 0.1, the kind of guess rate you get from a typed answer. The evidence step gives 0.27 / 0.34 = 0.794, and the learning step 0.835. Same tap, but the model is far more impressed.

Keep going with three boxes. A second right answer takes P(L) to 0.859, a third to 0.955, over the mastery line. With G = 0.1 the line is crossed on the second answer. And if the second answer is wrong instead, P(L) falls from 0.631 to 0.363 and the climb starts over.

That table is the whole case against plain BKT for tap-box practice: with three options, G sits above Corbett's bound, and one right answer is weak evidence. Baker, Corbett and Aleven (2008) relaxed the bounds to 0.5 (the research report flags this as something to check in the primary paper), but the deeper problem is van de Sande's 2013 finding in JEDM that BKT is not identifiable: several different parameter sets fit the same data equally well (https://jedm.educationaldatamining.org/index.php/JEDM/article/download/35/pdf_27). The fitted numbers cannot be read as facts about the child.

Elo: a rating for the child and a rating for the item

Math Garden, built by Klinkenberg, Straatemeier and van der Maas (2011, Computers and Education), took the chess rating system and applied it every trial to both the child and the item. Pelánek's 2016 paper in the same journal is the engineering reference (https://www.fi.muni.cz/~xpelanek/publications/CAE-elo.pdf).

A child has a rating θ and an item has a difficulty d. The chance of a right answer is 1 / (1 + e^(−(θ − d))), the same shape as a one-parameter Rasch model. After each answer, θ moves by K × (correct − P) and d moves the same amount the other way. A right answer that was expected barely moves anything; a surprise moves both a lot. Both start at 0.

Three refinements matter for a product like Peabody:

• An uncertainty function instead of a constant K. Early work used K = 0.4. Better is to shrink K with the number of answers n: U(n) = a / (1 + b·n), with fitted a = 1 and b = 0.05 (Papoušek et al. 2014; Nižnan et al. 2015). A child's first answer moves them with K = 1; their fiftieth with K = 0.29; their hundredth with K = 0.17. Klinkenberg's version lets uncertainty grow again with days away. Pelánek reports that the fully Bayesian versions (Glicko, TrueSkill) bring no significant advantage for student modelling.
• The k-option formula. With k answer boxes, P(correct) = 1/k + (1 − 1/k) / (1 + e^(−(θ − d))). For three boxes a child exactly matched to an item has P = 0.667, not 0.5. A right answer there moves θ by 0.4 × 0.333 = 0.13; a wrong one moves it by 0.4 × (0 − 0.667) = −0.27. The model knows a miss is twice as informative as a hit when a third of hits are luck.
• Speed. Math Garden's High-Speed-High-Stakes rule scores each answer as (2 × correct − 1)(a·d − a·t), with d the time limit and t the response time: fast and right scores high, fast and wrong scores very negative, slow either way tends to zero. Pelánek's caution is that this is only fair when the child can see the deadline.

Math Garden picks items so the child has about a 75% chance of being right, not the 50% that maximises measurement, because failing half the time is unacceptable in practice software (Hofman et al. 2018, Journal of Intelligence, https://www.mdpi.com/2079-3200/6/1/14/html). Simulations in Pelánek 2016 show Elo with an uncertainty function recovers item difficulties about as well as a full Rasch fit, and does it live, item by item.

Deep Knowledge Tracing: when it actually wins

Piech et al. (2015, NeurIPS) fed the whole sequence of (skill, right or wrong) to an LSTM and reported AUC 0.86 against 0.67 for BKT on the ASSISTments dataset (https://ganguli-gang.stanford.edu/pdf/DeepKnowledgeTracing.pdf). That 25% gain is the number most people have heard. It did not survive.

Khajah, Lindsey and Mozer (2016, EDM, https://arxiv.org/abs/1604.02416) showed the BKT baseline had been badly fit. Properly fit, BKT reaches 0.73 on ASSISTments, and BKT extended with forgetting, student ability and skill discovery matches DKT on average across four datasets: ahead by 0.05 AUC on one, behind by 0.03 on another. Xiong et al. (2016) found duplicated records in the dataset that had inflated DKT. Their conclusion: representation learning is not what drives DKT's results.

Gervet et al. (2020, JEDM, https://jedm.educationaldatamining.org/index.php/JEDM/article/download/451/123) ran nine datasets and asked when deep learning is the best approach. The answer: logistic regression with the right features (item difficulty plus counts of prior successes and failures per skill) leads on small and moderate datasets, because DKT overfits when there are few learners per item. DKT leads only above roughly one million interactions.

Even where DKT wins, the margin is 0.01 to 0.06. The pyKT benchmark (Liu et al. 2022, https://arxiv.org/abs/2206.11460) added a further caution: the same model on the same dataset had reported AUCs from 0.709 to 0.86 across papers because evaluation protocols differed. Under one protocol, attention models (AKT) lead, and simple IRT and PFA models sit within a few points.

Language-model tracing (2024 to 2026) helps with one thing: a brand-new item with no history can get a predicted difficulty from its text, where DKT falls to 0.5 AUC. Zero-shot prompting was "not successful" at tracking knowledge (Neshaei et al. 2024, https://arxiv.org/abs/2403.14661).

What each needs in data

Why Peabody uses rules and speed instead

Peabody's practice brain is ordinary code, documented rule by rule in its guide to how the next question is picked. Each item sits at a level per child: 0 unseen, 1 missed last time, 2 right but hinted or with only two options, 3 right first try, 4 quick and right first try with three options, 5 quick and right on two different days. "Quick" is a fixed threshold per strand: 3 seconds for adding, 2.5 for times tables, 6 for number sense, 5 for word meanings. A known item comes back after 1, 3, 7 days and then a 21-day gap; a miss resets it. A question answered right first try is never asked again, and a miss is asked once more, later in the round.

Set beside the models above, the design choices read like this:

• The unit of evidence is a skill, not one answer. The worked BKT table shows why: with three boxes, a single right answer barely moves a sound model. Level 5 needs quick rights on two separate days, and a rung passes only when 80% of what was seen is solid. A single question retires after one right (until 16 October 2026 it took two), so a lucky tap can retire a question, but it cannot pass a rung on its own. The research report's own recommendation for tap-box practice was two rights per question; Peabody chose not to repeat questions and to count evidence across the skill instead.
• Speed is a second channel. A fitted model sees right or wrong. Peabody also sees whether the answer came in under the threshold, which separates "knows it" from "worked it out", and keeps "fast in games" separate from "known in reading" in what grown-ups see. New items are never timed, and the clock is never shown, which sidesteps Pelánek's caveat on speed scoring.
• Nothing is fitted, so nothing is non-identifiable. The thresholds are constants, and a test suite (the learning invariants) checks that the never-twice constant is 2, review gaps strictly increase and no simulated child gets an empty deck.
• Data is thin and the stakes of a wrong level are low. A child who is cruising gets stretch items until they miss, which works as placement. A deck is about 85% known items, so a model that is slightly wrong about one item changes little.

What would make it switch is the same thing that makes any of these models worth it: data that shows the rules are wrong in a way a model would fix. Concretely, a k-option Elo tracker per strand, with Pelánek's uncertainty function, is the candidate, because it works from the first answer and puts 1/3 into the model instead of around it.

The test that would decide: run both side by side on the same children, the rules choosing the deck and the Elo model only watching, and compare them on the probe cards. Peabody already slips in a measurement-only question about once every 40 answers that moves no level. If the model predicts those probes better than the level table does, by a margin that survives a held-out month, it earns a place. If it does not, the rules stay, because they can be read.

What this doesn't show

• Every number here is about predicting the next answer, not about whether a child learns more. No knowledge-tracing paper in the report measures learning outcomes.
• The benchmark datasets are mostly older students doing maths homework. How these models behave on five-year-olds tapping boxes is not well studied.
• Peabody has no published efficacy study, and this post makes no claim that its rules teach better than a model would. It claims only that they are readable and fenced by tests.
• Peabody does not diagnose reading or learning difficulties, cannot hear a child read, and doesn't measure Lexile.

What a parent can do

Look at what the model is being fed. Watch your child answer ten questions and notice how many were taps before reading. A fast wrong answer is gaming, not knowledge, and Baker and colleagues (2004, CHI) found that behaviour predicted learning loss as strongly as prior knowledge did. Then open the free grown-up view, which shows known, learning and missed in plain words, and ask about one missed item at dinner. You can start free with no account.

Learn more elsewhere

• Duolingo: how we learn how you learn: how one big app predicts when each word is about to be forgotten. A company blog, but it reports real tests.
• ABC of FSRS: a plain explainer of the open scheduling algorithm Anki now uses.
• Stanford SCALE: the evidence base on AI in K-12: a 2026 review: of more than 800 papers, about 20 were strong causal studies, and hint-giving tutors did better than answer-giving chatbots.

For about seventy more, checked and sorted by who they're for, see where to learn about AI.

Questions parents ask

Does Peabody use AI to decide what my child knows?

No. A language model writes the stories and screens them for safety. Levels, review dates and the next question come from fixed rules in ordinary code. A child's level is never decided by a model's opinion.

Is a fitted model always better than rules?

No. Gervet et al. 2020 found simple logistic models beat deep ones below about a million interactions, and even the best gains are 0.06 AUC. For sparse data, an Elo tracker with an uncertainty function is the practical model, and readable rules are a defensible alternative until data shows otherwise.

Why does three-box multiple choice change the maths?

A blind guess is right a third of the time. BKT's original guess bound was 0.3; the k-option Elo formula puts 1/k straight into the prediction. Either you put the guess rate in the model or you ask for many rights before a skill counts, as Peabody does with its 80% pass bar.

What would "mastery" mean in each model?

BKT: P(L) ≥ 0.95. Elo: a rating well above the item's difficulty, with low uncertainty. Peabody: an item at level 5, quick and right first try on two different days, and a rung passed with 80% of it solid. A question being done (right once) is not mastery, only a reason not to ask it again. None of them is a diagnosis.