Evidence tiers cheat sheet: how to judge a learning app's claims

US education law sorts evidence into four tiers: Tier 1 "strong" needs a randomised trial, Tier 2 "moderate" a matched comparison, Tier 3 "promising" a correlation with controls, and Tier 4 only a written rationale. Most learning apps sit at Tier 3 or 4. Even strong apps typically show gains of 0.05 to 0.20 standard deviations.

This sheet is a page to keep beside any app's "research" page. It gives the tiers, the scale to read effect sizes on, the independent numbers for products parents ask about, and the red flags. It is written by Peabody (peabody.kids), a free AI-driven adaptive learning platform for kids, free through the end of 2026, which has no efficacy study of its own and says so. AI writes the stories; code decides the rest.

The four tiers

The tier labels come from the Every Student Succeeds Act; Johns Hopkins' Evidence for ESSA rates programs against them independently, and its 2023 standards set the size rules. The What Works Clearinghouse reviews single studies ("meets standards without reservations" is its top mark). A company badge that says "ESSA Tier 3" without a clearinghouse link is the company's own reading.

How big is big

An effect size is the difference between the two groups in standard deviations. Cohen's old rule of thumb (0.2 small, 0.5 medium, 0.8 large) was not built for school programs. Kraft (2020, Educational Researcher) looked at what education trials actually find and offered a scale for programs delivered at scale on standardised tests:

So a learning app that moves a standardised test by 0.10 is doing well. A claim of 0.5 or more on a standardised test, against an active comparison group, would be extraordinary.

Vendor numbers next to independent ones

The research report behind this sheet found the same pattern across the market: every strong vendor number has a weaker independent number beside it.

None of these numbers are bad. +0.10 from a supplement is what "it works" looks like. The point of the table is the ratio: a vendor quasi-experiment at 0.42 and a randomised trial at 0.07 are describing the same product.

Seven red flags

1. "Used as recommended." Results for the slice of students who used the product the most are selection, not effect. One vendor's widely quoted +0.36 applied to the 9% who hit 30 minutes a week (NEPC critique).
2. A test the company wrote. Researcher-made tests close to the practised content show two to three times the effect of standardised tests (Kim et al. 2021, ERIC).
3. No comparison group. A pre-test and post-test with nothing to compare against shows that children got older.
4. A comparison group that did nothing. Against an active control, game effects shrink; Sitzmann (2011) found trainees learned less from games than from comparison instruction that was itself active (Personnel Psychology).
5. Low starters "growing". Children placed low on a short check will score higher on a retest with no learning at all. That is regression to the mean.
6. A percentage with no method. "88% of kids read more" with no study behind it is a survey or a slogan.
7. Satisfaction as outcome. "97% of parents are happy" measures parents, not learning. Wouters et al. (2013) found games were not reliably more motivating than other instruction in the trials they pooled (ERIC).

Learn more elsewhere

• EEF toolkit: mastery learning: a UK research charity rates mastery learning by impact, cost and strength of evidence.
• Stanford SCALE: the evidence base on AI in K-12: a 2026 review: of more than 800 papers, about 20 were strong causal studies, and hint-giving tutors did better than answer-giving chatbots.
• Education Next: two-sigma tutoring, fiction and fact: why Bloom's famous tutoring number is overstated, and what real tutoring programs get.

For about seventy more, checked and sorted by who they're for, see where to learn about AI.

Questions to ask any app

Credentials a small product can earn

The research report lists what is open to a product without a big study. Digital Promise's ESSA Tier 4 design certification reviews a written logic model for $650 and takes about four weeks (certification policies); its Responsibly Designed AI certification checks privacy, security, bias, AI labels and a user override; Common Sense offers a free privacy evaluation (Common Sense Privacy). A Tier 1 trial costs $250,000 to $3 million by consultancy estimates.

Where Peabody stands

Peabody has no efficacy study, no certification and no tier. Its rules are published, each with the studies it draws on and their limits, on the science page, and the practice brain is written up in how Peabody picks the next question. The honest next step is a written logic model, then an in-house comparison of learning mode against Just play on an out-of-game check. Until then, every effect size on this site belongs to someone else's study. Start free and judge the design yourself.

Questions parents ask

What does ESSA Tier 1 mean?

That a well-run randomised trial found a significant positive effect. Evidence for ESSA's "Strong" rating adds size rules, including at least 350 students across more than one site. It is the strongest evidence a learning product can have, and few consumer apps have it.

Is a Tier 3 or Tier 4 app bad?

No. It means nobody has run a fair comparison yet. Many good products sit there, including free ones. What matters is that the company says which tier it is at and doesn't dress a survey up as a trial.

What effect size should I expect from a good learning app?

On a standardised test, about 0.05 to 0.20 standard deviations from a well-run supplement. On a test of exactly what was practised, more. Anything over 0.5 on a standardised test against an active control is rare enough to check twice.

Does Peabody have any evidence?

Not of its own. Peabody's rules follow published findings on spacing, retrieval, difficulty and games, and those findings have their own evidence; whether Peabody's version helps a child more than something else has not been tested.