Findings
What the results show beyond the ranking: profiles, ties, difficulty, topics and cost. Every chart opens with a sentence computed from the data and has its numbers in a table.
Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built
Every model on every test
#The quickest way to see each model's strengths and weaknesses.
Claude Fable 5.2 has the highest score in 5 of 5 tests, but in every test the top score is tied with at least one other model (95% ranges overlap).
Score of each model on each test; Overall counts every test equally. Darker cells are higher, shaded from 50% to 100%. highest in the column; tied with the highest (the 95% ranges overlap). Select a column heading to sort.
| Model | Overall | Lesson exercises | Word forms | Translation | Round trip | Tutor accuracy |
|---|---|---|---|---|---|---|
| Claude Fable 5.2 | 86.1% (highest) | 98.5% (highest) | 94.1% (highest) | 75.1% (highest) | 74.3% (highest) | 88.5% (highest) |
| Gemini 4 Hydrogen | 84.4% (tied with the highest) | 97.3% (tied with the highest) | 91.1% (tied with the highest) | 72.4% (tied with the highest) | 73.0% (tied with the highest) | 88.1% (tied with the highest) |
| GPT-6 Astra | 83.6% (tied with the highest) | 96.6% (tied with the highest) | 92.5% (tied with the highest) | 73.1% (tied with the highest) | 69.3% (tied with the highest) | 86.5% (tied with the highest) |
| Gemini 4 Argon | 81.9% | 98.2% (tied with the highest) | 85.4% | 71.9% (tied with the highest) | 70.0% (tied with the highest) | 84.1% (tied with the highest) |
| DeepSeek V4.1 Flash 1017 | 79.2% | 95.6% (tied with the highest) | 87.0% | 68.8% | 68.5% | 76.1% |
| Kimi K3.1 Pro | 78.4% | 95.8% (tied with the highest) | 82.1% | 69.4% | 65.3% | 79.3% (tied with the highest) |
| GPT-6 Luna | 75.6% | 93.6% | 80.5% | 68.0% | 64.5% | 71.3% |
| Claude Haiku 5.5 | 67.1% | 87.5% | 65.3% | 57.9% | 58.0% | 66.9% |
| Qwen3.8 Flash | 61.6% | 90.6% | 50.6% | 56.9% | 52.2% | 58.0% |
View as table
| Model | Overall | Lesson exercises | Word forms | Translation | Round trip | Tutor accuracy |
|---|---|---|---|---|---|---|
| Claude Fable 5.2 | 86.1% 84.9–87.3% | 98.5% 97.1–99.5% | 94.1% 91.7–96.2% | 75.1% 72.5–77.6% | 74.3% 71.7–77.0% | 88.5% 84.1–92.4% |
| Gemini 4 Hydrogen | 84.4% 83.1–85.6% | 97.3% 95.6–98.8% | 91.1% 88.3–93.6% | 72.4% 69.9–75.0% | 73.0% 70.4–75.6% | 88.1% 83.5–92.2% |
| GPT-6 Astra | 83.6% 82.3–84.9% | 96.6% 94.5–98.3% | 92.5% 89.9–94.8% | 73.1% 70.5–75.7% | 69.3% 66.6–72.1% | 86.5% 82.0–90.7% |
| Gemini 4 Argon | 81.9% 80.5–83.3% | 98.2% 96.9–99.3% | 85.4% 82.1–88.5% | 71.9% 69.4–74.4% | 70.0% 67.3–72.6% | 84.1% 78.9–88.9% |
| DeepSeek V4.1 Flash 1017 | 79.2% 77.7–80.7% | 95.6% 93.6–97.2% | 87.0% 83.9–90.1% | 68.8% 66.2–71.4% | 68.5% 65.7–71.1% | 76.1% 70.6–81.3% |
| Kimi K3.1 Pro | 78.4% 76.9–79.9% | 95.8% 93.9–97.6% | 82.1% 78.4–85.6% | 69.4% 66.9–72.0% | 65.3% 62.5–68.0% | 79.3% 74.3–84.1% |
| GPT-6 Luna | 75.6% 73.9–77.2% | 93.6% 91.1–95.7% | 80.5% 76.9–84.1% | 68.0% 65.4–70.8% | 64.5% 61.7–67.4% | 71.3% 65.4–77.0% |
| Claude Haiku 5.5 | 67.1% 65.3–68.9% | 87.5% 84.4–90.5% | 65.3% 60.9–69.4% | 57.9% 55.1–60.9% | 58.0% 54.8–61.2% | 66.9% 60.9–72.6% |
| Qwen3.8 Flash | 61.6% 59.8–63.4% | 90.6% 87.9–93.1% | 50.6% 46.2–55.1% | 56.9% 54.2–59.9% | 52.2% 48.8–55.4% | 58.0% 51.9–63.7% |
Who is tied with whom
#Scores come from a sample of items, so small gaps can be chance. Models whose 95% ranges overlap are not clearly different.
Claude Fable 5.2 scores highest (86.1%) but is tied with Gemini 4 Hydrogen and GPT-6 Astra: their 95% ranges overlap.
Dot: overall score; line: its 95% range; the axis is zoomed in. The shaded column is the leader’s range: a model whose line reaches it may be tied with the leader. Groups run from the top: each is its highest model plus every model left whose range overlaps that model’s.
View as table
| Model | Score | 95% range | Range overlaps |
|---|---|---|---|
| Claude Fable 5.2 | 86.1% | 84.9–87.3% | Gemini 4 Hydrogen, GPT-6 Astra |
| Gemini 4 Hydrogen | 84.4% | 83.1–85.6% | Claude Fable 5.2, GPT-6 Astra, Gemini 4 Argon |
| GPT-6 Astra | 83.6% | 82.3–84.9% | Claude Fable 5.2, Gemini 4 Hydrogen, Gemini 4 Argon |
| Gemini 4 Argon | 81.9% | 80.5–83.3% | Gemini 4 Hydrogen, GPT-6 Astra, DeepSeek V4.1 Flash 1017 |
| DeepSeek V4.1 Flash 1017 | 79.2% | 77.7–80.7% | Gemini 4 Argon, Kimi K3.1 Pro |
| Kimi K3.1 Pro | 78.4% | 76.9–79.9% | DeepSeek V4.1 Flash 1017, GPT-6 Luna |
| GPT-6 Luna | 75.6% | 73.9–77.2% | Kimi K3.1 Pro |
| Claude Haiku 5.5 | 67.1% | 65.3–68.9% | none |
| Qwen3.8 Flash | 61.6% | 59.8–63.4% | none |
Scores by CEFR level
#Items carry the CEFR level of the course content they come from, from A1 (beginner) up. Do models drop as items get harder?
Lesson exercises
3 of 9 models score lower at B2 than at A1, but none of these drops is clear: the 95% ranges overlap.
Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.
View as table
| Model | A1 n=125 | A2 n=135 | B1 n=73 | B2 n=24 | C1 n=3 |
|---|---|---|---|---|---|
| Claude Fable 5.2 | 99.7% 99.2–100.0% | 97.8% 95.3–99.8% | 97.3% 93.2–100.0% | 100.0% 100.0–100.0% | too few items |
| Gemini 4 Hydrogen | 98.7% 96.5–100.0% | 95.6% 91.9–98.5% | 97.3% 93.6–100.0% | 100.0% 100.0–100.0% | too few items |
| GPT-6 Astra | 97.1% 93.9–99.5% | 94.8% 91.1–98.5% | 97.7% 94.1–100.0% | 100.0% 100.0–100.0% | too few items |
| Gemini 4 Argon | 100.0% 100.0–100.0% | 96.8% 94.3–99.0% | 97.3% 93.2–100.0% | 100.0% 100.0–100.0% | too few items |
| DeepSeek V4.1 Flash 1017 | 97.3% 94.7–99.5% | 93.6% 89.9–97.0% | 95.9% 91.8–99.1% | 95.8% 87.5–100.0% | too few items |
| Kimi K3.1 Pro | 97.3% 94.4–99.5% | 93.8% 89.9–97.3% | 96.3% 91.8–99.5% | 97.2% 91.7–100.0% | too few items |
| GPT-6 Luna | 95.5% 91.7–98.1% | 90.9% 86.2–95.3% | 95.0% 90.0–99.1% | 94.4% 86.1–100.0% | too few items |
| Claude Haiku 5.5 | 88.5% 83.2–93.3% | 86.7% 81.2–91.6% | 86.3% 79.0–92.7% | 88.9% 75.0–100.0% | too few items |
| Qwen3.8 Flash | 90.1% 85.1–94.7% | 89.4% 84.7–93.8% | 92.7% 87.2–97.3% | 91.7% 80.6–100.0% | too few items |
Word forms
5 of 9 models score lower at B1 than at A1; the drop is clear (95% ranges do not overlap) for 3 of them.
Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.
View as table
| Model | A1 n=134 | A2 n=140 | B1 n=39 | B2 n=16 |
|---|---|---|---|---|
| Claude Fable 5.2 | 92.8% 88.3–96.8% | 92.9% 88.6–96.4% | 98.3% 95.7–100.0% | too few items |
| Gemini 4 Hydrogen | 92.0% 87.6–96.0% | 88.6% 83.8–92.9% | 94.0% 87.2–99.1% | too few items |
| GPT-6 Astra | 92.3% 87.8–96.3% | 90.7% 86.0–95.0% | 94.0% 87.2–98.3% | too few items |
| Gemini 4 Argon | 84.8% 79.1–90.0% | 85.5% 80.0–90.5% | 79.5% 67.5–90.6% | too few items |
| DeepSeek V4.1 Flash 1017 | 89.3% 84.3–94.0% | 82.9% 76.9–88.3% | 92.3% 85.5–97.4% | too few items |
| Kimi K3.1 Pro | 85.3% 79.6–90.3% | 80.7% 74.5–86.4% | 68.4% 55.6–80.3% | too few items |
| GPT-6 Luna | 86.3% 80.8–91.3% | 75.7% 68.8–81.9% | 67.5% 55.6–79.5% | too few items |
| Claude Haiku 5.5 | 71.1% 63.9–77.6% | 61.9% 55.0–68.8% | 45.3% 32.5–59.0% | too few items |
| Qwen3.8 Flash | 63.2% 56.2–69.9% | 45.5% 38.8–52.9% | 13.7% 7.7–20.5% | too few items |
Translation
All 9 models score lower at B2 than at A1, but none of these drops is clear: the 95% ranges overlap.
Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.
View as table
| Model | A1 n=112 | A2 n=91 | B1 n=47 | B2 n=30 | C1 n=14 | C2 n=6 |
|---|---|---|---|---|---|---|
| Claude Fable 5.2 | 79.2% 74.5–83.8% | 71.8% 66.6–77.1% | 67.3% 60.5–73.6% | 68.7% 59.0–78.0% | too few items | too few items |
| Gemini 4 Hydrogen | 76.5% 71.6–81.3% | 69.2% 64.1–74.0% | 64.9% 57.6–71.7% | 66.1% 56.4–75.0% | too few items | too few items |
| GPT-6 Astra | 76.6% 71.3–81.9% | 70.4% 65.4–75.3% | 65.9% 58.8–72.4% | 65.1% 55.4–74.6% | too few items | too few items |
| Gemini 4 Argon | 74.5% 69.5–79.3% | 68.0% 62.9–73.1% | 65.0% 58.5–71.7% | 66.9% 57.6–75.9% | too few items | too few items |
| DeepSeek V4.1 Flash 1017 | 72.9% 67.5–77.9% | 65.0% 59.7–70.0% | 64.6% 57.6–71.4% | 65.1% 55.0–74.3% | too few items | too few items |
| Kimi K3.1 Pro | 72.8% 67.6–77.7% | 66.7% 61.6–71.7% | 65.3% 58.3–72.3% | 63.8% 54.3–72.8% | too few items | too few items |
| GPT-6 Luna | 72.8% 67.6–78.2% | 62.0% 56.6–67.3% | 63.2% 56.5–69.8% | 62.9% 53.2–71.9% | too few items | too few items |
| Claude Haiku 5.5 | 60.3% 54.7–66.0% | 50.9% 45.4–56.6% | 55.3% 47.7–63.1% | 54.2% 43.2–64.3% | too few items | too few items |
| Qwen3.8 Flash | 61.6% 56.0–67.3% | 52.5% 46.9–58.0% | 52.5% 44.5–61.0% | 48.9% 37.4–58.9% | too few items | too few items |
Round trip
1 of 9 models score lower at B2 than at A1, but none of these drops is clear: the 95% ranges overlap.
Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.
View as table
| Model | A1 n=25 | A2 n=88 | B1 n=79 | B2 n=27 | C1 n=15 | C2 n=6 |
|---|---|---|---|---|---|---|
| Claude Fable 5.2 | 72.2% 63.0–81.4% | 77.2% 72.7–81.6% | 69.7% 65.5–73.6% | 72.8% 65.3–80.3% | too few items | too few items |
| Gemini 4 Hydrogen | 69.2% 60.4–78.0% | 77.0% 72.6–81.4% | 67.4% 63.1–71.5% | 70.9% 63.7–77.8% | too few items | too few items |
| GPT-6 Astra | 64.0% 55.8–72.5% | 73.7% 69.0–78.5% | 63.4% 58.5–67.9% | 68.7% 62.0–76.0% | too few items | too few items |
| Gemini 4 Argon | 68.2% 59.4–77.2% | 69.9% 65.5–74.5% | 67.0% 62.6–71.5% | 69.0% 61.4–76.6% | too few items | too few items |
| DeepSeek V4.1 Flash 1017 | 64.4% 57.3–71.8% | 70.5% 65.9–75.2% | 65.1% 60.3–69.9% | 68.1% 61.7–75.1% | too few items | too few items |
| Kimi K3.1 Pro | 66.8% 58.6–75.4% | 66.6% 61.9–71.2% | 59.1% 54.2–64.0% | 67.5% 60.5–74.4% | too few items | too few items |
| GPT-6 Luna | 64.7% 56.2–73.5% | 66.6% 61.8–71.5% | 57.4% 52.4–62.5% | 67.0% 59.5–74.5% | too few items | too few items |
| Claude Haiku 5.5 | 56.6% 46.3–67.3% | 59.0% 53.9–64.4% | 52.5% 47.5–57.6% | 57.5% 48.3–67.4% | too few items | too few items |
| Qwen3.8 Flash | 50.4% 39.1–61.8% | 54.4% 49.3–59.6% | 48.0% 42.5–53.8% | 44.8% 34.2–55.9% | too few items | too few items |
Tutor accuracy
All 9 models score lower at B2 than at A1; the drop is clear (95% ranges do not overlap) for 4 of them.
Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.
View as table
| Model | A1 n=22 | A2 n=27 | B1 n=42 | B2 n=23 | C1 n=10 | C2 n=2 |
|---|---|---|---|---|---|---|
| Claude Fable 5.2 | 97.0% 90.9–100.0% | 92.6% 82.7–100.0% | 84.1% 73.0–92.9% | 84.1% 69.6–97.1% | too few items | too few items |
| Gemini 4 Hydrogen | 97.0% 90.9–100.0% | 88.9% 76.5–97.6% | 83.3% 72.2–92.9% | 88.4% 73.9–100.0% | too few items | too few items |
| GPT-6 Astra | 100.0% 100.0–100.0% | 85.2% 71.6–96.3% | 85.7% 75.4–93.7% | 76.8% 59.4–92.8% | too few items | too few items |
| Gemini 4 Argon | 90.9% 80.3–100.0% | 87.7% 75.3–97.5% | 77.0% 64.3–88.1% | 73.9% 56.5–91.3% | too few items | too few items |
| DeepSeek V4.1 Flash 1017 | 93.9% 86.4–100.0% | 77.8% 64.2–88.9% | 64.3% 50.8–76.2% | 69.6% 53.6–84.1% | too few items | too few items |
| Kimi K3.1 Pro | 97.0% 92.4–100.0% | 74.1% 60.5–86.4% | 73.8% 61.9–84.9% | 75.4% 60.9–88.4% | too few items | too few items |
| GPT-6 Luna | 84.8% 71.2–97.0% | 74.1% 60.5–85.2% | 61.1% 48.4–73.8% | 65.2% 46.4–82.6% | too few items | too few items |
| Claude Haiku 5.5 | 75.8% 60.6–89.4% | 65.4% 49.4–81.5% | 61.1% 48.4–73.0% | 56.5% 37.7–73.9% | too few items | too few items |
| Qwen3.8 Flash | 78.8% 66.7–90.9% | 60.5% 45.7–74.1% | 54.8% 42.1–66.7% | 33.3% 18.8–47.8% | too few items | too few items |
Hardest and easiest topics
#Each item records its grammar topic, error type or course category. Which are hardest across models?
Word forms
Averaged over 9 models, the easiest grammar topic is article gender mutation (93.5%, 31 items) and the hardest is verb conditional (72.5%, 23 items); the gap is clear (95% ranges do not overlap) for 4 of 9 models.
Bar: average over models; ticks: single models. Shows the 5 easiest and 5 hardest of 11 grammar topics with 20 or more items; 10 smaller ones are left out.
View as table
| Grammar topic | Items | Average | Lowest model | Highest model |
|---|---|---|---|---|
| article gender mutation | 31 | 93.5% | Qwen3.8 Flash (77.4%) | GPT-6 Astra (100.0%) |
| copula | 31 | 86.9% | Claude Haiku 5.5 (67.7%) | Claude Fable 5.2 (96.8%) |
| number cardinal | 30 | 84.8% | Qwen3.8 Flash (63.3%) | Gemini 4 Hydrogen (96.7%) |
| verb present | 23 | 84.5% | Qwen3.8 Flash (49.3%) | GPT-6 Astra (97.1%) |
| verb imperative | 27 | 82.6% | Qwen3.8 Flash (53.1%) | GPT-6 Astra (93.8%) |
| prepositional pronoun | 28 | 82.3% | Qwen3.8 Flash (44.0%) | GPT-6 Astra (97.6%) |
| number ordinal | 30 | 81.2% | Qwen3.8 Flash (57.8%) | Gemini 4 Hydrogen (92.2%) |
| verb future | 23 | 77.0% | Qwen3.8 Flash (39.1%) | Gemini 4 Hydrogen (94.2%) |
| noun plural | 30 | 73.6% | Qwen3.8 Flash (53.3%) | Claude Fable 5.2 (85.6%) |
| verb past | 22 | 72.9% | Qwen3.8 Flash (34.8%) | Claude Fable 5.2 (100.0%) |
| verb conditional | 23 | 72.5% | Qwen3.8 Flash (14.5%) | Claude Fable 5.2 (98.6%) |
Lesson exercises
Averaged over 9 models, the easiest course category is recap (96.1%, 37 items) and the hardest is grammar (92.2%, 21 items); for every model the two 95% ranges overlap, so the gap is not clear.
Bar: average over models; ticks: single models. Shows every course category with 20 or more items; 70 smaller ones are left out.
View as table
| Course category | Items | Average | Lowest model | Highest model |
|---|---|---|---|---|
| recap | 37 | 96.1% | Claude Haiku 5.5 (88.3%) | GPT-6 Astra (100.0%) |
| guide | 32 | 95.4% | Claude Haiku 5.5 (82.3%) | Gemini 4 Hydrogen (100.0%) |
| unit focus | 51 | 92.3% | Claude Haiku 5.5 (82.4%) | Gemini 4 Argon (99.3%) |
| grammar | 21 | 92.2% | Claude Haiku 5.5 (79.4%) | DeepSeek V4.1 Flash 1017 (96.8%) |
Tutor accuracy
Averaged over 9 models, the easiest course category is guide (82.4%, 20 items) and the hardest is grammar (79.0%, 42 items); for every model the two 95% ranges overlap, so the gap is not clear.
Bar: average over models; ticks: single models. Shows every course category with 20 or more items; 25 smaller ones are left out.
View as table
| Course category | Items | Average | Lowest model | Highest model |
|---|---|---|---|---|
| guide | 20 | 82.4% | Claude Haiku 5.5 (70.0%) | Claude Fable 5.2 (95.0%) |
| no error | 30 | 79.6% | Qwen3.8 Flash (48.9%) | Claude Fable 5.2 (91.1%) |
| grammar | 42 | 79.0% | Claude Haiku 5.5 (61.1%) | Claude Fable 5.2 (91.3%) |
Topic labels come from the item sources. Where results include a joint breakdown, grammar topic by language shows how these averages differ by language.
Grammar topic by language
#Where each topic is hardest, averaged over the models with a recorded topic–language breakdown.
Lesson exercises
The lowest observed model-average score is unit focus for Manx (82.9%); there are too few fully represented topics to compare languages.
Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.
| Topic | Scottish Gaelic | Manx | Breton | Cornish |
|---|---|---|---|---|
| grammar | — | — | 87.7% (hardest in this column) | — |
| unit focus | 93.7% (hardest in this column) | 82.9% (hardest in this column) | — | 97.5% |
| narrative grammar | — | 91.9% | — | — |
| recap | — | 97.8% | — | 91.7% (hardest in this column) |
| guide | — | — | — | 96.5% |
| grammar point | 97.4% | — | — | — |
| chapter overview | — | 98.7% | — | — |
Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.
View as table
| Topic | Scottish Gaelic | Manx | Breton | Cornish |
|---|---|---|---|---|
| grammar | — | — | 87.7%; 13 items per model; 9 models | — |
| unit focus | 93.7%; 10 items per model; 9 models | 82.9%; 16 items per model; 9 models | — | 97.5%; 22 items per model; 9 models |
| narrative grammar | — | 91.9%; 5 items per model; 9 models | — | — |
| recap | — | 97.8%; 15 items per model; 9 models | — | 91.7%; 12 items per model; 9 models |
| guide | — | — | — | 96.5%; 23 items per model; 9 models |
| grammar point | 97.4%; 10 items per model; 9 models | — | — | — |
| chapter overview | — | 98.7%; 11 items per model; 9 models | — | — |
Word forms
The lowest observed model-average score is verb past for Cornish (40.0%); Manx is lowest on 3 of 5 topics with every language represented (point estimates, not clear differences).
Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.
| Topic | Welsh | Scottish Gaelic | Manx | Breton | Cornish |
|---|---|---|---|---|---|
| verb past | — | — | 88.1% | — | 40.0% (hardest in this column) |
| verb future | — | — | 77.0% | 74.1% | 51.9% |
| verb conditional | 85.2% | — | 65.2% | 68.9% (hardest in this column) | — |
| verb imperfect | 85.9% | — | — | — | 61.5% |
| noun plural | 85.9% | 96.3% | 44.4% (hardest in this column) | 75.6% | 72.2% |
| verb imperative | 79.3% (hardest in this column) | — | 74.8% | — | 71.9% |
| noun genitive | — | 77.0% | — | — | — |
| prepositional pronoun | 88.1% | — | 77.0% | 74.1% | 76.3% |
| number ordinal | 97.8% | 89.6% | 49.6% | 94.1% | 71.0% |
| number cardinal | 97.0% | 88.9% | 62.2% | 88.1% | 77.2% |
| verb pluperfect | 83.0% | — | — | — | — |
| verb preterite | 83.7% | — | — | — | — |
| copula | 94.1% | 47.4% (hardest in this column) | 87.0% | 99.3% | 95.7% |
| verb present | 94.8% | — | 94.8% | — | 66.7% |
| article gender mutation | 100.0% | 91.9% | 93.8% | 83.0% | 96.3% |
| number personal | — | 94.1% | — | — | — |
Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.
View as table
| Topic | Welsh | Scottish Gaelic | Manx | Breton | Cornish |
|---|---|---|---|---|---|
| verb past | — | — | 88.1%; 5 items per model; 9 models | — | 40.0%; 5 items per model; 9 models |
| verb future | — | — | 77.0%; 5 items per model; 9 models | 74.1%; 5 items per model; 9 models | 51.9%; 5 items per model; 9 models |
| verb conditional | 85.2%; 5 items per model; 9 models | — | 65.2%; 5 items per model; 9 models | 68.9%; 5 items per model; 9 models | — |
| verb imperfect | 85.9%; 5 items per model; 9 models | — | — | — | 61.5%; 5 items per model; 9 models |
| noun plural | 85.9%; 5 items per model; 9 models | 96.3%; 5 items per model; 9 models | 44.4%; 5 items per model; 9 models | 75.6%; 5 items per model; 9 models | 72.2%; 6 items per model; 9 models |
| verb imperative | 79.3%; 5 items per model; 9 models | — | 74.8%; 5 items per model; 9 models | — | 71.9%; 5 items per model; 9 models |
| noun genitive | — | 77.0%; 5 items per model; 9 models | — | — | — |
| prepositional pronoun | 88.1%; 5 items per model; 9 models | — | 77.0%; 5 items per model; 9 models | 74.1%; 5 items per model; 9 models | 76.3%; 5 items per model; 9 models |
| number ordinal | 97.8%; 5 items per model; 9 models | 89.6%; 5 items per model; 9 models | 49.6%; 5 items per model; 9 models | 94.1%; 5 items per model; 9 models | 71.0%; 6 items per model; 9 models |
| number cardinal | 97.0%; 5 items per model; 9 models | 88.9%; 5 items per model; 9 models | 62.2%; 5 items per model; 9 models | 88.1%; 5 items per model; 9 models | 77.2%; 6 items per model; 9 models |
| verb pluperfect | 83.0%; 5 items per model; 9 models | — | — | — | — |
| verb preterite | 83.7%; 5 items per model; 9 models | — | — | — | — |
| copula | 94.1%; 5 items per model; 9 models | 47.4%; 5 items per model; 9 models | 87.0%; 6 items per model; 9 models | 99.3%; 5 items per model; 9 models | 95.7%; 6 items per model; 9 models |
| verb present | 94.8%; 5 items per model; 9 models | — | 94.8%; 5 items per model; 9 models | — | 66.7%; 5 items per model; 9 models |
| article gender mutation | 100.0%; 5 items per model; 9 models | 91.9%; 5 items per model; 9 models | 93.8%; 6 items per model; 9 models | 83.0%; 5 items per model; 9 models | 96.3%; 6 items per model; 9 models |
| number personal | — | 94.1%; 5 items per model; 9 models | — | — | — |
Translation
The lowest observed model-average score is to english for Manx (42.7%); Manx is lowest on 2 of 2 topics with every language represented (point estimates, not clear differences).
Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.
| Topic | Irish | Welsh | Scottish Gaelic | Manx | Breton | Cornish |
|---|---|---|---|---|---|---|
| from english | 75.2% | 73.3% (hardest in this column) | 67.3% (hardest in this column) | 43.3% | 58.5% | 61.7% (hardest in this column) |
| legal or administrative | 70.6% (hardest in this column) | 84.1% | — | — | 57.0% (hardest in this column) | — |
| to english | 84.9% | 77.4% | 84.3% | 42.7% (hardest in this column) | 80.9% | 77.5% |
Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.
View as table
| Topic | Irish | Welsh | Scottish Gaelic | Manx | Breton | Cornish |
|---|---|---|---|---|---|---|
| from english | 75.2%; 20 items per model; 9 models | 73.3%; 20 items per model; 9 models | 67.3%; 30 items per model; 9 models | 43.3%; 30 items per model; 9 models | 58.5%; 20 items per model; 9 models | 61.7%; 30 items per model; 9 models |
| legal or administrative | 70.6%; 20 items per model; 9 models | 84.1%; 20 items per model; 9 models | — | — | 57.0%; 20 items per model; 9 models | — |
| to english | 84.9%; 20 items per model; 9 models | 77.4%; 20 items per model; 9 models | 84.3%; 30 items per model; 9 models | 42.7%; 30 items per model; 9 models | 80.9%; 20 items per model; 9 models | 77.5%; 30 items per model; 9 models |
Round trip
The lowest observed model-average score is round trip for Manx (49.9%); Manx is lowest on 1 of 1 topics with every language represented (point estimates, not clear differences).
Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.
| Topic | Irish | Welsh | Scottish Gaelic | Manx | Breton | Cornish |
|---|---|---|---|---|---|---|
| round trip | 76.7% (hardest in this column) | 80.5% (hardest in this column) | 72.2% (hardest in this column) | 49.9% (hardest in this column) | 58.7% (hardest in this column) | 58.7% (hardest in this column) |
Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.
View as table
| Topic | Irish | Welsh | Scottish Gaelic | Manx | Breton | Cornish |
|---|---|---|---|---|---|---|
| round trip | 76.7%; 40 items per model; 9 models | 80.5%; 40 items per model; 9 models | 72.2%; 40 items per model; 9 models | 49.9%; 40 items per model; 9 models | 58.7%; 40 items per model; 9 models | 58.7%; 40 items per model; 9 models |
Tutor accuracy
The lowest observed model-average score is application for Scottish Gaelic (57.9%); Scottish Gaelic is lowest on 1 of 1 topics with every language represented (point estimates, not clear differences).
Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.
| Topic | Irish | Welsh | Scottish Gaelic |
|---|---|---|---|
| application | — | — | 57.9% (hardest in this column) |
| grammar point | — | — | 70.1% |
| vocabulary | 71.3% (hardest in this column) | — | — |
| grammar | 80.9% | 78.2% (hardest in this column) | — |
| no error | 79.6% | 80.7% | 78.5% |
| guide | 80.6% | 85.7% | — |
| unit focus | — | — | 85.2% |
| orthography doubled letter | — | 96.3% | — |
Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.
View as table
| Topic | Irish | Welsh | Scottish Gaelic |
|---|---|---|---|
| application | — | — | 57.9%; 11 items per model; 9 models |
| grammar point | — | — | 70.1%; 12 items per model; 9 models |
| vocabulary | 71.3%; 12 items per model; 9 models | — | — |
| grammar | 80.9%; 13 items per model; 9 models | 78.2%; 29 items per model; 9 models | — |
| no error | 79.6%; 10 items per model; 9 models | 80.7%; 10 items per model; 9 models | 78.5%; 10 items per model; 9 models |
| guide | 80.6%; 13 items per model; 9 models | 85.7%; 7 items per model; 9 models | — |
| unit focus | — | — | 85.2%; 6 items per model; 9 models |
| orthography doubled letter | — | 96.3%; 7 items per model; 9 models | — |
Translation directions
#Translation items go into English or out of English, or come from legal and administrative parallel text.
Translation
All 9 models score higher translating into English than out of English; the gap is clear (95% ranges do not overlap) for 7 of them.
chrF++ score by direction for each model; the axis is zoomed in. Right: into English minus out of English, in percentage points. The legal and administrative items are published parallel text, in either direction; Breton is paired with French there.
- Into English 150 items
- Out of English 150 items
- Legal and administrative corpus 60 items
View as table
| Model | Into English n=150 | Out of English n=150 | Legal and administrative corpus n=60 |
|---|---|---|---|
| Claude Fable 5.2 | 77.6% 73.5–81.6% | 71.4% 67.3–75.4% | 77.9% 73.0–82.7% |
| Gemini 4 Hydrogen | 75.1% 71.2–78.9% | 68.6% 64.4–72.8% | 75.3% 70.8–80.0% |
| GPT-6 Astra | 78.0% 73.8–82.1% | 66.9% 62.7–71.0% | 76.3% 71.7–80.8% |
| Gemini 4 Argon | 75.3% 71.3–79.2% | 66.3% 62.2–70.1% | 77.4% 72.9–81.9% |
| DeepSeek V4.1 Flash 1017 | 74.0% 69.9–78.0% | 63.7% 59.5–67.8% | 68.3% 63.5–73.2% |
| Kimi K3.1 Pro | 75.8% 71.6–79.7% | 62.8% 58.8–66.6% | 69.9% 64.8–74.9% |
| GPT-6 Luna | 75.0% 70.7–79.2% | 60.4% 56.4–64.4% | 69.6% 64.8–74.3% |
| Claude Haiku 5.5 | 62.3% 57.5–67.2% | 51.7% 47.5–55.9% | 62.3% 56.8–67.6% |
| Qwen3.8 Flash | 66.7% 62.0–71.4% | 46.7% 42.5–50.8% | 58.2% 53.7–62.9% |
Tutor accuracy step by step
#A tutor item counts only when three rule-marked checks all pass on the same item. Which check fails most?
Tutor accuracy
Averaged over 9 models, the weakest rule-marked step is ‘fixed it’ (77.7%) and the strongest is ‘changed nothing else’ (84.7%); the judge-marked explanation passes 69.7% on average for the 7 models with a comparable explanation score.
Share of items that pass each check, over every ask. The score needs all three rule checks on the same item. The explanation (hatched) is marked by two judge models from other families: it is secondary, not human-validated and not part of the score.
| Model | Found the error | Fixed it | Changed nothing else | All three: the score | Explanation judge-marked |
|---|---|---|---|---|---|
| Claude Fable 5.2 | 88.5% | 88.5% | 93.0% | 88.5% | 78.5% |
| Gemini 4 Hydrogen | 88.3% | 88.1% | 91.3% | 88.1% | 76.4% |
| GPT-6 Astra | 86.7% | 86.5% | 89.4% | 86.5% | 69.9% |
| Gemini 4 Argon | 85.0% | 84.1% | 89.1% | 84.1% | 75.9% |
| Kimi K3.1 Pro | 79.6% | 79.3% | 88.0% | 79.3% | 70.4% |
| DeepSeek V4.1 Flash 1017 | 76.1% | 76.3% | 88.1% | 76.1% | not scored: same family as a judge |
| GPT-6 Luna | 72.2% | 71.5% | 82.6% | 71.3% | 57.4% |
| Claude Haiku 5.5 | 68.1% | 67.0% | 76.5% | 66.9% | 59.6% |
| Qwen3.8 Flash | 58.9% | 58.1% | 64.3% | 58.0% | not scored: same family as a judge |
View as table
| Model | Score and 95% range | Explanation and 95% range | Judges agree |
|---|---|---|---|
| Claude Fable 5.2 | 88.5% 84.1–92.4% | 78.5% 72.9–84.1% | 93.5% |
| Gemini 4 Hydrogen | 88.1% 83.5–92.2% | 76.4% 70.4–82.0% | 95.1% |
| GPT-6 Astra | 86.5% 82.0–90.7% | 69.9% 63.5–76.2% | 91.0% |
| Gemini 4 Argon | 84.1% 78.9–88.9% | 75.9% 70.1–81.7% | 95.2% |
| Kimi K3.1 Pro | 79.3% 74.3–84.1% | 70.4% 64.7–76.0% | 93.4% |
| DeepSeek V4.1 Flash 1017 | 76.1% 70.6–81.3% | n/a | n/a |
| GPT-6 Luna | 71.3% 65.4–77.0% | 57.4% 50.6–64.2% | 93.2% |
| Claude Haiku 5.5 | 66.9% 60.9–72.6% | 59.6% 53.1–65.8% | 95.2% |
| Qwen3.8 Flash | 58.0% 51.9–63.7% | n/a | n/a |
Evidence check
#Each gold answer has an evidence label: cross-checked against a dictionary, from the course alone, or marked unverified. Does weaker gold change the picture?
Labels, from strongest to weakest gold: cross-checked against a dictionary (BÚNaMo or Kaikki), from a published parallel corpus, from the course alone, or marked unverified or built by a rule. What each label means.
Round trip
Averaged over 9 models, the highest-scoring evidence group is ‘Course, form listed in Kaikki’ (75.0%, 64 items) and the lowest-scoring is ‘Course, marked unverified’ (58.7%, 40 items); the gap is clear (95% ranges do not overlap) for 5 of 9 models.
Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong.
View as table
| Evidence | Items | Average | Lowest model | Highest model |
|---|---|---|---|---|
| Course, form listed in Kaikki | 64 | 75.0% | Qwen3.8 Flash (67.5%) | Gemini 4 Argon (79.7%) |
| Course only | 136 | 64.1% | Qwen3.8 Flash (51.1%) | Claude Fable 5.2 (72.4%) |
| Course, marked unverified | 40 | 58.7% | Qwen3.8 Flash (31.2%) | Claude Fable 5.2 (72.9%) |
Lesson exercises
Averaged over 9 models, the highest-scoring evidence group is ‘Course only’ (95.1%, 257 items) and the lowest-scoring is ‘Course, form listed in Kaikki’ (93.8%, 96 items); for every model the two 95% ranges overlap, so the gap is not clear.
Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong. Left out, with fewer than 20 items: Course, form listed in BÚNaMo.
View as table
| Evidence | Items | Average | Lowest model | Highest model |
|---|---|---|---|---|
| Course only | 257 | 95.1% | Claude Haiku 5.5 (87.8%) | Claude Fable 5.2 (98.6%) |
| Course, form listed in Kaikki | 96 | 93.8% | Claude Haiku 5.5 (85.8%) | Gemini 4 Hydrogen (98.6%) |
Word forms
Averaged over 9 models, the highest-scoring evidence group is ‘Course only’ (82.8%, 173 items) and the lowest-scoring is ‘Course, form listed in Kaikki’ (79.0%, 176 items); for every model the two 95% ranges overlap, so the gap is not clear.
Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong. Left out, with fewer than 20 items: Course, form listed in BÚNaMo.
View as table
| Evidence | Items | Average | Lowest model | Highest model |
|---|---|---|---|---|
| Course only | 173 | 82.8% | Qwen3.8 Flash (54.3%) | GPT-6 Astra (95.2%) |
| Course, form listed in Kaikki | 176 | 79.0% | Qwen3.8 Flash (46.8%) | Claude Fable 5.2 (93.4%) |
Translation
Averaged over 9 models, the highest-scoring evidence group is ‘Parallel corpus’ (70.6%, 60 items) and the lowest-scoring is ‘Course only’ (66.7%, 196 items); the gap is clear (95% ranges do not overlap) for 1 of 9 models.
Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong.
View as table
| Evidence | Items | Average | Lowest model | Highest model |
|---|---|---|---|---|
| Parallel corpus | 60 | 70.6% | Qwen3.8 Flash (58.2%) | Claude Fable 5.2 (77.9%) |
| Course, form listed in Kaikki | 44 | 69.7% | Qwen3.8 Flash (58.2%) | Gemini 4 Argon (75.3%) |
| Course, marked unverified | 60 | 69.6% | Qwen3.8 Flash (48.3%) | Claude Fable 5.2 (83.0%) |
| Course only | 196 | 66.7% | Claude Haiku 5.5 (58.2%) | Claude Fable 5.2 (71.8%) |
Tutor accuracy
Averaged over 9 models, the highest-scoring evidence group is ‘Course only’ (84.4%, 60 items) and the lowest-scoring is ‘Built from exercise options’ (72.9%, 113 items); the gap is clear (95% ranges do not overlap) for 3 of 9 models.
Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong. Left out, with fewer than 20 items: Planted typo.
View as table
| Evidence | Items | Average | Lowest model | Highest model |
|---|---|---|---|---|
| Course only | 60 | 84.4% | Qwen3.8 Flash (57.8%) | Claude Fable 5.2 (92.2%) |
| Built from exercise options | 113 | 72.9% | Claude Haiku 5.5 (54.0%) | Gemini 4 Hydrogen (87.9%) |
Cost per correct answer
#What a correct answer costs at OpenRouter prices on the day of the run.
Cost per 1,000 correct answers runs from $0.22 (GPT-6 Luna) to $11.49 (Gemini 4 Hydrogen), 51 times as much.
Suite cost, candidate and judges, divided by the number of full-score answers and shown per 1,000 answers; cheapest first. Log scale: each grid step multiplies the cost, so equal gaps are equal ratios. Score against cost per run is on the Models page.
View as table
| Model | Score | Cost per run | Per 1,000 correct |
|---|---|---|---|
| GPT-6 Luna | 75.6% | $0.78 | $0.22 |
| DeepSeek V4.1 Flash 1017 | 79.2% | $1.04 | $0.29 |
| Claude Haiku 5.5 | 67.1% | $1.23 | $0.40 |
| Gemini 4 Argon | 81.9% | $3.10 | $0.83 |
| Qwen3.8 Flash | 61.6% | $10.45 | $3.70 |
| GPT-6 Astra | 83.6% | $26.95 | $7.10 |
| Claude Fable 5.2 | 86.1% | $31.79 | $8.14 |
| Kimi K3.1 Pro | 78.4% | $36.20 | $10.14 |
| Gemini 4 Hydrogen | 84.4% | $43.89 | $11.49 |
Consistency
#Each item is asked several times. How often does a model get the same score every time?
Mean consistency runs from 50.6% (Qwen3.8 Flash) to 83.1% (Claude Fable 5.2).
Consistency is the share of items that got the same score in all 3 asks: a model that often changes its answer is less dependable for learners. Bar: mean over the tests; ticks: single tests (point at one to name it). Translation tests give partial credit, so exact repeats are rarer there.
View as table
| Model | Lesson exercises | Word forms | Translation | Round trip | Tutor accuracy | Mean | Score |
|---|---|---|---|---|---|---|---|
| Claude Fable 5.2 | 99.2% | 96.9% | 70.8% | 55.8% | 92.8% | 83.1% | 86.1% |
| GPT-6 Astra | 99.4% | 94.2% | 61.1% | 47.9% | 88.3% | 78.2% | 83.6% |
| Gemini 4 Hydrogen | 99.2% | 92.2% | 57.5% | 45.0% | 94.4% | 77.7% | 84.4% |
| Gemini 4 Argon | 98.1% | 89.4% | 51.7% | 32.1% | 91.1% | 72.5% | 81.9% |
| DeepSeek V4.1 Flash 1017 | 95.3% | 90.8% | 46.9% | 27.5% | 81.7% | 68.4% | 79.2% |
| GPT-6 Luna | 96.1% | 85.6% | 43.9% | 32.9% | 79.4% | 67.6% | 75.6% |
| Kimi K3.1 Pro | 96.9% | 87.5% | 42.2% | 22.5% | 77.2% | 65.3% | 78.4% |
| Claude Haiku 5.5 | 91.4% | 75.8% | 32.8% | 18.3% | 70.6% | 57.8% | 67.1% |
| Qwen3.8 Flash | 92.2% | 69.4% | 19.7% | 7.9% | 63.9% | 50.6% | 61.6% |
Score against consistency
#Across 9 models, higher-scoring models tend to be more consistent: a strong link (Spearman rank correlation 0.97); this is descriptive, not causal.
Points are models, labelled by suite rank; whiskers are 95% score ranges. Crowded labels link back to their dots. x: share of items with the same score on all three asks, averaged equally over tests; y: overall suite score. Translation tests rarely repeat exactly, so consistency is lower there by construction. Consistency uses all suite tests. Models need three-ask consistency on every included test.
- Claude Fable 5.2 83.1% consistency; 86.1% score
- Gemini 4 Hydrogen 77.7% consistency; 84.4% score
- GPT-6 Astra 78.2% consistency; 83.6% score
- Gemini 4 Argon 72.5% consistency; 81.9% score
- DeepSeek V4.1 Flash 1017 68.4% consistency; 79.2% score
- Kimi K3.1 Pro 65.3% consistency; 78.4% score
- GPT-6 Luna 67.6% consistency; 75.6% score
- Claude Haiku 5.5 57.8% consistency; 67.1% score
- Qwen3.8 Flash 50.6% consistency; 61.6% score
View as table
Link strength uses |Spearman rho|: ≥0.7 strong, ≥0.4 moderate, ≥0.2 weak; otherwise no clear link. Descriptive, not a significance test.
| Model | Mean consistency | Overall suite score |
|---|---|---|
| Claude Fable 5.2 | 83.1% | 86.1% |
| Gemini 4 Hydrogen | 77.7% | 84.4% |
| GPT-6 Astra | 78.2% | 83.6% |
| Gemini 4 Argon | 72.5% | 81.9% |
| DeepSeek V4.1 Flash 1017 | 68.4% | 79.2% |
| Kimi K3.1 Pro | 65.3% | 78.4% |
| GPT-6 Luna | 67.6% | 75.6% |
| Claude Haiku 5.5 | 57.8% | 67.1% |
| Qwen3.8 Flash | 50.6% | 61.6% |
Accept-list tests only
Across 9 models, higher-scoring models tend to be more consistent: a strong link (Spearman rank correlation 0.98); this is descriptive, not causal.
Points are models, labelled by suite rank; whiskers are 95% score ranges. Crowded labels link back to their dots. x: share of items with the same score on all three asks, averaged equally over tests; y: overall suite score. Translation tests rarely repeat exactly, so consistency is lower there by construction. Consistency uses accept-list tests only; y still uses the overall suite score. Models need three-ask consistency on every included test.
- Claude Fable 5.2 98.1% consistency; 86.1% score
- Gemini 4 Hydrogen 95.7% consistency; 84.4% score
- GPT-6 Astra 96.8% consistency; 83.6% score
- Gemini 4 Argon 93.8% consistency; 81.9% score
- DeepSeek V4.1 Flash 1017 93.1% consistency; 79.2% score
- Kimi K3.1 Pro 92.2% consistency; 78.4% score
- GPT-6 Luna 90.8% consistency; 75.6% score
- Claude Haiku 5.5 83.6% consistency; 67.1% score
- Qwen3.8 Flash 80.8% consistency; 61.6% score
View as table
Link strength uses |Spearman rho|: ≥0.7 strong, ≥0.4 moderate, ≥0.2 weak; otherwise no clear link. Descriptive, not a significance test.
| Model | Mean consistency | Overall suite score |
|---|---|---|
| Claude Fable 5.2 | 98.1% | 86.1% |
| Gemini 4 Hydrogen | 95.7% | 84.4% |
| GPT-6 Astra | 96.8% | 83.6% |
| Gemini 4 Argon | 93.8% | 81.9% |
| DeepSeek V4.1 Flash 1017 | 93.1% | 79.2% |
| Kimi K3.1 Pro | 92.2% | 78.4% |
| GPT-6 Luna | 90.8% | 75.6% |
| Claude Haiku 5.5 | 83.6% | 67.1% |
| Qwen3.8 Flash | 80.8% | 61.6% |