Teanga Bench

Findings

What the results show beyond the ranking: profiles, ties, difficulty, topics and cost. Every chart opens with a sentence computed from the data and has its numbers in a table.

Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built

Every model on every test

#

The quickest way to see each model's strengths and weaknesses.

Claude Fable 5.2 has the highest score in 5 of 5 tests, but in every test the top score is tied with at least one other model (95% ranges overlap).

Score of each model on each test; Overall counts every test equally. Darker cells are higher, shaded from 50% to 100%. highest in the column; tied with the highest (the 95% ranges overlap). Select a column heading to sort.

ModelOverallLesson exercisesWord formsTranslationRound tripTutor accuracy
Claude Fable 5.286.1% (highest)98.5% (highest)94.1% (highest)75.1% (highest)74.3% (highest)88.5% (highest)
Gemini 4 Hydrogen84.4% (tied with the highest)97.3% (tied with the highest)91.1% (tied with the highest)72.4% (tied with the highest)73.0% (tied with the highest)88.1% (tied with the highest)
GPT-6 Astra83.6% (tied with the highest)96.6% (tied with the highest)92.5% (tied with the highest)73.1% (tied with the highest)69.3% (tied with the highest)86.5% (tied with the highest)
Gemini 4 Argon81.9%98.2% (tied with the highest)85.4%71.9% (tied with the highest)70.0% (tied with the highest)84.1% (tied with the highest)
DeepSeek V4.1 Flash 101779.2%95.6% (tied with the highest)87.0%68.8%68.5%76.1%
Kimi K3.1 Pro78.4%95.8% (tied with the highest)82.1%69.4%65.3%79.3% (tied with the highest)
GPT-6 Luna75.6%93.6%80.5%68.0%64.5%71.3%
Claude Haiku 5.567.1%87.5%65.3%57.9%58.0%66.9%
Qwen3.8 Flash61.6%90.6%50.6%56.9%52.2%58.0%
View as table
ModelOverallLesson exercisesWord formsTranslationRound tripTutor accuracy
Claude Fable 5.286.1% 84.9–87.3%98.5% 97.1–99.5%94.1% 91.7–96.2%75.1% 72.5–77.6%74.3% 71.7–77.0%88.5% 84.1–92.4%
Gemini 4 Hydrogen84.4% 83.1–85.6%97.3% 95.6–98.8%91.1% 88.3–93.6%72.4% 69.9–75.0%73.0% 70.4–75.6%88.1% 83.5–92.2%
GPT-6 Astra83.6% 82.3–84.9%96.6% 94.5–98.3%92.5% 89.9–94.8%73.1% 70.5–75.7%69.3% 66.6–72.1%86.5% 82.0–90.7%
Gemini 4 Argon81.9% 80.5–83.3%98.2% 96.9–99.3%85.4% 82.1–88.5%71.9% 69.4–74.4%70.0% 67.3–72.6%84.1% 78.9–88.9%
DeepSeek V4.1 Flash 101779.2% 77.7–80.7%95.6% 93.6–97.2%87.0% 83.9–90.1%68.8% 66.2–71.4%68.5% 65.7–71.1%76.1% 70.6–81.3%
Kimi K3.1 Pro78.4% 76.9–79.9%95.8% 93.9–97.6%82.1% 78.4–85.6%69.4% 66.9–72.0%65.3% 62.5–68.0%79.3% 74.3–84.1%
GPT-6 Luna75.6% 73.9–77.2%93.6% 91.1–95.7%80.5% 76.9–84.1%68.0% 65.4–70.8%64.5% 61.7–67.4%71.3% 65.4–77.0%
Claude Haiku 5.567.1% 65.3–68.9%87.5% 84.4–90.5%65.3% 60.9–69.4%57.9% 55.1–60.9%58.0% 54.8–61.2%66.9% 60.9–72.6%
Qwen3.8 Flash61.6% 59.8–63.4%90.6% 87.9–93.1%50.6% 46.2–55.1%56.9% 54.2–59.9%52.2% 48.8–55.4%58.0% 51.9–63.7%

Who is tied with whom

#

Scores come from a sample of items, so small gaps can be chance. Models whose 95% ranges overlap are not clearly different.

Claude Fable 5.2 scores highest (86.1%) but is tied with Gemini 4 Hydrogen and GPT-6 Astra: their 95% ranges overlap.

Dot: overall score; line: its 95% range; the axis is zoomed in. The shaded column is the leader’s range: a model whose line reaches it may be tied with the leader. Groups run from the top: each is its highest model plus every model left whose range overlaps that model’s.

  1. Tied at the top: 3 models whose 95% range overlaps Claude Fable 5.2’s
  2. Claude Fable 5.2Claude Fable 5.2: 86.1%, 95% range 84.9% to 87.3%
  3. Gemini 4 HydrogenGemini 4 Hydrogen: 84.4%, 95% range 83.1% to 85.6%
  4. GPT-6 AstraGPT-6 Astra: 83.6%, 95% range 82.3% to 84.9%
  5. Group 2: 2 models whose range overlaps Gemini 4 Argon’s
  6. Gemini 4 ArgonGemini 4 Argon: 81.9%, 95% range 80.5% to 83.3%
  7. DeepSeek V4.1 Flash 1017DeepSeek V4.1 Flash 1017: 79.2%, 95% range 77.7% to 80.7%
  8. Group 3: 2 models whose range overlaps Kimi K3.1 Pro’s
  9. Kimi K3.1 ProKimi K3.1 Pro: 78.4%, 95% range 76.9% to 79.9%
  10. GPT-6 LunaGPT-6 Luna: 75.6%, 95% range 73.9% to 77.2%
  11. Group 4: Claude Haiku 5.5, overlapping no lower model
  12. Claude Haiku 5.5Claude Haiku 5.5: 67.1%, 95% range 65.3% to 68.9%
  13. Group 5: Qwen3.8 Flash, overlapping no lower model
  14. Qwen3.8 FlashQwen3.8 Flash: 61.6%, 95% range 59.8% to 63.4%
View as table
ModelScore95% rangeRange overlaps
Claude Fable 5.286.1%84.9–87.3%Gemini 4 Hydrogen, GPT-6 Astra
Gemini 4 Hydrogen84.4%83.1–85.6%Claude Fable 5.2, GPT-6 Astra, Gemini 4 Argon
GPT-6 Astra83.6%82.3–84.9%Claude Fable 5.2, Gemini 4 Hydrogen, Gemini 4 Argon
Gemini 4 Argon81.9%80.5–83.3%Gemini 4 Hydrogen, GPT-6 Astra, DeepSeek V4.1 Flash 1017
DeepSeek V4.1 Flash 101779.2%77.7–80.7%Gemini 4 Argon, Kimi K3.1 Pro
Kimi K3.1 Pro78.4%76.9–79.9%DeepSeek V4.1 Flash 1017, GPT-6 Luna
GPT-6 Luna75.6%73.9–77.2%Kimi K3.1 Pro
Claude Haiku 5.567.1%65.3–68.9%none
Qwen3.8 Flash61.6%59.8–63.4%none

Scores by CEFR level

#

Items carry the CEFR level of the course content they come from, from A1 (beginner) up. Do models drop as items get harder?

Lesson exercises

3 of 9 models score lower at B2 than at A1, but none of these drops is clear: the 95% ranges overlap.

Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.

Lesson exercises: score by CEFR levelMedian: A1 97.3%, A2 93.8%, B1 96.3%, B2 97.2%.80%85%90%95%100%A1n=125A2n=135B1n=73B2n=24Claude Fable 5.2: A1 99.7%; A2 97.8%; B1 97.3%; B2 100.0%Gemini 4 Hydrogen: A1 98.7%; A2 95.6%; B1 97.3%; B2 100.0%GPT-6 Astra: A1 97.1%; A2 94.8%; B1 97.7%; B2 100.0%Gemini 4 Argon: A1 100.0%; A2 96.8%; B1 97.3%; B2 100.0%DeepSeek V4.1 Flash 1017: A1 97.3%; A2 93.6%; B1 95.9%; B2 95.8%Kimi K3.1 Pro: A1 97.3%; A2 93.8%; B1 96.3%; B2 97.2%GPT-6 Luna: A1 95.5%; A2 90.9%; B1 95.0%; B2 94.4%Claude Haiku 5.5: A1 88.5%; A2 86.7%; B1 86.3%; B2 88.9%Qwen3.8 Flash: A1 90.1%; A2 89.4%; B1 92.7%; B2 91.7%Median: A1 97.3%; A2 93.8%; B1 96.3%; B2 97.2%Median
View as table
ModelA1 n=125A2 n=135B1 n=73B2 n=24C1 n=3
Claude Fable 5.299.7% 99.2–100.0%97.8% 95.3–99.8%97.3% 93.2–100.0%100.0% 100.0–100.0%too few items
Gemini 4 Hydrogen98.7% 96.5–100.0%95.6% 91.9–98.5%97.3% 93.6–100.0%100.0% 100.0–100.0%too few items
GPT-6 Astra97.1% 93.9–99.5%94.8% 91.1–98.5%97.7% 94.1–100.0%100.0% 100.0–100.0%too few items
Gemini 4 Argon100.0% 100.0–100.0%96.8% 94.3–99.0%97.3% 93.2–100.0%100.0% 100.0–100.0%too few items
DeepSeek V4.1 Flash 101797.3% 94.7–99.5%93.6% 89.9–97.0%95.9% 91.8–99.1%95.8% 87.5–100.0%too few items
Kimi K3.1 Pro97.3% 94.4–99.5%93.8% 89.9–97.3%96.3% 91.8–99.5%97.2% 91.7–100.0%too few items
GPT-6 Luna95.5% 91.7–98.1%90.9% 86.2–95.3%95.0% 90.0–99.1%94.4% 86.1–100.0%too few items
Claude Haiku 5.588.5% 83.2–93.3%86.7% 81.2–91.6%86.3% 79.0–92.7%88.9% 75.0–100.0%too few items
Qwen3.8 Flash90.1% 85.1–94.7%89.4% 84.7–93.8%92.7% 87.2–97.3%91.7% 80.6–100.0%too few items

Word forms

5 of 9 models score lower at B1 than at A1; the drop is clear (95% ranges do not overlap) for 3 of them.

Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.

Word forms: score by CEFR levelMedian: A1 86.3%, A2 82.9%, B1 79.5%.0%25%50%75%100%A1n=134A2n=140B1n=39Claude Fable 5.2: A1 92.8%; A2 92.9%; B1 98.3%Gemini 4 Hydrogen: A1 92.0%; A2 88.6%; B1 94.0%GPT-6 Astra: A1 92.3%; A2 90.7%; B1 94.0%Gemini 4 Argon: A1 84.8%; A2 85.5%; B1 79.5%DeepSeek V4.1 Flash 1017: A1 89.3%; A2 82.9%; B1 92.3%Kimi K3.1 Pro: A1 85.3%; A2 80.7%; B1 68.4%GPT-6 Luna: A1 86.3%; A2 75.7%; B1 67.5%Claude Haiku 5.5: A1 71.1%; A2 61.9%; B1 45.3%Qwen3.8 Flash: A1 63.2%; A2 45.5%; B1 13.7%Median: A1 86.3%; A2 82.9%; B1 79.5%Median
View as table
ModelA1 n=134A2 n=140B1 n=39B2 n=16
Claude Fable 5.292.8% 88.3–96.8%92.9% 88.6–96.4%98.3% 95.7–100.0%too few items
Gemini 4 Hydrogen92.0% 87.6–96.0%88.6% 83.8–92.9%94.0% 87.2–99.1%too few items
GPT-6 Astra92.3% 87.8–96.3%90.7% 86.0–95.0%94.0% 87.2–98.3%too few items
Gemini 4 Argon84.8% 79.1–90.0%85.5% 80.0–90.5%79.5% 67.5–90.6%too few items
DeepSeek V4.1 Flash 101789.3% 84.3–94.0%82.9% 76.9–88.3%92.3% 85.5–97.4%too few items
Kimi K3.1 Pro85.3% 79.6–90.3%80.7% 74.5–86.4%68.4% 55.6–80.3%too few items
GPT-6 Luna86.3% 80.8–91.3%75.7% 68.8–81.9%67.5% 55.6–79.5%too few items
Claude Haiku 5.571.1% 63.9–77.6%61.9% 55.0–68.8%45.3% 32.5–59.0%too few items
Qwen3.8 Flash63.2% 56.2–69.9%45.5% 38.8–52.9%13.7% 7.7–20.5%too few items

Translation

All 9 models score lower at B2 than at A1, but none of these drops is clear: the 95% ranges overlap.

Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.

Translation: score by CEFR levelMedian: A1 72.9%, A2 66.7%, B1 64.9%, B2 65.1%.40%60%80%100%A1n=112A2n=91B1n=47B2n=30Claude Fable 5.2: A1 79.2%; A2 71.8%; B1 67.3%; B2 68.7%Gemini 4 Hydrogen: A1 76.5%; A2 69.2%; B1 64.9%; B2 66.1%GPT-6 Astra: A1 76.6%; A2 70.4%; B1 65.9%; B2 65.1%Gemini 4 Argon: A1 74.5%; A2 68.0%; B1 65.0%; B2 66.9%DeepSeek V4.1 Flash 1017: A1 72.9%; A2 65.0%; B1 64.6%; B2 65.1%Kimi K3.1 Pro: A1 72.8%; A2 66.7%; B1 65.3%; B2 63.8%GPT-6 Luna: A1 72.8%; A2 62.0%; B1 63.2%; B2 62.9%Claude Haiku 5.5: A1 60.3%; A2 50.9%; B1 55.3%; B2 54.2%Qwen3.8 Flash: A1 61.6%; A2 52.5%; B1 52.5%; B2 48.9%Median: A1 72.9%; A2 66.7%; B1 64.9%; B2 65.1%Median
View as table
ModelA1 n=112A2 n=91B1 n=47B2 n=30C1 n=14C2 n=6
Claude Fable 5.279.2% 74.5–83.8%71.8% 66.6–77.1%67.3% 60.5–73.6%68.7% 59.0–78.0%too few itemstoo few items
Gemini 4 Hydrogen76.5% 71.6–81.3%69.2% 64.1–74.0%64.9% 57.6–71.7%66.1% 56.4–75.0%too few itemstoo few items
GPT-6 Astra76.6% 71.3–81.9%70.4% 65.4–75.3%65.9% 58.8–72.4%65.1% 55.4–74.6%too few itemstoo few items
Gemini 4 Argon74.5% 69.5–79.3%68.0% 62.9–73.1%65.0% 58.5–71.7%66.9% 57.6–75.9%too few itemstoo few items
DeepSeek V4.1 Flash 101772.9% 67.5–77.9%65.0% 59.7–70.0%64.6% 57.6–71.4%65.1% 55.0–74.3%too few itemstoo few items
Kimi K3.1 Pro72.8% 67.6–77.7%66.7% 61.6–71.7%65.3% 58.3–72.3%63.8% 54.3–72.8%too few itemstoo few items
GPT-6 Luna72.8% 67.6–78.2%62.0% 56.6–67.3%63.2% 56.5–69.8%62.9% 53.2–71.9%too few itemstoo few items
Claude Haiku 5.560.3% 54.7–66.0%50.9% 45.4–56.6%55.3% 47.7–63.1%54.2% 43.2–64.3%too few itemstoo few items
Qwen3.8 Flash61.6% 56.0–67.3%52.5% 46.9–58.0%52.5% 44.5–61.0%48.9% 37.4–58.9%too few itemstoo few items

Round trip

1 of 9 models score lower at B2 than at A1, but none of these drops is clear: the 95% ranges overlap.

Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.

Round trip: score by CEFR levelMedian: A1 64.7%, A2 69.9%, B1 63.4%, B2 68.1%.40%50%60%70%80%A1n=25A2n=88B1n=79B2n=27Claude Fable 5.2: A1 72.2%; A2 77.2%; B1 69.7%; B2 72.8%Gemini 4 Hydrogen: A1 69.2%; A2 77.0%; B1 67.4%; B2 70.9%GPT-6 Astra: A1 64.0%; A2 73.7%; B1 63.4%; B2 68.7%Gemini 4 Argon: A1 68.2%; A2 69.9%; B1 67.0%; B2 69.0%DeepSeek V4.1 Flash 1017: A1 64.4%; A2 70.5%; B1 65.1%; B2 68.1%Kimi K3.1 Pro: A1 66.8%; A2 66.6%; B1 59.1%; B2 67.5%GPT-6 Luna: A1 64.7%; A2 66.6%; B1 57.4%; B2 67.0%Claude Haiku 5.5: A1 56.6%; A2 59.0%; B1 52.5%; B2 57.5%Qwen3.8 Flash: A1 50.4%; A2 54.4%; B1 48.0%; B2 44.8%Median: A1 64.7%; A2 69.9%; B1 63.4%; B2 68.1%Median
View as table
ModelA1 n=25A2 n=88B1 n=79B2 n=27C1 n=15C2 n=6
Claude Fable 5.272.2% 63.0–81.4%77.2% 72.7–81.6%69.7% 65.5–73.6%72.8% 65.3–80.3%too few itemstoo few items
Gemini 4 Hydrogen69.2% 60.4–78.0%77.0% 72.6–81.4%67.4% 63.1–71.5%70.9% 63.7–77.8%too few itemstoo few items
GPT-6 Astra64.0% 55.8–72.5%73.7% 69.0–78.5%63.4% 58.5–67.9%68.7% 62.0–76.0%too few itemstoo few items
Gemini 4 Argon68.2% 59.4–77.2%69.9% 65.5–74.5%67.0% 62.6–71.5%69.0% 61.4–76.6%too few itemstoo few items
DeepSeek V4.1 Flash 101764.4% 57.3–71.8%70.5% 65.9–75.2%65.1% 60.3–69.9%68.1% 61.7–75.1%too few itemstoo few items
Kimi K3.1 Pro66.8% 58.6–75.4%66.6% 61.9–71.2%59.1% 54.2–64.0%67.5% 60.5–74.4%too few itemstoo few items
GPT-6 Luna64.7% 56.2–73.5%66.6% 61.8–71.5%57.4% 52.4–62.5%67.0% 59.5–74.5%too few itemstoo few items
Claude Haiku 5.556.6% 46.3–67.3%59.0% 53.9–64.4%52.5% 47.5–57.6%57.5% 48.3–67.4%too few itemstoo few items
Qwen3.8 Flash50.4% 39.1–61.8%54.4% 49.3–59.6%48.0% 42.5–53.8%44.8% 34.2–55.9%too few itemstoo few items

Tutor accuracy

All 9 models score lower at B2 than at A1; the drop is clear (95% ranges do not overlap) for 4 of them.

Thin lines: one per model (point at a line to name it). Bold line: the median model. Band: lowest to highest model. n: items at that level; levels with fewer than 20 items are left out.

Tutor accuracy: score by CEFR levelMedian: A1 93.9%, A2 77.8%, B1 73.8%, B2 73.9%.20%40%60%80%100%A1n=22A2n=27B1n=42B2n=23Claude Fable 5.2: A1 97.0%; A2 92.6%; B1 84.1%; B2 84.1%Gemini 4 Hydrogen: A1 97.0%; A2 88.9%; B1 83.3%; B2 88.4%GPT-6 Astra: A1 100.0%; A2 85.2%; B1 85.7%; B2 76.8%Gemini 4 Argon: A1 90.9%; A2 87.7%; B1 77.0%; B2 73.9%DeepSeek V4.1 Flash 1017: A1 93.9%; A2 77.8%; B1 64.3%; B2 69.6%Kimi K3.1 Pro: A1 97.0%; A2 74.1%; B1 73.8%; B2 75.4%GPT-6 Luna: A1 84.8%; A2 74.1%; B1 61.1%; B2 65.2%Claude Haiku 5.5: A1 75.8%; A2 65.4%; B1 61.1%; B2 56.5%Qwen3.8 Flash: A1 78.8%; A2 60.5%; B1 54.8%; B2 33.3%Median: A1 93.9%; A2 77.8%; B1 73.8%; B2 73.9%Median
View as table
ModelA1 n=22A2 n=27B1 n=42B2 n=23C1 n=10C2 n=2
Claude Fable 5.297.0% 90.9–100.0%92.6% 82.7–100.0%84.1% 73.0–92.9%84.1% 69.6–97.1%too few itemstoo few items
Gemini 4 Hydrogen97.0% 90.9–100.0%88.9% 76.5–97.6%83.3% 72.2–92.9%88.4% 73.9–100.0%too few itemstoo few items
GPT-6 Astra100.0% 100.0–100.0%85.2% 71.6–96.3%85.7% 75.4–93.7%76.8% 59.4–92.8%too few itemstoo few items
Gemini 4 Argon90.9% 80.3–100.0%87.7% 75.3–97.5%77.0% 64.3–88.1%73.9% 56.5–91.3%too few itemstoo few items
DeepSeek V4.1 Flash 101793.9% 86.4–100.0%77.8% 64.2–88.9%64.3% 50.8–76.2%69.6% 53.6–84.1%too few itemstoo few items
Kimi K3.1 Pro97.0% 92.4–100.0%74.1% 60.5–86.4%73.8% 61.9–84.9%75.4% 60.9–88.4%too few itemstoo few items
GPT-6 Luna84.8% 71.2–97.0%74.1% 60.5–85.2%61.1% 48.4–73.8%65.2% 46.4–82.6%too few itemstoo few items
Claude Haiku 5.575.8% 60.6–89.4%65.4% 49.4–81.5%61.1% 48.4–73.0%56.5% 37.7–73.9%too few itemstoo few items
Qwen3.8 Flash78.8% 66.7–90.9%60.5% 45.7–74.1%54.8% 42.1–66.7%33.3% 18.8–47.8%too few itemstoo few items

Hardest and easiest topics

#

Each item records its grammar topic, error type or course category. Which are hardest across models?

Word forms

Averaged over 9 models, the easiest grammar topic is article gender mutation (93.5%, 31 items) and the hardest is verb conditional (72.5%, 23 items); the gap is clear (95% ranges do not overlap) for 4 of 9 models.

Bar: average over models; ticks: single models. Shows the 5 easiest and 5 hardest of 11 grammar topics with 20 or more items; 10 smaller ones are left out.

  1. Easiest 5
  2. article gender mutation31 itemsAverage over 9 models: 93.5%Claude Fable 5.2: 100.0%Gemini 4 Hydrogen: 96.8%GPT-6 Astra: 100.0%Gemini 4 Argon: 96.8%DeepSeek V4.1 Flash 1017: 95.7%Kimi K3.1 Pro: 97.8%GPT-6 Luna: 97.8%Claude Haiku 5.5: 79.6%Qwen3.8 Flash: 77.4%
  3. copula31 itemsAverage over 9 models: 86.9%Claude Fable 5.2: 96.8%Gemini 4 Hydrogen: 87.1%GPT-6 Astra: 90.3%Gemini 4 Argon: 90.3%DeepSeek V4.1 Flash 1017: 90.3%Kimi K3.1 Pro: 91.4%GPT-6 Luna: 88.2%Claude Haiku 5.5: 67.7%Qwen3.8 Flash: 79.6%
  4. number cardinal30 itemsAverage over 9 models: 84.8%Claude Fable 5.2: 91.1%Gemini 4 Hydrogen: 96.7%GPT-6 Astra: 94.4%Gemini 4 Argon: 87.8%DeepSeek V4.1 Flash 1017: 86.7%Kimi K3.1 Pro: 84.4%GPT-6 Luna: 87.8%Claude Haiku 5.5: 71.1%Qwen3.8 Flash: 63.3%
  5. verb present23 itemsAverage over 9 models: 84.5%Claude Fable 5.2: 95.7%Gemini 4 Hydrogen: 95.7%GPT-6 Astra: 97.1%Gemini 4 Argon: 81.2%DeepSeek V4.1 Flash 1017: 94.2%Kimi K3.1 Pro: 85.5%GPT-6 Luna: 82.6%Claude Haiku 5.5: 79.7%Qwen3.8 Flash: 49.3%
  6. verb imperative27 itemsAverage over 9 models: 82.6%Claude Fable 5.2: 93.8%Gemini 4 Hydrogen: 91.4%GPT-6 Astra: 93.8%Gemini 4 Argon: 92.6%DeepSeek V4.1 Flash 1017: 77.8%Kimi K3.1 Pro: 84.0%GPT-6 Luna: 85.2%Claude Haiku 5.5: 71.6%Qwen3.8 Flash: 53.1%
  7. Hardest 5
  8. number ordinal30 itemsAverage over 9 models: 81.2%Claude Fable 5.2: 92.2%Gemini 4 Hydrogen: 92.2%GPT-6 Astra: 92.2%Gemini 4 Argon: 82.2%DeepSeek V4.1 Flash 1017: 88.9%Kimi K3.1 Pro: 80.0%GPT-6 Luna: 82.2%Claude Haiku 5.5: 63.3%Qwen3.8 Flash: 57.8%
  9. verb future23 itemsAverage over 9 models: 77.0%Claude Fable 5.2: 87.0%Gemini 4 Hydrogen: 94.2%GPT-6 Astra: 87.0%Gemini 4 Argon: 75.4%DeepSeek V4.1 Flash 1017: 79.7%Kimi K3.1 Pro: 84.1%GPT-6 Luna: 84.1%Claude Haiku 5.5: 62.3%Qwen3.8 Flash: 39.1%
  10. noun plural30 itemsAverage over 9 models: 73.6%Claude Fable 5.2: 85.6%Gemini 4 Hydrogen: 74.4%GPT-6 Astra: 84.4%Gemini 4 Argon: 77.8%DeepSeek V4.1 Flash 1017: 76.7%Kimi K3.1 Pro: 75.6%GPT-6 Luna: 73.3%Claude Haiku 5.5: 61.1%Qwen3.8 Flash: 53.3%
  11. verb past22 itemsAverage over 9 models: 72.9%Claude Fable 5.2: 100.0%Gemini 4 Hydrogen: 89.4%GPT-6 Astra: 90.9%Gemini 4 Argon: 78.8%DeepSeek V4.1 Flash 1017: 80.3%Kimi K3.1 Pro: 71.2%GPT-6 Luna: 66.7%Claude Haiku 5.5: 43.9%Qwen3.8 Flash: 34.8%
  12. verb conditional23 itemsAverage over 9 models: 72.5%Claude Fable 5.2: 98.6%Gemini 4 Hydrogen: 95.7%GPT-6 Astra: 97.1%Gemini 4 Argon: 87.0%DeepSeek V4.1 Flash 1017: 92.8%Kimi K3.1 Pro: 62.3%GPT-6 Luna: 68.1%Claude Haiku 5.5: 36.2%Qwen3.8 Flash: 14.5%
View as table
Grammar topicItemsAverageLowest modelHighest model
article gender mutation3193.5%Qwen3.8 Flash (77.4%)GPT-6 Astra (100.0%)
copula3186.9%Claude Haiku 5.5 (67.7%)Claude Fable 5.2 (96.8%)
number cardinal3084.8%Qwen3.8 Flash (63.3%)Gemini 4 Hydrogen (96.7%)
verb present2384.5%Qwen3.8 Flash (49.3%)GPT-6 Astra (97.1%)
verb imperative2782.6%Qwen3.8 Flash (53.1%)GPT-6 Astra (93.8%)
prepositional pronoun2882.3%Qwen3.8 Flash (44.0%)GPT-6 Astra (97.6%)
number ordinal3081.2%Qwen3.8 Flash (57.8%)Gemini 4 Hydrogen (92.2%)
verb future2377.0%Qwen3.8 Flash (39.1%)Gemini 4 Hydrogen (94.2%)
noun plural3073.6%Qwen3.8 Flash (53.3%)Claude Fable 5.2 (85.6%)
verb past2272.9%Qwen3.8 Flash (34.8%)Claude Fable 5.2 (100.0%)
verb conditional2372.5%Qwen3.8 Flash (14.5%)Claude Fable 5.2 (98.6%)

Lesson exercises

Averaged over 9 models, the easiest course category is recap (96.1%, 37 items) and the hardest is grammar (92.2%, 21 items); for every model the two 95% ranges overlap, so the gap is not clear.

Bar: average over models; ticks: single models. Shows every course category with 20 or more items; 70 smaller ones are left out.

  1. recap37 itemsAverage over 9 models: 96.1%Claude Fable 5.2: 100.0%Gemini 4 Hydrogen: 97.3%GPT-6 Astra: 100.0%Gemini 4 Argon: 98.2%DeepSeek V4.1 Flash 1017: 97.3%Kimi K3.1 Pro: 97.3%GPT-6 Luna: 94.6%Claude Haiku 5.5: 88.3%Qwen3.8 Flash: 91.9%
  2. guide32 itemsAverage over 9 models: 95.4%Claude Fable 5.2: 99.0%Gemini 4 Hydrogen: 100.0%GPT-6 Astra: 96.9%Gemini 4 Argon: 99.0%DeepSeek V4.1 Flash 1017: 97.9%Kimi K3.1 Pro: 96.9%GPT-6 Luna: 93.8%Claude Haiku 5.5: 82.3%Qwen3.8 Flash: 92.7%
  3. unit focus51 itemsAverage over 9 models: 92.3%Claude Fable 5.2: 98.0%Gemini 4 Hydrogen: 96.7%GPT-6 Astra: 94.1%Gemini 4 Argon: 99.3%DeepSeek V4.1 Flash 1017: 91.5%Kimi K3.1 Pro: 92.8%GPT-6 Luna: 90.8%Claude Haiku 5.5: 82.4%Qwen3.8 Flash: 85.0%
  4. grammar21 itemsAverage over 9 models: 92.2%Claude Fable 5.2: 95.2%Gemini 4 Hydrogen: 95.2%GPT-6 Astra: 95.2%Gemini 4 Argon: 95.2%DeepSeek V4.1 Flash 1017: 96.8%Kimi K3.1 Pro: 95.2%GPT-6 Luna: 90.5%Claude Haiku 5.5: 79.4%Qwen3.8 Flash: 87.3%
View as table
Course categoryItemsAverageLowest modelHighest model
recap3796.1%Claude Haiku 5.5 (88.3%)GPT-6 Astra (100.0%)
guide3295.4%Claude Haiku 5.5 (82.3%)Gemini 4 Hydrogen (100.0%)
unit focus5192.3%Claude Haiku 5.5 (82.4%)Gemini 4 Argon (99.3%)
grammar2192.2%Claude Haiku 5.5 (79.4%)DeepSeek V4.1 Flash 1017 (96.8%)

Tutor accuracy

Averaged over 9 models, the easiest course category is guide (82.4%, 20 items) and the hardest is grammar (79.0%, 42 items); for every model the two 95% ranges overlap, so the gap is not clear.

Bar: average over models; ticks: single models. Shows every course category with 20 or more items; 25 smaller ones are left out.

  1. guide20 itemsAverage over 9 models: 82.4%Claude Fable 5.2: 95.0%Gemini 4 Hydrogen: 90.0%GPT-6 Astra: 90.0%Gemini 4 Argon: 91.7%DeepSeek V4.1 Flash 1017: 75.0%Kimi K3.1 Pro: 83.3%GPT-6 Luna: 75.0%Claude Haiku 5.5: 70.0%Qwen3.8 Flash: 71.7%
  2. no error30 itemsAverage over 9 models: 79.6%Claude Fable 5.2: 91.1%Gemini 4 Hydrogen: 86.7%GPT-6 Astra: 81.1%Gemini 4 Argon: 82.2%DeepSeek V4.1 Flash 1017: 80.0%Kimi K3.1 Pro: 84.4%GPT-6 Luna: 80.0%Claude Haiku 5.5: 82.2%Qwen3.8 Flash: 48.9%
  3. grammar42 itemsAverage over 9 models: 79.0%Claude Fable 5.2: 91.3%Gemini 4 Hydrogen: 88.1%GPT-6 Astra: 89.7%Gemini 4 Argon: 90.5%DeepSeek V4.1 Flash 1017: 82.5%Kimi K3.1 Pro: 73.8%GPT-6 Luna: 71.4%Claude Haiku 5.5: 61.1%Qwen3.8 Flash: 62.7%
View as table
Course categoryItemsAverageLowest modelHighest model
guide2082.4%Claude Haiku 5.5 (70.0%)Claude Fable 5.2 (95.0%)
no error3079.6%Qwen3.8 Flash (48.9%)Claude Fable 5.2 (91.1%)
grammar4279.0%Claude Haiku 5.5 (61.1%)Claude Fable 5.2 (91.3%)

Topic labels come from the item sources. Where results include a joint breakdown, grammar topic by language shows how these averages differ by language.

Grammar topic by language

#

Where each topic is hardest, averaged over the models with a recorded topic–language breakdown.

Lesson exercises

The lowest observed model-average score is unit focus for Manx (82.9%); there are too few fully represented topics to compare languages.

Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.

TopicScottish GaelicManxBretonCornish
grammar——87.7% (hardest in this column)—
unit focus93.7% (hardest in this column)82.9% (hardest in this column)—97.5%
narrative grammar—91.9%——
recap—97.8%—91.7% (hardest in this column)
guide———96.5%
grammar point97.4%———
chapter overview—98.7%——

Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.

View as table
TopicScottish GaelicManxBretonCornish
grammar——87.7%; 13 items per model; 9 models—
unit focus93.7%; 10 items per model; 9 models82.9%; 16 items per model; 9 models—97.5%; 22 items per model; 9 models
narrative grammar—91.9%; 5 items per model; 9 models——
recap—97.8%; 15 items per model; 9 models—91.7%; 12 items per model; 9 models
guide———96.5%; 23 items per model; 9 models
grammar point97.4%; 10 items per model; 9 models———
chapter overview—98.7%; 11 items per model; 9 models——

Word forms

The lowest observed model-average score is verb past for Cornish (40.0%); Manx is lowest on 3 of 5 topics with every language represented (point estimates, not clear differences).

Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.

TopicWelshScottish GaelicManxBretonCornish
verb past——88.1%—40.0% (hardest in this column)
verb future——77.0%74.1%51.9%
verb conditional85.2%—65.2%68.9% (hardest in this column)—
verb imperfect85.9%———61.5%
noun plural85.9%96.3%44.4% (hardest in this column)75.6%72.2%
verb imperative79.3% (hardest in this column)—74.8%—71.9%
noun genitive—77.0%———
prepositional pronoun88.1%—77.0%74.1%76.3%
number ordinal97.8%89.6%49.6%94.1%71.0%
number cardinal97.0%88.9%62.2%88.1%77.2%
verb pluperfect83.0%————
verb preterite83.7%————
copula94.1%47.4% (hardest in this column)87.0%99.3%95.7%
verb present94.8%—94.8%—66.7%
article gender mutation100.0%91.9%93.8%83.0%96.3%
number personal—94.1%———

Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.

View as table
TopicWelshScottish GaelicManxBretonCornish
verb past——88.1%; 5 items per model; 9 models—40.0%; 5 items per model; 9 models
verb future——77.0%; 5 items per model; 9 models74.1%; 5 items per model; 9 models51.9%; 5 items per model; 9 models
verb conditional85.2%; 5 items per model; 9 models—65.2%; 5 items per model; 9 models68.9%; 5 items per model; 9 models—
verb imperfect85.9%; 5 items per model; 9 models———61.5%; 5 items per model; 9 models
noun plural85.9%; 5 items per model; 9 models96.3%; 5 items per model; 9 models44.4%; 5 items per model; 9 models75.6%; 5 items per model; 9 models72.2%; 6 items per model; 9 models
verb imperative79.3%; 5 items per model; 9 models—74.8%; 5 items per model; 9 models—71.9%; 5 items per model; 9 models
noun genitive—77.0%; 5 items per model; 9 models———
prepositional pronoun88.1%; 5 items per model; 9 models—77.0%; 5 items per model; 9 models74.1%; 5 items per model; 9 models76.3%; 5 items per model; 9 models
number ordinal97.8%; 5 items per model; 9 models89.6%; 5 items per model; 9 models49.6%; 5 items per model; 9 models94.1%; 5 items per model; 9 models71.0%; 6 items per model; 9 models
number cardinal97.0%; 5 items per model; 9 models88.9%; 5 items per model; 9 models62.2%; 5 items per model; 9 models88.1%; 5 items per model; 9 models77.2%; 6 items per model; 9 models
verb pluperfect83.0%; 5 items per model; 9 models————
verb preterite83.7%; 5 items per model; 9 models————
copula94.1%; 5 items per model; 9 models47.4%; 5 items per model; 9 models87.0%; 6 items per model; 9 models99.3%; 5 items per model; 9 models95.7%; 6 items per model; 9 models
verb present94.8%; 5 items per model; 9 models—94.8%; 5 items per model; 9 models—66.7%; 5 items per model; 9 models
article gender mutation100.0%; 5 items per model; 9 models91.9%; 5 items per model; 9 models93.8%; 6 items per model; 9 models83.0%; 5 items per model; 9 models96.3%; 6 items per model; 9 models
number personal—94.1%; 5 items per model; 9 models———

Translation

The lowest observed model-average score is to english for Manx (42.7%); Manx is lowest on 2 of 2 topics with every language represented (point estimates, not clear differences).

Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.

TopicIrishWelshScottish GaelicManxBretonCornish
from english75.2%73.3% (hardest in this column)67.3% (hardest in this column)43.3%58.5%61.7% (hardest in this column)
legal or administrative70.6% (hardest in this column)84.1%——57.0% (hardest in this column)—
to english84.9%77.4%84.3%42.7% (hardest in this column)80.9%77.5%

Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.

View as table
TopicIrishWelshScottish GaelicManxBretonCornish
from english75.2%; 20 items per model; 9 models73.3%; 20 items per model; 9 models67.3%; 30 items per model; 9 models43.3%; 30 items per model; 9 models58.5%; 20 items per model; 9 models61.7%; 30 items per model; 9 models
legal or administrative70.6%; 20 items per model; 9 models84.1%; 20 items per model; 9 models——57.0%; 20 items per model; 9 models—
to english84.9%; 20 items per model; 9 models77.4%; 20 items per model; 9 models84.3%; 30 items per model; 9 models42.7%; 30 items per model; 9 models80.9%; 20 items per model; 9 models77.5%; 30 items per model; 9 models

Round trip

The lowest observed model-average score is round trip for Manx (49.9%); Manx is lowest on 1 of 1 topics with every language represented (point estimates, not clear differences).

Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.

TopicIrishWelshScottish GaelicManxBretonCornish
round trip76.7% (hardest in this column)80.5% (hardest in this column)72.2% (hardest in this column)49.9% (hardest in this column)58.7% (hardest in this column)58.7% (hardest in this column)

Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.

View as table
TopicIrishWelshScottish GaelicManxBretonCornish
round trip76.7%; 40 items per model; 9 models80.5%; 40 items per model; 9 models72.2%; 40 items per model; 9 models49.9%; 40 items per model; 9 models58.7%; 40 items per model; 9 models58.7%; 40 items per model; 9 models

Tutor accuracy

The lowest observed model-average score is application for Scottish Gaelic (57.9%); Scottish Gaelic is lowest on 1 of 1 topics with every language represented (point estimates, not clear differences).

Mean score across available models, each model weighted equally; topics sorted by mean, hardest first. Darker cells mean higher scores (0–100%). ▼ marks every tied hardest cell in each column. Model coverage can differ between cells; averages have no pooled confidence interval.

TopicIrishWelshScottish Gaelic
application——57.9% (hardest in this column)
grammar point——70.1%
vocabulary71.3% (hardest in this column)——
grammar80.9%78.2% (hardest in this column)—
no error79.6%80.7%78.5%
guide80.6%85.7%—
unit focus——85.2%
orthography doubled letter—96.3%—

Minimum: 5 dataset items per topic–language pair in each contributing model result. A dash means no qualifying result. Counts are dataset items, not repeated asks or sums across models.

View as table
TopicIrishWelshScottish Gaelic
application——57.9%; 11 items per model; 9 models
grammar point——70.1%; 12 items per model; 9 models
vocabulary71.3%; 12 items per model; 9 models——
grammar80.9%; 13 items per model; 9 models78.2%; 29 items per model; 9 models—
no error79.6%; 10 items per model; 9 models80.7%; 10 items per model; 9 models78.5%; 10 items per model; 9 models
guide80.6%; 13 items per model; 9 models85.7%; 7 items per model; 9 models—
unit focus——85.2%; 6 items per model; 9 models
orthography doubled letter—96.3%; 7 items per model; 9 models—

Translation directions

#

Translation items go into English or out of English, or come from legal and administrative parallel text.

Translation

All 9 models score higher translating into English than out of English; the gap is clear (95% ranges do not overlap) for 7 of them.

chrF++ score by direction for each model; the axis is zoomed in. Right: into English minus out of English, in percentage points. The legal and administrative items are published parallel text, in either direction; Breton is paired with French there.

  • Into English 150 items
  • Out of English 150 items
  • Legal and administrative corpus 60 items
  1. Claude Fable 5.2Claude Fable 5.2, legal and administrative corpus: 77.9%Claude Fable 5.2, into english: 77.6%Claude Fable 5.2, out of english: 71.4%
  2. Gemini 4 HydrogenGemini 4 Hydrogen, legal and administrative corpus: 75.3%Gemini 4 Hydrogen, into english: 75.1%Gemini 4 Hydrogen, out of english: 68.6%
  3. GPT-6 AstraGPT-6 Astra, legal and administrative corpus: 76.3%GPT-6 Astra, into english: 78.0%GPT-6 Astra, out of english: 66.9%
  4. Gemini 4 ArgonGemini 4 Argon, legal and administrative corpus: 77.4%Gemini 4 Argon, into english: 75.3%Gemini 4 Argon, out of english: 66.3%
  5. DeepSeek V4.1 Flash 1017DeepSeek V4.1 Flash 1017, legal and administrative corpus: 68.3%DeepSeek V4.1 Flash 1017, into english: 74.0%DeepSeek V4.1 Flash 1017, out of english: 63.7%
  6. Kimi K3.1 ProKimi K3.1 Pro, legal and administrative corpus: 69.9%Kimi K3.1 Pro, into english: 75.8%Kimi K3.1 Pro, out of english: 62.8%
  7. GPT-6 LunaGPT-6 Luna, legal and administrative corpus: 69.6%GPT-6 Luna, into english: 75.0%GPT-6 Luna, out of english: 60.4%
  8. Claude Haiku 5.5Claude Haiku 5.5, legal and administrative corpus: 62.3%Claude Haiku 5.5, into english: 62.3%Claude Haiku 5.5, out of english: 51.7%
  9. Qwen3.8 FlashQwen3.8 Flash, legal and administrative corpus: 58.2%Qwen3.8 Flash, into english: 66.7%Qwen3.8 Flash, out of english: 46.7%
View as table
ModelInto English n=150Out of English n=150Legal and administrative corpus n=60
Claude Fable 5.277.6% 73.5–81.6%71.4% 67.3–75.4%77.9% 73.0–82.7%
Gemini 4 Hydrogen75.1% 71.2–78.9%68.6% 64.4–72.8%75.3% 70.8–80.0%
GPT-6 Astra78.0% 73.8–82.1%66.9% 62.7–71.0%76.3% 71.7–80.8%
Gemini 4 Argon75.3% 71.3–79.2%66.3% 62.2–70.1%77.4% 72.9–81.9%
DeepSeek V4.1 Flash 101774.0% 69.9–78.0%63.7% 59.5–67.8%68.3% 63.5–73.2%
Kimi K3.1 Pro75.8% 71.6–79.7%62.8% 58.8–66.6%69.9% 64.8–74.9%
GPT-6 Luna75.0% 70.7–79.2%60.4% 56.4–64.4%69.6% 64.8–74.3%
Claude Haiku 5.562.3% 57.5–67.2%51.7% 47.5–55.9%62.3% 56.8–67.6%
Qwen3.8 Flash66.7% 62.0–71.4%46.7% 42.5–50.8%58.2% 53.7–62.9%

Tutor accuracy step by step

#

A tutor item counts only when three rule-marked checks all pass on the same item. Which check fails most?

Tutor accuracy

Averaged over 9 models, the weakest rule-marked step is ‘fixed it’ (77.7%) and the strongest is ‘changed nothing else’ (84.7%); the judge-marked explanation passes 69.7% on average for the 7 models with a comparable explanation score.

Share of items that pass each check, over every ask. The score needs all three rule checks on the same item. The explanation (hatched) is marked by two judge models from other families: it is secondary, not human-validated and not part of the score.

ModelFound the errorFixed itChanged nothing elseAll three: the scoreExplanation judge-marked
Claude Fable 5.288.5%88.5%93.0%88.5%78.5%
Gemini 4 Hydrogen88.3%88.1%91.3%88.1%76.4%
GPT-6 Astra86.7%86.5%89.4%86.5%69.9%
Gemini 4 Argon85.0%84.1%89.1%84.1%75.9%
Kimi K3.1 Pro79.6%79.3%88.0%79.3%70.4%
DeepSeek V4.1 Flash 101776.1%76.3%88.1%76.1%not scored: same family as a judge
GPT-6 Luna72.2%71.5%82.6%71.3%57.4%
Claude Haiku 5.568.1%67.0%76.5%66.9%59.6%
Qwen3.8 Flash58.9%58.1%64.3%58.0%not scored: same family as a judge
View as table
ModelScore and 95% rangeExplanation and 95% rangeJudges agree
Claude Fable 5.288.5% 84.1–92.4%78.5% 72.9–84.1%93.5%
Gemini 4 Hydrogen88.1% 83.5–92.2%76.4% 70.4–82.0%95.1%
GPT-6 Astra86.5% 82.0–90.7%69.9% 63.5–76.2%91.0%
Gemini 4 Argon84.1% 78.9–88.9%75.9% 70.1–81.7%95.2%
Kimi K3.1 Pro79.3% 74.3–84.1%70.4% 64.7–76.0%93.4%
DeepSeek V4.1 Flash 101776.1% 70.6–81.3%n/an/a
GPT-6 Luna71.3% 65.4–77.0%57.4% 50.6–64.2%93.2%
Claude Haiku 5.566.9% 60.9–72.6%59.6% 53.1–65.8%95.2%
Qwen3.8 Flash58.0% 51.9–63.7%n/an/a

Evidence check

#

Each gold answer has an evidence label: cross-checked against a dictionary, from the course alone, or marked unverified. Does weaker gold change the picture?

Labels, from strongest to weakest gold: cross-checked against a dictionary (BÚNaMo or Kaikki), from a published parallel corpus, from the course alone, or marked unverified or built by a rule. What each label means.

Round trip

Averaged over 9 models, the highest-scoring evidence group is ‘Course, form listed in Kaikki’ (75.0%, 64 items) and the lowest-scoring is ‘Course, marked unverified’ (58.7%, 40 items); the gap is clear (95% ranges do not overlap) for 5 of 9 models.

Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong.

  1. Course, form listed in Kaikki64 items · blas-app+kaikkiAverage over 9 models: 75.0%Claude Fable 5.2: 79.3%Gemini 4 Hydrogen: 79.6%GPT-6 Astra: 74.6%Gemini 4 Argon: 79.7%DeepSeek V4.1 Flash 1017: 75.2%Kimi K3.1 Pro: 76.3%GPT-6 Luna: 73.9%Claude Haiku 5.5: 69.1%Qwen3.8 Flash: 67.5%
  2. Course only136 items · blas-app-onlyAverage over 9 models: 64.1%Claude Fable 5.2: 72.4%Gemini 4 Hydrogen: 70.2%GPT-6 Astra: 67.7%Gemini 4 Argon: 67.2%DeepSeek V4.1 Flash 1017: 66.2%Kimi K3.1 Pro: 62.5%GPT-6 Luna: 63.5%Claude Haiku 5.5: 56.0%Qwen3.8 Flash: 51.1%
  3. Course, marked unverified40 items · blas-app-unverifiedAverage over 9 models: 58.7%Claude Fable 5.2: 72.9%Gemini 4 Hydrogen: 71.8%GPT-6 Astra: 66.1%Gemini 4 Argon: 63.8%DeepSeek V4.1 Flash 1017: 65.3%Kimi K3.1 Pro: 57.5%GPT-6 Luna: 52.9%Claude Haiku 5.5: 47.2%Qwen3.8 Flash: 31.2%
View as table
EvidenceItemsAverageLowest modelHighest model
Course, form listed in Kaikki6475.0%Qwen3.8 Flash (67.5%)Gemini 4 Argon (79.7%)
Course only13664.1%Qwen3.8 Flash (51.1%)Claude Fable 5.2 (72.4%)
Course, marked unverified4058.7%Qwen3.8 Flash (31.2%)Claude Fable 5.2 (72.9%)

Lesson exercises

Averaged over 9 models, the highest-scoring evidence group is ‘Course only’ (95.1%, 257 items) and the lowest-scoring is ‘Course, form listed in Kaikki’ (93.8%, 96 items); for every model the two 95% ranges overlap, so the gap is not clear.

Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong. Left out, with fewer than 20 items: Course, form listed in BÚNaMo.

  1. Course only257 items · blas-app-onlyAverage over 9 models: 95.1%Claude Fable 5.2: 98.6%Gemini 4 Hydrogen: 96.8%GPT-6 Astra: 97.0%Gemini 4 Argon: 98.2%DeepSeek V4.1 Flash 1017: 96.0%Kimi K3.1 Pro: 96.1%GPT-6 Luna: 94.0%Claude Haiku 5.5: 87.8%Qwen3.8 Flash: 91.7%
  2. Course, form listed in Kaikki96 items · blas-app+kaikkiAverage over 9 models: 93.8%Claude Fable 5.2: 98.3%Gemini 4 Hydrogen: 98.6%GPT-6 Astra: 95.1%Gemini 4 Argon: 98.3%DeepSeek V4.1 Flash 1017: 94.1%Kimi K3.1 Pro: 94.8%GPT-6 Luna: 92.0%Claude Haiku 5.5: 85.8%Qwen3.8 Flash: 86.8%
View as table
EvidenceItemsAverageLowest modelHighest model
Course only25795.1%Claude Haiku 5.5 (87.8%)Claude Fable 5.2 (98.6%)
Course, form listed in Kaikki9693.8%Claude Haiku 5.5 (85.8%)Gemini 4 Hydrogen (98.6%)

Word forms

Averaged over 9 models, the highest-scoring evidence group is ‘Course only’ (82.8%, 173 items) and the lowest-scoring is ‘Course, form listed in Kaikki’ (79.0%, 176 items); for every model the two 95% ranges overlap, so the gap is not clear.

Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong. Left out, with fewer than 20 items: Course, form listed in BÚNaMo.

  1. Course only173 items · blas-app-onlyAverage over 9 models: 82.8%Claude Fable 5.2: 94.4%Gemini 4 Hydrogen: 91.7%GPT-6 Astra: 95.2%Gemini 4 Argon: 87.7%DeepSeek V4.1 Flash 1017: 90.0%Kimi K3.1 Pro: 81.3%GPT-6 Luna: 82.5%Claude Haiku 5.5: 67.8%Qwen3.8 Flash: 54.3%
  2. Course, form listed in Kaikki176 items · blas-app+kaikkiAverage over 9 models: 79.0%Claude Fable 5.2: 93.4%Gemini 4 Hydrogen: 90.3%GPT-6 Astra: 89.6%Gemini 4 Argon: 82.8%DeepSeek V4.1 Flash 1017: 83.9%Kimi K3.1 Pro: 82.6%GPT-6 Luna: 79.0%Claude Haiku 5.5: 62.3%Qwen3.8 Flash: 46.8%
View as table
EvidenceItemsAverageLowest modelHighest model
Course only17382.8%Qwen3.8 Flash (54.3%)GPT-6 Astra (95.2%)
Course, form listed in Kaikki17679.0%Qwen3.8 Flash (46.8%)Claude Fable 5.2 (93.4%)

Translation

Averaged over 9 models, the highest-scoring evidence group is ‘Parallel corpus’ (70.6%, 60 items) and the lowest-scoring is ‘Course only’ (66.7%, 196 items); the gap is clear (95% ranges do not overlap) for 1 of 9 models.

Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong.

  1. Parallel corpus60 items · corpus-referenceAverage over 9 models: 70.6%Claude Fable 5.2: 77.9%Gemini 4 Hydrogen: 75.3%GPT-6 Astra: 76.3%Gemini 4 Argon: 77.4%DeepSeek V4.1 Flash 1017: 68.3%Kimi K3.1 Pro: 69.9%GPT-6 Luna: 69.6%Claude Haiku 5.5: 62.3%Qwen3.8 Flash: 58.2%
  2. Course, form listed in Kaikki44 items · blas-app+kaikkiAverage over 9 models: 69.7%Claude Fable 5.2: 75.0%Gemini 4 Hydrogen: 73.3%GPT-6 Astra: 72.4%Gemini 4 Argon: 75.3%DeepSeek V4.1 Flash 1017: 71.2%Kimi K3.1 Pro: 71.7%GPT-6 Luna: 69.8%Claude Haiku 5.5: 60.7%Qwen3.8 Flash: 58.2%
  3. Course, marked unverified60 items · blas-app-unverifiedAverage over 9 models: 69.6%Claude Fable 5.2: 83.0%Gemini 4 Hydrogen: 79.1%GPT-6 Astra: 78.9%Gemini 4 Argon: 75.3%DeepSeek V4.1 Flash 1017: 71.2%Kimi K3.1 Pro: 71.6%GPT-6 Luna: 68.4%Claude Haiku 5.5: 50.3%Qwen3.8 Flash: 48.3%
  4. Course only196 items · blas-app-onlyAverage over 9 models: 66.7%Claude Fable 5.2: 71.8%Gemini 4 Hydrogen: 69.3%GPT-6 Astra: 70.5%Gemini 4 Argon: 68.5%DeepSeek V4.1 Flash 1017: 67.6%Kimi K3.1 Pro: 68.1%GPT-6 Luna: 67.0%Claude Haiku 5.5: 58.2%Qwen3.8 Flash: 58.9%
View as table
EvidenceItemsAverageLowest modelHighest model
Parallel corpus6070.6%Qwen3.8 Flash (58.2%)Claude Fable 5.2 (77.9%)
Course, form listed in Kaikki4469.7%Qwen3.8 Flash (58.2%)Gemini 4 Argon (75.3%)
Course, marked unverified6069.6%Qwen3.8 Flash (48.3%)Claude Fable 5.2 (83.0%)
Course only19666.7%Claude Haiku 5.5 (58.2%)Claude Fable 5.2 (71.8%)

Tutor accuracy

Averaged over 9 models, the highest-scoring evidence group is ‘Course only’ (84.4%, 60 items) and the lowest-scoring is ‘Built from exercise options’ (72.9%, 113 items); the gap is clear (95% ranges do not overlap) for 3 of 9 models.

Bar: average over models; ticks: single models. The groups also differ in what they contain, so a gap is not by itself proof that weaker gold is wrong. Left out, with fewer than 20 items: Planted typo.

  1. Course only60 items · blas-app-onlyAverage over 9 models: 84.4%Claude Fable 5.2: 92.2%Gemini 4 Hydrogen: 87.2%GPT-6 Astra: 85.6%Gemini 4 Argon: 87.8%DeepSeek V4.1 Flash 1017: 85.0%Kimi K3.1 Pro: 90.0%GPT-6 Luna: 87.2%Claude Haiku 5.5: 87.2%Qwen3.8 Flash: 57.8%
  2. Built from exercise options113 items · distractor-derived-unverifiedAverage over 9 models: 72.9%Claude Fable 5.2: 86.7%Gemini 4 Hydrogen: 87.9%GPT-6 Astra: 86.1%Gemini 4 Argon: 81.4%DeepSeek V4.1 Flash 1017: 69.9%Kimi K3.1 Pro: 72.3%GPT-6 Luna: 61.1%Claude Haiku 5.5: 54.0%Qwen3.8 Flash: 56.3%
View as table
EvidenceItemsAverageLowest modelHighest model
Course only6084.4%Qwen3.8 Flash (57.8%)Claude Fable 5.2 (92.2%)
Built from exercise options11372.9%Claude Haiku 5.5 (54.0%)Gemini 4 Hydrogen (87.9%)

Cost per correct answer

#

What a correct answer costs at OpenRouter prices on the day of the run.

Cost per 1,000 correct answers runs from $0.22 (GPT-6 Luna) to $11.49 (Gemini 4 Hydrogen), 51 times as much.

Suite cost, candidate and judges, divided by the number of full-score answers and shown per 1,000 answers; cheapest first. Log scale: each grid step multiplies the cost, so equal gaps are equal ratios. Score against cost per run is on the Models page.

  1. GPT-6 LunaGPT-6 Luna: $0.22 per 1,000 correct
  2. DeepSeek V4.1 Flash 1017DeepSeek V4.1 Flash 1017: $0.29 per 1,000 correct
  3. Claude Haiku 5.5Claude Haiku 5.5: $0.40 per 1,000 correct
  4. Gemini 4 ArgonGemini 4 Argon: $0.83 per 1,000 correct
  5. Qwen3.8 FlashQwen3.8 Flash: $3.70 per 1,000 correct
  6. GPT-6 AstraGPT-6 Astra: $7.10 per 1,000 correct
  7. Claude Fable 5.2Claude Fable 5.2: $8.14 per 1,000 correct
  8. Kimi K3.1 ProKimi K3.1 Pro: $10.14 per 1,000 correct
  9. Gemini 4 HydrogenGemini 4 Hydrogen: $11.49 per 1,000 correct
View as table
ModelScoreCost per runPer 1,000 correct
GPT-6 Luna75.6%$0.78$0.22
DeepSeek V4.1 Flash 101779.2%$1.04$0.29
Claude Haiku 5.567.1%$1.23$0.40
Gemini 4 Argon81.9%$3.10$0.83
Qwen3.8 Flash61.6%$10.45$3.70
GPT-6 Astra83.6%$26.95$7.10
Claude Fable 5.286.1%$31.79$8.14
Kimi K3.1 Pro78.4%$36.20$10.14
Gemini 4 Hydrogen84.4%$43.89$11.49

Consistency

#

Each item is asked several times. How often does a model get the same score every time?

Mean consistency runs from 50.6% (Qwen3.8 Flash) to 83.1% (Claude Fable 5.2).

Consistency is the share of items that got the same score in all 3 asks: a model that often changes its answer is less dependable for learners. Bar: mean over the tests; ticks: single tests (point at one to name it). Translation tests give partial credit, so exact repeats are rarer there.

  1. Claude Fable 5.2Mean over 5 tests: 83.1%Lesson exercises: 99.2%Word forms: 96.9%Translation: 70.8%Round trip: 55.8%Tutor accuracy: 92.8%
  2. GPT-6 AstraMean over 5 tests: 78.2%Lesson exercises: 99.4%Word forms: 94.2%Translation: 61.1%Round trip: 47.9%Tutor accuracy: 88.3%
  3. Gemini 4 HydrogenMean over 5 tests: 77.7%Lesson exercises: 99.2%Word forms: 92.2%Translation: 57.5%Round trip: 45.0%Tutor accuracy: 94.4%
  4. Gemini 4 ArgonMean over 5 tests: 72.5%Lesson exercises: 98.1%Word forms: 89.4%Translation: 51.7%Round trip: 32.1%Tutor accuracy: 91.1%
  5. DeepSeek V4.1 Flash 1017Mean over 5 tests: 68.4%Lesson exercises: 95.3%Word forms: 90.8%Translation: 46.9%Round trip: 27.5%Tutor accuracy: 81.7%
  6. GPT-6 LunaMean over 5 tests: 67.6%Lesson exercises: 96.1%Word forms: 85.6%Translation: 43.9%Round trip: 32.9%Tutor accuracy: 79.4%
  7. Kimi K3.1 ProMean over 5 tests: 65.3%Lesson exercises: 96.9%Word forms: 87.5%Translation: 42.2%Round trip: 22.5%Tutor accuracy: 77.2%
  8. Claude Haiku 5.5Mean over 5 tests: 57.8%Lesson exercises: 91.4%Word forms: 75.8%Translation: 32.8%Round trip: 18.3%Tutor accuracy: 70.6%
  9. Qwen3.8 FlashMean over 5 tests: 50.6%Lesson exercises: 92.2%Word forms: 69.4%Translation: 19.7%Round trip: 7.9%Tutor accuracy: 63.9%
View as table
ModelLesson exercisesWord formsTranslationRound tripTutor accuracyMeanScore
Claude Fable 5.299.2%96.9%70.8%55.8%92.8%83.1%86.1%
GPT-6 Astra99.4%94.2%61.1%47.9%88.3%78.2%83.6%
Gemini 4 Hydrogen99.2%92.2%57.5%45.0%94.4%77.7%84.4%
Gemini 4 Argon98.1%89.4%51.7%32.1%91.1%72.5%81.9%
DeepSeek V4.1 Flash 101795.3%90.8%46.9%27.5%81.7%68.4%79.2%
GPT-6 Luna96.1%85.6%43.9%32.9%79.4%67.6%75.6%
Kimi K3.1 Pro96.9%87.5%42.2%22.5%77.2%65.3%78.4%
Claude Haiku 5.591.4%75.8%32.8%18.3%70.6%57.8%67.1%
Qwen3.8 Flash92.2%69.4%19.7%7.9%63.9%50.6%61.6%

Score against consistency

#

Across 9 models, higher-scoring models tend to be more consistent: a strong link (Spearman rank correlation 0.97); this is descriptive, not causal.

Points are models, labelled by suite rank; whiskers are 95% score ranges. Crowded labels link back to their dots. x: share of items with the same score on all three asks, averaged equally over tests; y: overall suite score. Translation tests rarely repeat exactly, so consistency is lower there by construction. Consistency uses all suite tests. Models need three-ask consistency on every included test.

Score against consistency1. Claude Fable 5.2: 86.1%, consistency 83.1%; 2. Gemini 4 Hydrogen: 84.4%, consistency 77.7%; 3. GPT-6 Astra: 83.6%, consistency 78.2%; 4. Gemini 4 Argon: 81.9%, consistency 72.5%; 5. DeepSeek V4.1 Flash 1017: 79.2%, consistency 68.4%; 6. Kimi K3.1 Pro: 78.4%, consistency 65.3%; 7. GPT-6 Luna: 75.6%, consistency 67.6%; 8. Claude Haiku 5.5: 67.1%, consistency 57.8%; 9. Qwen3.8 Flash: 61.6%, consistency 50.6%50%60%70%80%90%100%0%25%50%75%100%Mean consistencyScoreQwen3.8 Flash9Claude Haiku 5.58GPT-6 Luna7Kimi K3.1 Pro6DeepSeek V4.1 Flash 10175Gemini 4 Argon4GPT-6 Astra3Gemini 4 Hydrogen2Claude Fable 5.21
  1. Claude Fable 5.2 83.1% consistency; 86.1% score
  2. Gemini 4 Hydrogen 77.7% consistency; 84.4% score
  3. GPT-6 Astra 78.2% consistency; 83.6% score
  4. Gemini 4 Argon 72.5% consistency; 81.9% score
  5. DeepSeek V4.1 Flash 1017 68.4% consistency; 79.2% score
  6. Kimi K3.1 Pro 65.3% consistency; 78.4% score
  7. GPT-6 Luna 67.6% consistency; 75.6% score
  8. Claude Haiku 5.5 57.8% consistency; 67.1% score
  9. Qwen3.8 Flash 50.6% consistency; 61.6% score
View as table

Link strength uses |Spearman rho|: ≥0.7 strong, ≥0.4 moderate, ≥0.2 weak; otherwise no clear link. Descriptive, not a significance test.

ModelMean consistencyOverall suite score
Claude Fable 5.283.1%86.1%
Gemini 4 Hydrogen77.7%84.4%
GPT-6 Astra78.2%83.6%
Gemini 4 Argon72.5%81.9%
DeepSeek V4.1 Flash 101768.4%79.2%
Kimi K3.1 Pro65.3%78.4%
GPT-6 Luna67.6%75.6%
Claude Haiku 5.557.8%67.1%
Qwen3.8 Flash50.6%61.6%

Accept-list tests only

Across 9 models, higher-scoring models tend to be more consistent: a strong link (Spearman rank correlation 0.98); this is descriptive, not causal.

Points are models, labelled by suite rank; whiskers are 95% score ranges. Crowded labels link back to their dots. x: share of items with the same score on all three asks, averaged equally over tests; y: overall suite score. Translation tests rarely repeat exactly, so consistency is lower there by construction. Consistency uses accept-list tests only; y still uses the overall suite score. Models need three-ask consistency on every included test.

Score against consistency1. Claude Fable 5.2: 86.1%, consistency 98.1%; 2. Gemini 4 Hydrogen: 84.4%, consistency 95.7%; 3. GPT-6 Astra: 83.6%, consistency 96.8%; 4. Gemini 4 Argon: 81.9%, consistency 93.8%; 5. DeepSeek V4.1 Flash 1017: 79.2%, consistency 93.1%; 6. Kimi K3.1 Pro: 78.4%, consistency 92.2%; 7. GPT-6 Luna: 75.6%, consistency 90.8%; 8. Claude Haiku 5.5: 67.1%, consistency 83.6%; 9. Qwen3.8 Flash: 61.6%, consistency 80.8%50%60%70%80%90%100%0%25%50%75%100%Mean consistencyScoreQwen3.8 Flash9Claude Haiku 5.58GPT-6 Luna7Kimi K3.1 Pro6DeepSeek V4.1 Flash 10175Gemini 4 Argon4GPT-6 Astra3Gemini 4 Hydrogen2Claude Fable 5.21
  1. Claude Fable 5.2 98.1% consistency; 86.1% score
  2. Gemini 4 Hydrogen 95.7% consistency; 84.4% score
  3. GPT-6 Astra 96.8% consistency; 83.6% score
  4. Gemini 4 Argon 93.8% consistency; 81.9% score
  5. DeepSeek V4.1 Flash 1017 93.1% consistency; 79.2% score
  6. Kimi K3.1 Pro 92.2% consistency; 78.4% score
  7. GPT-6 Luna 90.8% consistency; 75.6% score
  8. Claude Haiku 5.5 83.6% consistency; 67.1% score
  9. Qwen3.8 Flash 80.8% consistency; 61.6% score
View as table

Link strength uses |Spearman rho|: ≥0.7 strong, ≥0.4 moderate, ≥0.2 weak; otherwise no clear link. Descriptive, not a significance test.

ModelMean consistencyOverall suite score
Claude Fable 5.298.1%86.1%
Gemini 4 Hydrogen95.7%84.4%
GPT-6 Astra96.8%83.6%
Gemini 4 Argon93.8%81.9%
DeepSeek V4.1 Flash 101793.1%79.2%
Kimi K3.1 Pro92.2%78.4%
GPT-6 Luna90.8%75.6%
Claude Haiku 5.583.6%67.1%
Qwen3.8 Flash80.8%61.6%