Teanga Bench

Models

9 models ranked. Select a model for every score, breakdown and cost.

Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built

  1. #1Claude Fable 5.2Anthropic · anthropic/claude-fable-5.286.1%95% range 84.9–87.3%$31.79 per run$8.14 per 1,000 correct
  2. #2Gemini 4 HydrogenGoogle · google/gemini-4-hydrogen84.4%95% range 83.1–85.6%$43.89 per run$11.49 per 1,000 correct
  3. #3GPT-6 AstraOpenAI · openai/gpt-6-astra83.6%95% range 82.3–84.9%$26.95 per run$7.10 per 1,000 correct
  4. #4Gemini 4 ArgonGoogle · google/gemini-4-argon81.9%95% range 80.5–83.3%$3.10 per run$0.83 per 1,000 correct
  5. #5DeepSeek V4.1 Flash 1017DeepSeek · deepseek/deepseek-v4.1-flash-101779.2%95% range 77.7–80.7%$1.04 per run$0.29 per 1,000 correct
  6. #6Kimi K3.1 ProMoonshot AI · moonshotai/kimi-k3.1-pro78.4%95% range 76.9–79.9%$36.20 per run$10.14 per 1,000 correct
  7. #7GPT-6 LunaOpenAI · openai/gpt-6-luna75.6%95% range 73.9–77.2%$0.78 per run$0.22 per 1,000 correct
  8. #8Claude Haiku 5.5Anthropic · anthropic/claude-haiku-5.567.1%95% range 65.3–68.9%$1.23 per run$0.40 per 1,000 correct
  9. #9Qwen3.8 FlashQwen · qwen/qwen3.8-flash61.6%95% range 59.8–63.4%$10.45 per run$3.70 per 1,000 correct

Model profiles

#

Claude Fable 5.2 has the highest score in 5 of 5 tests, but in every test the top score is tied with at least one other model (95% ranges overlap).

One axis per test, with the same scale on every profile: 0% at the centre to 100% at the outer ring. Solid accent: this model; faint dashed polygon: median model score on each test; faint dotted outer polygon: best score on each test (not necessarily one model). Vertex labels show this model’s scores. Compare scores, not polygon area; no confidence intervals are shown. Select a profile to open the model page; ordered by suite rank.

  1. Claude Fable 5.2Claude Fable 5.2 test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 98.5%, median 95.8%, best 98.5%; Word forms: 94.1%, median 85.4%, best 94.1%; Translation: 75.1%, median 69.4%, best 75.1%; Round trip: 74.3%, median 68.5%, best 74.3%; Tutor accuracy: 88.5%, median 79.3%, best 88.5%98.5%Lessonexercises94.1%Word forms75.1%Translation74.3%Round trip88.5%Tutor accuracy
  2. Gemini 4 HydrogenGemini 4 Hydrogen test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 97.3%, median 95.8%, best 98.5%; Word forms: 91.1%, median 85.4%, best 94.1%; Translation: 72.4%, median 69.4%, best 75.1%; Round trip: 73.0%, median 68.5%, best 74.3%; Tutor accuracy: 88.1%, median 79.3%, best 88.5%97.3%Lessonexercises91.1%Word forms72.4%Translation73.0%Round trip88.1%Tutor accuracy
  3. GPT-6 AstraGPT-6 Astra test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 96.6%, median 95.8%, best 98.5%; Word forms: 92.5%, median 85.4%, best 94.1%; Translation: 73.1%, median 69.4%, best 75.1%; Round trip: 69.3%, median 68.5%, best 74.3%; Tutor accuracy: 86.5%, median 79.3%, best 88.5%96.6%Lessonexercises92.5%Word forms73.1%Translation69.3%Round trip86.5%Tutor accuracy
  4. Gemini 4 ArgonGemini 4 Argon test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 98.2%, median 95.8%, best 98.5%; Word forms: 85.4%, median 85.4%, best 94.1%; Translation: 71.9%, median 69.4%, best 75.1%; Round trip: 70.0%, median 68.5%, best 74.3%; Tutor accuracy: 84.1%, median 79.3%, best 88.5%98.2%Lessonexercises85.4%Word forms71.9%Translation70.0%Round trip84.1%Tutor accuracy
  5. DeepSeek V4.1 Flash 1017DeepSeek V4.1 Flash 1017 test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 95.6%, median 95.8%, best 98.5%; Word forms: 87.0%, median 85.4%, best 94.1%; Translation: 68.8%, median 69.4%, best 75.1%; Round trip: 68.5%, median 68.5%, best 74.3%; Tutor accuracy: 76.1%, median 79.3%, best 88.5%95.6%Lessonexercises87.0%Word forms68.8%Translation68.5%Round trip76.1%Tutor accuracy
  6. Kimi K3.1 ProKimi K3.1 Pro test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 95.8%, median 95.8%, best 98.5%; Word forms: 82.1%, median 85.4%, best 94.1%; Translation: 69.4%, median 69.4%, best 75.1%; Round trip: 65.3%, median 68.5%, best 74.3%; Tutor accuracy: 79.3%, median 79.3%, best 88.5%95.8%Lessonexercises82.1%Word forms69.4%Translation65.3%Round trip79.3%Tutor accuracy
  7. GPT-6 LunaGPT-6 Luna test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 93.6%, median 95.8%, best 98.5%; Word forms: 80.5%, median 85.4%, best 94.1%; Translation: 68.0%, median 69.4%, best 75.1%; Round trip: 64.5%, median 68.5%, best 74.3%; Tutor accuracy: 71.3%, median 79.3%, best 88.5%93.6%Lessonexercises80.5%Word forms68.0%Translation64.5%Round trip71.3%Tutor accuracy
  8. Claude Haiku 5.5Claude Haiku 5.5 test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 87.5%, median 95.8%, best 98.5%; Word forms: 65.3%, median 85.4%, best 94.1%; Translation: 57.9%, median 69.4%, best 75.1%; Round trip: 58.0%, median 68.5%, best 74.3%; Tutor accuracy: 66.9%, median 79.3%, best 88.5%87.5%Lessonexercises65.3%Word forms57.9%Translation58.0%Round trip66.9%Tutor accuracy
  9. Qwen3.8 FlashQwen3.8 Flash test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 90.6%, median 95.8%, best 98.5%; Word forms: 50.6%, median 85.4%, best 94.1%; Translation: 56.9%, median 69.4%, best 75.1%; Round trip: 52.2%, median 68.5%, best 74.3%; Tutor accuracy: 58.0%, median 79.3%, best 88.5%90.6%Lessonexercises50.6%Word forms56.9%Translation52.2%Round trip58.0%Tutor accuracy
View as table
ModelLesson exercisesWord formsTranslationRound tripTutor accuracy
Claude Fable 5.298.5%94.1%75.1%74.3%88.5%
Gemini 4 Hydrogen97.3%91.1%72.4%73.0%88.1%
GPT-6 Astra96.6%92.5%73.1%69.3%86.5%
Gemini 4 Argon98.2%85.4%71.9%70.0%84.1%
DeepSeek V4.1 Flash 101795.6%87.0%68.8%68.5%76.1%
Kimi K3.1 Pro95.8%82.1%69.4%65.3%79.3%
GPT-6 Luna93.6%80.5%68.0%64.5%71.3%
Claude Haiku 5.587.5%65.3%57.9%58.0%66.9%
Qwen3.8 Flash90.6%50.6%56.9%52.2%58.0%

Score against cost

#

GPT-6 Astra is the cheapest model tied with the leader (95% ranges overlap): 83.6% for $26.95, against 86.1% for $31.79 from Claude Fable 5.2.

Each circle is a model, numbered by rank. Higher is better; further left is cheaper. The vertical line is the 95% range. Cost is one run of this suite, judges included.

Overall score against cost, Blas suite1. Claude Fable 5.2: 86.1% for $31.79; 2. Gemini 4 Hydrogen: 84.4% for $43.89; 3. GPT-6 Astra: 83.6% for $26.95; 4. Gemini 4 Argon: 81.9% for $3.10; 5. DeepSeek V4.1 Flash 1017: 79.2% for $1.04; 6. Kimi K3.1 Pro: 78.4% for $36.20; 7. GPT-6 Luna: 75.6% for $0.78; 8. Claude Haiku 5.5: 67.1% for $1.23; 9. Qwen3.8 Flash: 61.6% for $10.4550%60%70%80%90%100%$0.10$0.50$1$5$10$50$100Cost of one run of this suite (USD, log scale)ScoreQwen3.8 Flash9Claude Haiku 5.58GPT-6 Luna7Kimi K3.1 Pro6DeepSeek V4.1 Flash 10175Gemini 4 Argon4GPT-6 Astra3Gemini 4 Hydrogen2Claude Fable 5.21
  1. Claude Fable 5.2 86.1% · $31.79
  2. Gemini 4 Hydrogen 84.4% · $43.89
  3. GPT-6 Astra 83.6% · $26.95
  4. Gemini 4 Argon 81.9% · $3.10
  5. DeepSeek V4.1 Flash 1017 79.2% · $1.04
  6. Kimi K3.1 Pro 78.4% · $36.20
  7. GPT-6 Luna 75.6% · $0.78
  8. Claude Haiku 5.5 67.1% · $1.23
  9. Qwen3.8 Flash 61.6% · $10.45
View as table
ModelRankScore95% rangeCost per run
Claude Fable 5.2186.1%84.9–87.3%$31.79
Gemini 4 Hydrogen284.4%83.1–85.6%$43.89
GPT-6 Astra383.6%82.3–84.9%$26.95
Gemini 4 Argon481.9%80.5–83.3%$3.10
DeepSeek V4.1 Flash 1017579.2%77.7–80.7%$1.04
Kimi K3.1 Pro678.4%76.9–79.9%$36.20
GPT-6 Luna775.6%73.9–77.2%$0.78
Claude Haiku 5.5867.1%65.3–68.9%$1.23
Qwen3.8 Flash961.6%59.8–63.4%$10.45