Teanga Bench

Claude Fable 5.2

Anthropic · anthropic/claude-fable-5.2 · run 16 November 2026 · each item asked 3 times

Overall score
86.1%95% range 84.9–87.3%
Rank
1 of 9
Cost per run
$31.79this suite, judges included
Per 1,000 correct
$8.14cost of 1,000 full-score answers
Consistency
83.1%same score in all 3 asks, mean over tests

Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built

Scores by test

#

Claude Fable 5.2 ranks 1st in every test, but in every one another model is tied with it (95% ranges overlap).

Score on each test. Right: the score and its place among the ranked models. Select a test to compare every model.

  • This model, with its 95% range
  • Each other ranked model
  1. Lesson exercisesGemini 4 Hydrogen: 97.3%GPT-6 Astra: 96.6%Gemini 4 Argon: 98.2%DeepSeek V4.1 Flash 1017: 95.6%Kimi K3.1 Pro: 95.8%GPT-6 Luna: 93.6%Claude Haiku 5.5: 87.5%Qwen3.8 Flash: 90.6%Claude Fable 5.2: 98.5%, 95% range 97.1% to 99.5%, 1st of 9
  2. Word formsGemini 4 Hydrogen: 91.1%GPT-6 Astra: 92.5%Gemini 4 Argon: 85.4%DeepSeek V4.1 Flash 1017: 87.0%Kimi K3.1 Pro: 82.1%GPT-6 Luna: 80.5%Claude Haiku 5.5: 65.3%Qwen3.8 Flash: 50.6%Claude Fable 5.2: 94.1%, 95% range 91.7% to 96.2%, 1st of 9
  3. TranslationGemini 4 Hydrogen: 72.4%GPT-6 Astra: 73.1%Gemini 4 Argon: 71.9%DeepSeek V4.1 Flash 1017: 68.8%Kimi K3.1 Pro: 69.4%GPT-6 Luna: 68.0%Claude Haiku 5.5: 57.9%Qwen3.8 Flash: 56.9%Claude Fable 5.2: 75.1%, 95% range 72.5% to 77.6%, 1st of 9
  4. Round tripGemini 4 Hydrogen: 73.0%GPT-6 Astra: 69.3%Gemini 4 Argon: 70.0%DeepSeek V4.1 Flash 1017: 68.5%Kimi K3.1 Pro: 65.3%GPT-6 Luna: 64.5%Claude Haiku 5.5: 58.0%Qwen3.8 Flash: 52.2%Claude Fable 5.2: 74.3%, 95% range 71.7% to 77.0%, 1st of 9
  5. Tutor accuracyGemini 4 Hydrogen: 88.1%GPT-6 Astra: 86.5%Gemini 4 Argon: 84.1%DeepSeek V4.1 Flash 1017: 76.1%Kimi K3.1 Pro: 79.3%GPT-6 Luna: 71.3%Claude Haiku 5.5: 66.9%Qwen3.8 Flash: 58.0%Claude Fable 5.2: 88.5%, 95% range 84.1% to 92.4%, 1st of 9
View as table
TestScore and 95% rangePlaceHighestMedian model
Lesson exercises98.5% 97.1–99.5%1st of 9Claude Fable 5.2 (98.5%)95.8%
Word forms94.1% 91.7–96.2%1st of 9Claude Fable 5.2 (94.1%)85.4%
Translation75.1% 72.5–77.6%1st of 9Claude Fable 5.2 (75.1%)69.4%
Round trip74.3% 71.7–77.0%1st of 9Claude Fable 5.2 (74.3%)68.5%
Tutor accuracy88.5% 84.1–92.4%1st of 9Claude Fable 5.2 (88.5%)79.3%

Claude Fable 5.2 ranks 1st in every test, but in every one another model is tied with it (95% ranges overlap).

One axis per test, with the same scale on every profile: 0% at the centre to 100% at the outer ring. Solid accent: this model; faint dashed polygon: median model score on each test; faint dotted outer polygon: best score on each test (not necessarily one model). Vertex labels show this model’s scores. Compare scores, not polygon area; no confidence intervals are shown.

Claude Fable 5.2 test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 98.5%, median 95.8%, best 98.5%; Word forms: 94.1%, median 85.4%, best 94.1%; Translation: 75.1%, median 69.4%, best 75.1%; Round trip: 74.3%, median 68.5%, best 74.3%; Tutor accuracy: 88.5%, median 79.3%, best 88.5%98.5%Lessonexercises94.1%Word forms75.1%Translation74.3%Round trip88.5%Tutor accuracy
View as table
TestThis modelMedianBest
Lesson exercises98.5%95.8%98.5%
Word forms94.1%85.4%94.1%
Translation75.1%69.4%75.1%
Round trip74.3%68.5%74.3%
Tutor accuracy88.5%79.3%88.5%

Items, consistency and cost by test

Consistency is the share of items that got the same score in every repeat ask.

TestScore95% rangeItemsConsistencyCostPer 1,000 correct
Lesson exercises98.5%97.1–99.5%36099.2%$3.96$3.73
Word forms94.1%91.7–96.2%36096.9%$6.18$6.08
Translation75.1%72.5–77.6%36070.8%$6.24$7.69
Round trip74.3%71.7–77.0%24055.8%$7.62$14.23
Tutor accuracy88.5%84.1–92.4%18092.8%$7.79$16.30

By language

Score, 95% range and number of items (n) for each test.

LanguageLesson exercisesWord formsTranslationRound tripTutor accuracy
Irish97.2% 92.2–100.0% · n=6096.7% 91.7–100.0% · n=6082.0% 77.3–86.4% · n=6082.0% 76.8–86.7% · n=4091.1% 83.9–96.7% · n=60
Welsh98.9% 97.2–100.0% · n=6097.8% 93.9–100.0% · n=6083.8% 79.0–88.0% · n=6083.6% 77.9–88.6% · n=4093.3% 87.2–98.3% · n=60
Scottish Gaelic100.0% 100.0–100.0% · n=6096.7% 91.7–100.0% · n=6077.3% 71.0–83.5% · n=6077.8% 72.5–83.0% · n=4081.1% 71.1–89.4% · n=60
Manx96.7% 91.7–100.0% · n=6086.1% 77.2–93.9% · n=6052.5% 46.4–59.2% · n=6064.1% 59.0–69.7% · n=40n/a
Breton98.3% 95.0–100.0% · n=6097.8% 93.9–100.0% · n=6071.9% 65.8–77.5% · n=6065.7% 59.1–72.5% · n=40n/a
Cornish100.0% 100.0–100.0% · n=6089.4% 82.2–95.6% · n=6083.0% 77.3–88.4% · n=6072.9% 66.1–79.6% · n=40n/a

By CEFR level

The CEFR level comes from the course content; items without one are listed as unspecified.

Score, 95% range and number of items (n) for each test.

CEFR levelLesson exercisesWord formsTranslationRound tripTutor accuracy
A199.7% 99.2–100.0% · n=12592.8% 88.3–96.8% · n=13479.2% 74.5–83.8% · n=11272.2% 63.0–81.4% · n=2597.0% 90.9–100.0% · n=22
A297.8% 95.3–99.8% · n=13592.9% 88.6–96.4% · n=14071.8% 66.6–77.1% · n=9177.2% 72.7–81.6% · n=8892.6% 82.7–100.0% · n=27
B197.3% 93.2–100.0% · n=7398.3% 95.7–100.0% · n=3967.3% 60.5–73.6% · n=4769.7% 65.5–73.6% · n=7984.1% 73.0–92.9% · n=42
B2100.0% 100.0–100.0% · n=2493.8% 81.2–100.0% · n=1668.7% 59.0–78.0% · n=3072.8% 65.3–80.3% · n=2784.1% 69.6–97.1% · n=23
C1100.0% 100.0–100.0% · n=3n/a86.9% 80.5–92.5% · n=1481.4% 73.9–89.0% · n=1580.0% 56.7–100.0% · n=10
C2n/an/a86.0% 75.9–95.4% · n=691.9% 84.5–97.7% · n=6100.0% 100.0–100.0% · n=2
Unspecifiedn/a100.0% 100.0–100.0% · n=3177.9% 72.8–82.8% · n=60n/a89.5% 81.5–96.3% · n=54

Grammar topics

The topic is the grammar point or error type recorded on each item.

Lesson exercises

Strongest 5 topics.

TopicScoreItems
adjectives100.0%5
chapter overview100.0%11
conversation100.0%6
daily life100.0%5
food100.0%5

Weakest 5 topics.

TopicScoreItems
narrative grammar80.0%5
vocabulary90.5%7
grammar95.2%21
unit focus98.0%51
guide99.0%32

Only topics with at least 5 items are ranked.

Word forms

Strongest 5 topics.

TopicScoreItems
article gender mutation100.0%31
verb past100.0%22
verb pluperfect100.0%5
verb preterite100.0%5
verb conditional98.6%23

Weakest 5 topics.

TopicScoreItems
number counting80.0%5
noun plural85.6%30
verb future87.0%23
noun genitive88.9%9
number personal88.9%9

Only topics with at least 5 items are ranked.

Translation

All topics, highest score first.

TopicScoreItems
legal or administrative77.9%60
to english77.6%150
from english71.4%150

Tutor accuracy

All topics, highest score first.

TopicScoreItems
unit focus100.0%10
guide95.0%20
grammar91.3%42
no error91.1%30
vocabulary86.7%15
orthography doubled letter85.7%7
grammar point83.3%12
application72.7%11

Only topics with at least 5 items are ranked.

Tutor explanations

judge-marked, secondary

Tutor items also ask for a short explanation. Two judge models from other model families mark it. This is a secondary number: it is judge-marked and not human-validated.

Tutor accuracy

Headline score
88.5%: found the error, fixed it and changed nothing else (rule-only, no judge).
Parts of the headline
Found 88.5% · fixed 88.5% · no over-correction 93.0%
Explanation score
78.5% (95% range 72.9–84.1%), judge-marked, not human-validated. An explanation passes only when both judges pass it.
Judge agreement
93.5% of 508 judged answers
Unscored
32 answers a judge could not mark, after 32 failed judge calls; they are left out, not scored 0.
Judges
deepseek/deepseek-v4.1-flash-1017, qwen/qwen3.8-flash

Cost and prices

This suite
$31.79, of which $0.38 was judge spend
Whole run
$46.32 for both suites, against a cap of $90.00
Cost per correct answer
$0.00814 ($8.14 per 1,000): suite cost divided by the number of full-score answers
Run date
16 November 2026
Prices for anthropic/claude-fable-5.2
$8.00 input, $40.00 output, $0.20 cache read, $10.00 cache write per million tokens. Retrieved 16 November 2026 from https://openrouter.ai/api/v1/models.
Judge deepseek/deepseek-v4.1-flash-1017
$0.15 input, $0.60 output, $0.0030 cache read per million tokens. Retrieved 16 November 2026 from OpenRouter.
Judge qwen/qwen3.8-flash
$0.15 input, $0.47 output, $0.0160 cache read per million tokens. Retrieved 16 November 2026 from OpenRouter.

Failure examples

10

The lowest-scoring items from the public samples, with the worst of the repeats. Private items are never shown. Model output is shown exactly as text.

Word forms

blas-morph:manx:number_ordinal:gv-num-12

Manx · number ordinal · score 0.00

Prompt

Language: Manx
Task: Give the ordinal numeral for the number below: the ordinal ('first', 'second', ...), including the definite article yn as it stands before the ordinal.
Number: 12 (twelve)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

yn nah jeig

Model output

ANSWER: yn daa-yeigoo

blas-morph:scots-gaelic:number_counting:gd-num-2

Scottish Gaelic · number counting · score 0.00

Prompt

Language: Scottish Gaelic
Task: Give the counting numeral for the number below: the form used when counting aloud (not before a noun).
Number: 2 (two)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

dhà

Model output

ANSWER: a dhà

Translation

blas-translation-scots-gaelic-gd-sent-0223-from_english

Scottish Gaelic · from english · score 0.15

Prompt

Translate this English sentence into Scottish Gaelic.
Reply with the Scottish Gaelic translation only, on one line, with no notes or explanation.

English: He is a good carpenter.

Expected

Tha e na dheagh shaor.

Model output

'S e saor math a th' ann.

blas-translation-manx-gv-sent-parallel-4509-from_english

Manx · from english · score 0.18

Prompt

Translate this English sentence into Manx.
Reply with the Manx translation only, on one line, with no notes or explanation.

English: I don't know where to turn.

Expected

Cha nel fys aym cre dy yannoo.

Model output

Cha s'aym c'raad dy hyndaa.

blas-translation-manx-gv-sent-parallel-15649-from_english

Manx · from english · score 0.41

Prompt

Translate this English sentence into Manx.
Reply with the Manx translation only, on one line, with no notes or explanation.

English: Have you made my bed?

Expected

Vel shiu er chiartaghey my lhiabbee?

Model output

Vel oo er n'yannoo my lhiabbee?

Round trip

blas-round_trip-manx-gv-sent-parallel-6412

Manx · round trip · score 0.22

Prompt

Translate this Manx sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Manx: Nee eh costal queig keead stiagh as magh.

Expected

Nee eh costal queig keead stiagh as magh.

Model output

Costee eh mysh queig cheead, ny smoo ny ny sloo.

Pivot text

It will cost about five hundred, more or less.

blas-round_trip-cornish-kw-sent-tat-2363

Cornish · round trip · score 0.27

Prompt

Translate this Cornish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Cornish: My re bia owth oberi ena rag tri mis.

Expected

My re bia owth oberi ena rag tri mis.

Model output

Yth esen vy ow kweytha ena nans o tri mis.

Pivot text

I had been working there for three months.

blas-round_trip-cornish-kw-sent-tat-1830

Cornish · round trip · score 0.26

Prompt

Translate this Cornish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Cornish: Kas yw genev an traow na.

Expected

Kas yw genev an traow na.

Model output

My a gas an taclow na.

Pivot text

I hate those things.

Tutor accuracy

blas-tutor-welsh-exd-d6fa3b01fb

Welsh · grammar · score 0.00

Prompt

You are a Welsh language tutor checking a sentence written by a learner.
Language: Welsh
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Byddai yna adar y tu allan er i fyddai merlod na defaid.
Reply with one JSON object and nothing …

Expected

Byddai yna adar y tu allan er na fyddai merlod na defaid.

Model output

{"has_error": true, "error_span": "er i", "correction": "er na", "corrected_sentence": "Byddai yna adar y tu allan er na fyddai merlod na defaid.", "explanation": "A negative clause after 'er' with a conjugated verb takes the negative particle 'na' (er na fyddai…), not the preposition 'i'. 'Er i' is only used before a subject and verbnoun (e.g. 'er i mi fynd'), not before a conjugated verb like 'fyddai'."}

blas-tutor-welsh-typo-6085cad894

Welsh · orthography doubled letter · score 0.00

Prompt

You are a Welsh language tutor checking a sentence written by a learner.
Language: Welsh
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: GGeiriadur termau sy'n ymwneud â therapi galwedigaethol gan Delyth Prys a Owain Lloyd Davies (Go…

Expected

Geiriadur termau sy'n ymwneud â therapi galwedigaethol gan Delyth Prys a Owain Lloyd Davies (Golygyddion) yw Termau Therapi Galwedigaethol.

Model output

{"has_error": true, "error_span": "a Owain", "correction": "ac Owain", "corrected_sentence": "GGeiriadur termau sy'n ymwneud â therapi galwedigaethol gan Delyth Prys ac Owain Lloyd Davies (Golygyddion) yw Termau Therapi Galwedigaethol.", "explanation": "The conjunction 'a' (and) becomes 'ac' before a vowel, so it must be 'ac Owain'. Separately, 'GGeiriadur' at the start looks like a typo for 'Geiriadur', but I left it unchanged because only one correction is allowed."}