Teanga Bench

Claude Haiku 5.5

Anthropic · anthropic/claude-haiku-5.5 · run 16 November 2026 · each item asked 3 times

Overall score
67.1%95% range 65.3–68.9%
Rank
8 of 9
Cost per run
$1.23this suite, judges included
Per 1,000 correct
$0.40cost of 1,000 full-score answers
Consistency
57.8%same score in all 3 asks, mean over tests

Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built

Scores by test

#

Claude Haiku 5.5 places best in Tutor accuracy (8th of 9) and worst in Lesson exercises (9th of 9, 11.0 points below the top).

Score on each test. Right: the score and its place among the ranked models. Select a test to compare every model.

  • This model, with its 95% range
  • Each other ranked model
  1. Lesson exercisesClaude Fable 5.2: 98.5%Gemini 4 Hydrogen: 97.3%GPT-6 Astra: 96.6%Gemini 4 Argon: 98.2%DeepSeek V4.1 Flash 1017: 95.6%Kimi K3.1 Pro: 95.8%GPT-6 Luna: 93.6%Qwen3.8 Flash: 90.6%Claude Haiku 5.5: 87.5%, 95% range 84.4% to 90.5%, 9th of 9
  2. Word formsClaude Fable 5.2: 94.1%Gemini 4 Hydrogen: 91.1%GPT-6 Astra: 92.5%Gemini 4 Argon: 85.4%DeepSeek V4.1 Flash 1017: 87.0%Kimi K3.1 Pro: 82.1%GPT-6 Luna: 80.5%Qwen3.8 Flash: 50.6%Claude Haiku 5.5: 65.3%, 95% range 60.9% to 69.4%, 8th of 9
  3. TranslationClaude Fable 5.2: 75.1%Gemini 4 Hydrogen: 72.4%GPT-6 Astra: 73.1%Gemini 4 Argon: 71.9%DeepSeek V4.1 Flash 1017: 68.8%Kimi K3.1 Pro: 69.4%GPT-6 Luna: 68.0%Qwen3.8 Flash: 56.9%Claude Haiku 5.5: 57.9%, 95% range 55.1% to 60.9%, 8th of 9
  4. Round tripClaude Fable 5.2: 74.3%Gemini 4 Hydrogen: 73.0%GPT-6 Astra: 69.3%Gemini 4 Argon: 70.0%DeepSeek V4.1 Flash 1017: 68.5%Kimi K3.1 Pro: 65.3%GPT-6 Luna: 64.5%Qwen3.8 Flash: 52.2%Claude Haiku 5.5: 58.0%, 95% range 54.8% to 61.2%, 8th of 9
  5. Tutor accuracyClaude Fable 5.2: 88.5%Gemini 4 Hydrogen: 88.1%GPT-6 Astra: 86.5%Gemini 4 Argon: 84.1%DeepSeek V4.1 Flash 1017: 76.1%Kimi K3.1 Pro: 79.3%GPT-6 Luna: 71.3%Qwen3.8 Flash: 58.0%Claude Haiku 5.5: 66.9%, 95% range 60.9% to 72.6%, 8th of 9
View as table
TestScore and 95% rangePlaceHighestMedian model
Lesson exercises87.5% 84.4–90.5%9th of 9Claude Fable 5.2 (98.5%)95.8%
Word forms65.3% 60.9–69.4%8th of 9Claude Fable 5.2 (94.1%)85.4%
Translation57.9% 55.1–60.9%8th of 9Claude Fable 5.2 (75.1%)69.4%
Round trip58.0% 54.8–61.2%8th of 9Claude Fable 5.2 (74.3%)68.5%
Tutor accuracy66.9% 60.9–72.6%8th of 9Claude Fable 5.2 (88.5%)79.3%

Claude Haiku 5.5 places best in Tutor accuracy (8th of 9) and worst in Lesson exercises (9th of 9, 11.0 points below the top).

One axis per test, with the same scale on every profile: 0% at the centre to 100% at the outer ring. Solid accent: this model; faint dashed polygon: median model score on each test; faint dotted outer polygon: best score on each test (not necessarily one model). Vertex labels show this model’s scores. Compare scores, not polygon area; no confidence intervals are shown.

Claude Haiku 5.5 test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 87.5%, median 95.8%, best 98.5%; Word forms: 65.3%, median 85.4%, best 94.1%; Translation: 57.9%, median 69.4%, best 75.1%; Round trip: 58.0%, median 68.5%, best 74.3%; Tutor accuracy: 66.9%, median 79.3%, best 88.5%87.5%Lessonexercises65.3%Word forms57.9%Translation58.0%Round trip66.9%Tutor accuracy
View as table
TestThis modelMedianBest
Lesson exercises87.5%95.8%98.5%
Word forms65.3%85.4%94.1%
Translation57.9%69.4%75.1%
Round trip58.0%68.5%74.3%
Tutor accuracy66.9%79.3%88.5%

Items, consistency and cost by test

Consistency is the share of items that got the same score in every repeat ask.

TestScore95% rangeItemsConsistencyCostPer 1,000 correct
Lesson exercises87.5%84.4–90.5%36091.4%$0.13$0.14
Word forms65.3%60.9–69.4%36075.8%$0.22$0.32
Translation57.9%55.1–60.9%36032.8%$0.15$0.25
Round trip58.0%54.8–61.2%24018.3%$0.21$0.51
Tutor accuracy66.9%60.9–72.6%18070.6%$0.51$1.41

By language

Score, 95% range and number of items (n) for each test.

LanguageLesson exercisesWord formsTranslationRound tripTutor accuracy
Irish87.2% 78.3–94.4% · n=6075.6% 66.1–84.4% · n=6069.7% 64.4–74.9% · n=6068.9% 62.1–75.4% · n=4066.7% 56.1–76.7% · n=60
Welsh91.7% 85.0–97.8% · n=6090.0% 83.3–95.6% · n=6077.1% 72.1–82.0% · n=6083.0% 78.6–87.3% · n=4071.7% 62.2–81.1% · n=60
Scottish Gaelic90.6% 82.2–97.2% · n=6073.3% 63.9–82.2% · n=6070.5% 64.0–76.5% · n=6069.5% 64.0–75.4% · n=4062.2% 51.7–72.2% · n=60
Manx79.4% 70.0–87.8% · n=6042.2% 32.2–53.3% · n=6026.5% 23.0–30.2% · n=6031.6% 28.1–35.2% · n=40n/a
Breton88.3% 80.6–94.4% · n=6053.9% 42.8–64.4% · n=6052.9% 46.5–59.4% · n=6047.9% 41.8–53.9% · n=40n/a
Cornish87.8% 80.5–95.0% · n=6056.7% 45.6–66.7% · n=6050.3% 44.4–56.2% · n=6047.2% 40.1–54.1% · n=40n/a

By CEFR level

The CEFR level comes from the course content; items without one are listed as unspecified.

Score, 95% range and number of items (n) for each test.

CEFR levelLesson exercisesWord formsTranslationRound tripTutor accuracy
A188.5% 83.2–93.3% · n=12571.1% 63.9–77.6% · n=13460.3% 54.7–66.0% · n=11256.6% 46.3–67.3% · n=2575.8% 60.6–89.4% · n=22
A286.7% 81.2–91.6% · n=13561.9% 55.0–68.8% · n=14050.9% 45.4–56.6% · n=9159.0% 53.9–64.4% · n=8865.4% 49.4–81.5% · n=27
B186.3% 79.0–92.7% · n=7345.3% 32.5–59.0% · n=3955.3% 47.7–63.1% · n=4752.5% 47.5–57.6% · n=7961.1% 48.4–73.0% · n=42
B288.9% 75.0–100.0% · n=2466.7% 43.8–87.5% · n=1654.2% 43.2–64.3% · n=3057.5% 48.3–67.4% · n=2756.5% 37.7–73.9% · n=23
C1100.0% 100.0–100.0% · n=3n/a72.7% 61.9–81.6% · n=1470.2% 59.3–80.3% · n=1556.7% 36.7–76.7% · n=10
C2n/an/a76.5% 66.2–86.8% · n=694.1% 86.6–99.5% · n=6100.0% 100.0–100.0% · n=2
Unspecifiedn/a79.6% 68.8–89.2% · n=3162.3% 56.9–67.2% · n=60n/a73.5% 63.0–83.3% · n=54

Grammar topics

The topic is the grammar point or error type recorded on each item.

Lesson exercises

Strongest 5 topics.

TopicScoreItems
adjectives100.0%5
health100.0%5
nouns100.0%6
pronouns100.0%5
review100.0%8

Weakest 5 topics.

TopicScoreItems
syntax60.0%5
daily life73.3%5
vocabulary76.2%7
grammar79.4%21
grammar point80.0%10

Only topics with at least 5 items are ranked.

Word forms

Strongest 5 topics.

TopicScoreItems
verb pluperfect100.0%5
number personal92.6%9
verb present79.7%23
article gender mutation79.6%31
verb preterite73.3%5

Weakest 5 topics.

TopicScoreItems
verb conditional36.2%23
verb past43.9%22
number counting53.3%5
prepositional pronoun60.7%28
noun plural61.1%30

Only topics with at least 5 items are ranked.

Translation

All topics, highest score first.

TopicScoreItems
to english62.3%150
legal or administrative62.3%60
from english51.7%150

Tutor accuracy

All topics, highest score first.

TopicScoreItems
orthography doubled letter100.0%7
no error82.2%30
unit focus76.7%10
guide70.0%20
grammar61.1%42
vocabulary57.8%15
grammar point47.2%12
application42.4%11

Only topics with at least 5 items are ranked.

Tutor explanations

judge-marked, secondary

Tutor items also ask for a short explanation. Two judge models from other model families mark it. This is a secondary number: it is judge-marked and not human-validated.

Tutor accuracy

Headline score
66.9%: found the error, fixed it and changed nothing else (rule-only, no judge).
Parts of the headline
Found 68.1% · fixed 67.0% · no over-correction 76.5%
Explanation score
59.6% (95% range 53.1–65.8%), judge-marked, not human-validated. An explanation passes only when both judges pass it.
Judge agreement
95.2% of 518 judged answers
Unscored
20 answers a judge could not mark, after 20 failed judge calls; they are left out, not scored 0.
Judges
deepseek/deepseek-v4.1-flash-1017, qwen/qwen3.8-flash

Cost and prices

This suite
$1.23, of which $0.33 was judge spend
Whole run
$1.90 for both suites, against a cap of $3.00
Cost per correct answer
$0.000403 ($0.40 per 1,000): suite cost divided by the number of full-score answers
Run date
16 November 2026
Prices for anthropic/claude-haiku-5.5
$0.1000 input, $0.50 output, $0.0100 cache read, $0.12 cache write per million tokens. Retrieved 16 November 2026 from https://openrouter.ai/api/v1/models.
Judge deepseek/deepseek-v4.1-flash-1017
$0.30 input, $1.20 output, $0.0060 cache read per million tokens. Retrieved 16 November 2026 from OpenRouter.
Judge qwen/qwen3.8-flash
$0.15 input, $0.47 output, $0.0160 cache read per million tokens. Retrieved 16 November 2026 from OpenRouter.

Failure examples

13

The lowest-scoring items from the public samples, with the worst of the repeats. Private items are never shown. Model output is shown exactly as text.

Lesson exercises

blx-manx-gv_flow_ch09_u02_grammar_v2-sc1

Manx · unit focus · score 0.00

Prompt

This is a sentence-building exercise from the Manx course of a language-learning app. Arrange the tiles into the Manx phrase that means “fetch the cake”. Use the tiles in the right order (some tiles are decoys and do not belong in the phrase).

Tiles: cur | y | lhaih | lesh | verreen

Give the finished phrase.
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

cur lesh y verreen

Model output

ANSWER: verreen y lhaih

Word forms

blas-morph:cornish:verb_imperative:kw-verb-bos:2p

Cornish · verb imperative · score 0.00

Prompt

Language: Cornish
Task: Give the imperative form of the verb for the person given. Write one word: the inflected verb form, with no pronoun, particle or auxiliary.
Verb: bos
Meaning: to be
Tense: imperative
Person: 2nd person plural (several people addressed)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

bedhewgh

Model output

ANSWER: bydewgh

blas-morph:manx:number_ordinal:gv-num-12

Manx · number ordinal · score 0.00

Prompt

Language: Manx
Task: Give the ordinal numeral for the number below: the ordinal ('first', 'second', ...), including the definite article yn as it stands before the ordinal.
Number: 12 (twelve)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

yn nah jeig

Model output

ANSWER: yn dooiney as jeih

blas-morph:manx:prepositional_pronoun:pp-gv-er:3sg_fem

Manx · prepositional pronoun · score 0.00

Prompt

Language: Manx
Task: Give the prepositional pronoun (the preposition fused with a personal pronoun) for the slot below. Write the standard simple form, not an emphatic form.
Preposition: er (on)
Slot: 3rd person singular feminine, meaning 'on her'
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

urree

Model output

ANSWER: urrey

Translation

blas-translation-cornish-kw-sent-tat-1356-from_english

Cornish · from english · score 0.06

Prompt

Translate this English sentence into Cornish.
Reply with the Cornish translation only, on one line, with no notes or explanation.

English: Do you like dogs?

Expected

Yw da genes keun?

Model output

A wra ta hwithra kiow?

blas-translation-manx-gv-sent-parallel-15295-from_english

Manx · from english · score 0.12

Prompt

Translate this English sentence into Manx.
Reply with the Manx translation only, on one line, with no notes or explanation.

English: I shall come back at ten o'clock at the latest.

Expected

Higym er-ash dyn s'anmee na jeih er y chlag.

Model output

Bee mee er tilgey ec yn dean cloc, ny smoo.

blas-translation-manx-gv-sent-parallel-15649-from_english

Manx · from english · score 0.13

Prompt

Translate this English sentence into Manx.
Reply with the Manx translation only, on one line, with no notes or explanation.

English: Have you made my bed?

Expected

Vel shiu er chiartaghey my lhiabbee?

Model output

Ta oo er n-jannoo my leaby?

Round trip

blas-round_trip-manx-gv-sent-parallel-2423

Manx · round trip · score 0.19

Prompt

Translate this Manx sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Manx: Dreill eh smaght er eh hene.

Expected

Dreill eh smaght er eh hene.

Model output

Ta'n lhean gheddyn hene.

Pivot text

A net catches itself.

blas-round_trip-cornish-kw-sent-tat-1830

Cornish · round trip · score 0.20

Prompt

Translate this Cornish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Cornish: Kas yw genev an traow na.

Expected

Kas yw genev an traow na.

Model output

Yw an tra na ganso?

Pivot text

Does he have that thing?

blas-round_trip-manx-gv-sent-parallel-4693

Manx · round trip · score 0.16

Prompt

Translate this Manx sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Manx: Va mee sheiltyn dy darragh eh.

Expected

Va mee sheiltyn dy darragh eh.

Model output

Va aggle orrym jeh shen.

Pivot text

I was afraid of it.

Tutor accuracy

blas-tutor-irish-clean-sent-9f6457b394

Irish · no error · score 0.00

Prompt

You are a Irish language tutor checking a sentence written by a learner.
Language: Irish
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Tá sé tábhachtach nach sárófaí a gcearta daonna.
Reply with one JSON object and nothing else:
{"…

Expected

Tá sé tábhachtach nach sárófaí a gcearta daonna.

Model output

{"has_error": true, "error_span": "sárófaí", "correction": "sárófar", "corrected_sentence": "Tá sé tábhachtach nach sárófar a gcearta daonna.", "explanation": "The conditional 'sárófaí' (would be violated) expresses a hypothetical, but the sentence states a real concern about human rights. The future analytic form 'sárófar' fits the meaning better. This is a judgement about tense rather than a clear grammatical break, so the minimal fix is to change only the verb form."}

blas-tutor-scots-gaelic-exd-5821a3d892

Scottish Gaelic · unit focus · score 0.00

Prompt

You are a Scottish Gaelic language tutor checking a sentence written by a learner.
Language: Scottish Gaelic
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Tha biadh anns a' cidsin.
Reply with one JSON object and nothing else:
{"has…

Expected

Tha biadh anns a' chidsin.

Model output

{"has_error": true, "error_span": "a' cidsin", "correction": "a' chidsin", "corrected_sentence": "Tha biadh anns a' chidsin.", "explanation": "After the article a' (here used with the preposition anns, 'in'), a feminine noun beginning with c is lenited, so c becomes ch: a' chidsin ('in the kitchen')."}

blas-tutor-scots-gaelic-exd-6e0581d3d2

Scottish Gaelic · grammar point · score 0.00

Prompt

You are a Scottish Gaelic language tutor checking a sentence written by a learner.
Language: Scottish Gaelic
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Cha tèid agam air deagh Fraingis a sgrìobhadh.
Reply with one JSON object an…

Expected

Cha tèid agam air deagh Fhraingis a sgrìobhadh.

Model output

{"has_error": true, "error_span": "deagh Fraingis", "correction": "deagh Fhraingis", "corrected_sentence": "Cha tèid agam air deagh Fhraingis a sgrìobhadh.", "explanation": "The adjective 'deagh' (good) lenites the noun that follows it, so 'Fraingis' should become 'Fhraingis'. The rest of the sentence is correct: 'Cha tèid agam air' is the idiom for 'I can't', and 'a sgrìobhadh' is the correct verbal noun construction."}