Teanga Bench

Tests

5 tests, each a few hundred items. Each test says what the model is asked to do, shows a real item and how it is marked, then compares every model.

Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built

Blas suite 5 tests · 1,500 items · each item asked 3 times
  1. Lesson exercises360 itemsRule: accepted answers
  2. Word forms360 itemsRule: accepted answers
  3. Translation360 itemsRule: chrF++ overlap with a reference
  4. Round trip240 itemsRule: chrF++ overlap with the original
  5. Tutor accuracy180 itemsRule: found, fixed, nothing else changedJudges: the explanation, secondary

Rule marked by a fixed rule; no model is involved. Judges marked by two judge models; reported apart and not part of the score.

Lesson exercises

#

The task. Answer one exercise from the Blas course: pick the right option, put words in order or fill a gap.

Marking. By rule: the answer must match one of the course's accepted answers; case, apostrophe style and outer spaces do not matter.

360 items · 6 languages · each asked 3 times · 9 models with a complete result

Example item from the public sample blx-cornish-kw_course_kw_flow_ch18_u01_v1-s1d0

Prompt

This is a multiple-choice exercise from the Cornish course of a language-learning app.

Question: Which Cornish word means “buy”?

Options:
- pe
- pris
- gwertha
- prena

Choose the one correct option and give its text exactly as written above.
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Accepted answer

prena

Results

Claude Fable 5.2 scores highest (98.5%) but is tied with 5 other models: their 95% ranges overlap.

Score of each model with its 95% range (the thin line).

  1. Claude Fable 5.2Anthropic
  2. Gemini 4 ArgonGoogle
  3. Gemini 4 HydrogenGoogle
  4. GPT-6 AstraOpenAI
  5. Kimi K3.1 ProMoonshot AI
  6. DeepSeek V4.1 Flash 1017DeepSeek
  7. GPT-6 LunaOpenAI
  8. Qwen3.8 FlashQwen
  9. Claude Haiku 5.5Anthropic
View as table
ModelScore95% rangeItemsConsistency
Claude Fable 5.298.5%97.1–99.5%36099.2%
Gemini 4 Argon98.2%96.9–99.3%36098.1%
Gemini 4 Hydrogen97.3%95.6–98.8%36099.2%
GPT-6 Astra96.6%94.5–98.3%36099.4%
Kimi K3.1 Pro95.8%93.9–97.6%36096.9%
DeepSeek V4.1 Flash 101795.6%93.6–97.2%36095.3%
GPT-6 Luna93.6%91.1–95.7%36096.1%
Qwen3.8 Flash90.6%87.9–93.1%36092.2%
Claude Haiku 5.587.5%84.4–90.5%36091.4%

Word forms

#

The task. Give one word form, such as a verb in a given tense, a plural, a number or a preposition fused with a pronoun.

Marking. By rule: the form must match one of the accepted forms from the course tables.

360 items · 6 languages · each asked 3 times · 9 models with a complete result

Example item from the public sample blas-morph:cornish:noun_plural:kw-decl-breselores

Prompt

Language: Cornish
Task: Give the plural (nominative plural) of the noun below.
Noun: breselores
Meaning: militant
Gender: feminine
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Accepted answer

breseloresow

Results

Claude Fable 5.2 scores highest (94.1%) but is tied with GPT-6 Astra and Gemini 4 Hydrogen: their 95% ranges overlap.

Score of each model with its 95% range (the thin line).

  1. Claude Fable 5.2Anthropic
  2. GPT-6 AstraOpenAI
  3. Gemini 4 HydrogenGoogle
  4. DeepSeek V4.1 Flash 1017DeepSeek
  5. Gemini 4 ArgonGoogle
  6. Kimi K3.1 ProMoonshot AI
  7. GPT-6 LunaOpenAI
  8. Claude Haiku 5.5Anthropic
  9. Qwen3.8 FlashQwen
View as table
ModelScore95% rangeItemsConsistency
Claude Fable 5.294.1%91.7–96.2%36096.9%
GPT-6 Astra92.5%89.9–94.8%36094.2%
Gemini 4 Hydrogen91.1%88.3–93.6%36092.2%
DeepSeek V4.1 Flash 101787.0%83.9–90.1%36090.8%
Gemini 4 Argon85.4%82.1–88.5%36089.4%
Kimi K3.1 Pro82.1%78.4–85.6%36087.5%
GPT-6 Luna80.5%76.9–84.1%36085.6%
Claude Haiku 5.565.3%60.9–69.4%36075.8%
Qwen3.8 Flash50.6%46.2–55.1%36069.4%

Translation

#

The task. Translate one sentence into English or out of English, or a sentence of legal or administrative text (paired with French for Breton).

Marking. By rule: chrF++, the overlap of characters and words with a reference translation, from 0 to 100%, so close translations get partial credit.

360 items · 6 languages · each asked 3 times · 9 models with a complete result

Example item from the public sample blas-translation-cornish-kw-sent-tat-1356-from_english

Prompt

Translate this English sentence into Cornish.
Reply with the Cornish translation only, on one line, with no notes or explanation.

English: Do you like dogs?

Reference translation

Yw da genes keun?

Results

Claude Fable 5.2 scores highest (75.1%) but is tied with GPT-6 Astra, Gemini 4 Hydrogen and Gemini 4 Argon: their 95% ranges overlap.

Score of each model with its 95% range (the thin line).

  1. Claude Fable 5.2Anthropic
  2. GPT-6 AstraOpenAI
  3. Gemini 4 HydrogenGoogle
  4. Gemini 4 ArgonGoogle
  5. Kimi K3.1 ProMoonshot AI
  6. DeepSeek V4.1 Flash 1017DeepSeek
  7. GPT-6 LunaOpenAI
  8. Claude Haiku 5.5Anthropic
  9. Qwen3.8 FlashQwen
View as table
ModelScore95% rangeItemsConsistency
Claude Fable 5.275.1%72.5–77.6%36070.8%
GPT-6 Astra73.1%70.5–75.7%36061.1%
Gemini 4 Hydrogen72.4%69.9–75.0%36057.5%
Gemini 4 Argon71.9%69.4–74.4%36051.7%
Kimi K3.1 Pro69.4%66.9–72.0%36042.2%
DeepSeek V4.1 Flash 101768.8%66.2–71.4%36046.9%
GPT-6 Luna68.0%65.4–70.8%36043.9%
Claude Haiku 5.557.9%55.1–60.9%36032.8%
Qwen3.8 Flash56.9%54.2–59.9%36019.7%

Round trip

#

The task. Translate a sentence into English, then, in a new conversation, translate that English back.

Marking. By rule: chrF++ overlap between the sentence that comes back and the original.

240 items · 6 languages · each asked 3 times · 9 models with a complete result

Example item from the public sample blas-round_trip-breton-br-sent-0526

Prompt

Translate this Breton sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Breton: Oh ma Doue, ma zi eo!

Marked against the original sentence

Oh ma Doue, ma zi eo!

Results

Claude Fable 5.2 scores highest (74.3%) but is tied with Gemini 4 Hydrogen, Gemini 4 Argon and GPT-6 Astra: their 95% ranges overlap.

Score of each model with its 95% range (the thin line).

  1. Claude Fable 5.2Anthropic
  2. Gemini 4 HydrogenGoogle
  3. Gemini 4 ArgonGoogle
  4. GPT-6 AstraOpenAI
  5. DeepSeek V4.1 Flash 1017DeepSeek
  6. Kimi K3.1 ProMoonshot AI
  7. GPT-6 LunaOpenAI
  8. Claude Haiku 5.5Anthropic
  9. Qwen3.8 FlashQwen
View as table
ModelScore95% rangeItemsConsistency
Claude Fable 5.274.3%71.7–77.0%24055.8%
Gemini 4 Hydrogen73.0%70.4–75.6%24045.0%
Gemini 4 Argon70.0%67.3–72.6%24032.1%
GPT-6 Astra69.3%66.6–72.1%24047.9%
DeepSeek V4.1 Flash 101768.5%65.7–71.1%24027.5%
Kimi K3.1 Pro65.3%62.5–68.0%24022.5%
GPT-6 Luna64.5%61.7–67.4%24032.9%
Claude Haiku 5.558.0%54.8–61.2%24018.3%
Qwen3.8 Flash52.2%48.8–55.4%2407.9%

Tutor accuracy

#

The task. Check a learner's sentence: say whether it has an error, give the wrong words and the correction, and explain.

Marking. By rule: an item counts only when the error is found and fixed and nothing else changes, or a correct sentence is left alone. Two judge models mark the explanation separately.

180 items · 3 languages · each asked 3 times · 9 models with a complete result

Example item from the public sample blas-tutor-welsh-exd-bc34d0d6aa

Prompt

You are a Welsh language tutor checking a sentence written by a learner.
Language: Welsh
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Mae'r dŵr yn glân.
Reply with one JSON object and nothing else:
{"has_error": true|false, "error_span": "...", "correction": "...", "corrected_sentence": "...", "explanation": "..."}

Expected correction

glân → lân
Corrected sentence: Mae'r dŵr yn lân.

Results

Claude Fable 5.2 scores highest (88.5%) but is tied with Gemini 4 Hydrogen, GPT-6 Astra, Gemini 4 Argon and Kimi K3.1 Pro: their 95% ranges overlap.

Score of each model with its 95% range (the thin line). Marked by rule only; the judge-marked explanation score is separate, below.

  1. Claude Fable 5.2Anthropic
  2. Gemini 4 HydrogenGoogle
  3. GPT-6 AstraOpenAI
  4. Gemini 4 ArgonGoogle
  5. Kimi K3.1 ProMoonshot AI
  6. DeepSeek V4.1 Flash 1017DeepSeek
  7. GPT-6 LunaOpenAI
  8. Claude Haiku 5.5Anthropic
  9. Qwen3.8 FlashQwen
View as table
ModelScore95% rangeItemsConsistency
Claude Fable 5.288.5%84.1–92.4%18092.8%
Gemini 4 Hydrogen88.1%83.5–92.2%18094.4%
GPT-6 Astra86.5%82.0–90.7%18088.3%
Gemini 4 Argon84.1%78.9–88.9%18091.1%
Kimi K3.1 Pro79.3%74.3–84.1%18077.2%
DeepSeek V4.1 Flash 101776.1%70.6–81.3%18081.7%
GPT-6 Luna71.3%65.4–77.0%18079.4%
Claude Haiku 5.566.9%60.9–72.6%18070.6%
Qwen3.8 Flash58.0%51.9–63.7%18063.9%

Tutor explanations

judge-marked, secondary

Tutor items also ask for a short explanation. Two judge models from other model families mark it. This is a secondary number: judge-marked, not human-validated, and not part of the score above.

ModelExplanation95% rangeJudges agree
Claude Fable 5.278.5%72.9–84.1%93.5%
Gemini 4 Hydrogen76.4%70.4–82.0%95.1%
GPT-6 Astra69.9%63.5–76.2%91.0%
Gemini 4 Argon75.9%70.1–81.7%95.2%
Kimi K3.1 Pro70.4%64.7–76.0%93.4%
GPT-6 Luna57.4%50.6–64.2%93.2%
Claude Haiku 5.559.6%53.1–65.8%95.2%

Not shown, because the run used a different judge pair or was not judged: DeepSeek V4.1 Flash 1017, Qwen3.8 Flash. Their own pages list what happened.

What the tests measure

Teanga Bench measures how well large language models handle six Celtic languages: Irish, Welsh, Scottish Gaelic, Manx, Breton and Cornish. Every model answers the same few hundred items per test, taken from the Blas course, dictionaries and parallel corpora: lesson exercises, word forms, translation, round-trip translation, and finding and fixing errors as a tutor would. Each item is asked several times. The score is the share of items answered correctly; translation tests give partial credit for closeness to reference translations.

  • Lesson exercises. Multiple-choice, word-order and fill-the-gap questions from the Blas course, marked against the course's accepted answers. 360 items.
  • Word forms. Give one inflected form, such as a verb tense, a plural or a prepositional pronoun, marked against a list of accepted forms. 360 items.
  • Translation. Translate a sentence into or out of the language, scored by chrF++ similarity to one or more reference translations. 360 items.
  • Round trip. Translate a sentence into another language, then back again in a fresh conversation, scored by chrF++ similarity to the original. 240 items.
  • Tutor accuracy. Find and fix the error in a sentence, or leave a correct sentence alone; an item counts only when the error is found and fixed and nothing else is changed. 180 items.