Tests
5 tests, each a few hundred items. Each test says what the model is asked to do, shows a real item and how it is marked, then compares every model.
Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built
- Lesson exercises360 itemsRule: accepted answers
- Word forms360 itemsRule: accepted answers
- Translation360 itemsRule: chrF++ overlap with a reference
- Round trip240 itemsRule: chrF++ overlap with the original
- Tutor accuracy180 itemsRule: found, fixed, nothing else changedJudges: the explanation, secondary
Rule marked by a fixed rule; no model is involved. Judges marked by two judge models; reported apart and not part of the score.
Lesson exercises
#The task. Answer one exercise from the Blas course: pick the right option, put words in order or fill a gap.
Marking. By rule: the answer must match one of the course's accepted answers; case, apostrophe style and outer spaces do not matter.
Example item from the public sample blx-cornish-kw_course_kw_flow_ch18_u01_v1-s1d0
Prompt
This is a multiple-choice exercise from the Cornish course of a language-learning app. Question: Which Cornish word means “buy”? Options: - pe - pris - gwertha - prena Choose the one correct option and give its text exactly as written above. Reply with the answer on one line, in this exact form: ANSWER: <answer>
Accepted answer
prena
Results
Claude Fable 5.2 scores highest (98.5%) but is tied with 5 other models: their 95% ranges overlap.
Score of each model with its 95% range (the thin line).
View as table
| Model | Score | 95% range | Items | Consistency |
|---|---|---|---|---|
| Claude Fable 5.2 | 98.5% | 97.1–99.5% | 360 | 99.2% |
| Gemini 4 Argon | 98.2% | 96.9–99.3% | 360 | 98.1% |
| Gemini 4 Hydrogen | 97.3% | 95.6–98.8% | 360 | 99.2% |
| GPT-6 Astra | 96.6% | 94.5–98.3% | 360 | 99.4% |
| Kimi K3.1 Pro | 95.8% | 93.9–97.6% | 360 | 96.9% |
| DeepSeek V4.1 Flash 1017 | 95.6% | 93.6–97.2% | 360 | 95.3% |
| GPT-6 Luna | 93.6% | 91.1–95.7% | 360 | 96.1% |
| Qwen3.8 Flash | 90.6% | 87.9–93.1% | 360 | 92.2% |
| Claude Haiku 5.5 | 87.5% | 84.4–90.5% | 360 | 91.4% |
On the Findings page: by CEFR level, hardest topics, by evidence
Word forms
#The task. Give one word form, such as a verb in a given tense, a plural, a number or a preposition fused with a pronoun.
Marking. By rule: the form must match one of the accepted forms from the course tables.
Example item from the public sample blas-morph:cornish:noun_plural:kw-decl-breselores
Prompt
Language: Cornish Task: Give the plural (nominative plural) of the noun below. Noun: breselores Meaning: militant Gender: feminine Reply with the answer on one line, in this exact form: ANSWER: <answer>
Accepted answer
breseloresow
Results
Claude Fable 5.2 scores highest (94.1%) but is tied with GPT-6 Astra and Gemini 4 Hydrogen: their 95% ranges overlap.
Score of each model with its 95% range (the thin line).
View as table
| Model | Score | 95% range | Items | Consistency |
|---|---|---|---|---|
| Claude Fable 5.2 | 94.1% | 91.7–96.2% | 360 | 96.9% |
| GPT-6 Astra | 92.5% | 89.9–94.8% | 360 | 94.2% |
| Gemini 4 Hydrogen | 91.1% | 88.3–93.6% | 360 | 92.2% |
| DeepSeek V4.1 Flash 1017 | 87.0% | 83.9–90.1% | 360 | 90.8% |
| Gemini 4 Argon | 85.4% | 82.1–88.5% | 360 | 89.4% |
| Kimi K3.1 Pro | 82.1% | 78.4–85.6% | 360 | 87.5% |
| GPT-6 Luna | 80.5% | 76.9–84.1% | 360 | 85.6% |
| Claude Haiku 5.5 | 65.3% | 60.9–69.4% | 360 | 75.8% |
| Qwen3.8 Flash | 50.6% | 46.2–55.1% | 360 | 69.4% |
On the Findings page: by CEFR level, hardest topics, by evidence
Translation
#The task. Translate one sentence into English or out of English, or a sentence of legal or administrative text (paired with French for Breton).
Marking. By rule: chrF++, the overlap of characters and words with a reference translation, from 0 to 100%, so close translations get partial credit.
Example item from the public sample blas-translation-cornish-kw-sent-tat-1356-from_english
Prompt
Translate this English sentence into Cornish. Reply with the Cornish translation only, on one line, with no notes or explanation. English: Do you like dogs?
Reference translation
Yw da genes keun?
Results
Claude Fable 5.2 scores highest (75.1%) but is tied with GPT-6 Astra, Gemini 4 Hydrogen and Gemini 4 Argon: their 95% ranges overlap.
Score of each model with its 95% range (the thin line).
View as table
| Model | Score | 95% range | Items | Consistency |
|---|---|---|---|---|
| Claude Fable 5.2 | 75.1% | 72.5–77.6% | 360 | 70.8% |
| GPT-6 Astra | 73.1% | 70.5–75.7% | 360 | 61.1% |
| Gemini 4 Hydrogen | 72.4% | 69.9–75.0% | 360 | 57.5% |
| Gemini 4 Argon | 71.9% | 69.4–74.4% | 360 | 51.7% |
| Kimi K3.1 Pro | 69.4% | 66.9–72.0% | 360 | 42.2% |
| DeepSeek V4.1 Flash 1017 | 68.8% | 66.2–71.4% | 360 | 46.9% |
| GPT-6 Luna | 68.0% | 65.4–70.8% | 360 | 43.9% |
| Claude Haiku 5.5 | 57.9% | 55.1–60.9% | 360 | 32.8% |
| Qwen3.8 Flash | 56.9% | 54.2–59.9% | 360 | 19.7% |
On the Findings page: by CEFR level, by direction, by evidence
Round trip
#The task. Translate a sentence into English, then, in a new conversation, translate that English back.
Marking. By rule: chrF++ overlap between the sentence that comes back and the original.
Example item from the public sample blas-round_trip-breton-br-sent-0526
Prompt
Translate this Breton sentence into English. Reply with the English translation only, on one line, with no notes or explanation. Breton: Oh ma Doue, ma zi eo!
Marked against the original sentence
Oh ma Doue, ma zi eo!
Results
Claude Fable 5.2 scores highest (74.3%) but is tied with Gemini 4 Hydrogen, Gemini 4 Argon and GPT-6 Astra: their 95% ranges overlap.
Score of each model with its 95% range (the thin line).
View as table
| Model | Score | 95% range | Items | Consistency |
|---|---|---|---|---|
| Claude Fable 5.2 | 74.3% | 71.7–77.0% | 240 | 55.8% |
| Gemini 4 Hydrogen | 73.0% | 70.4–75.6% | 240 | 45.0% |
| Gemini 4 Argon | 70.0% | 67.3–72.6% | 240 | 32.1% |
| GPT-6 Astra | 69.3% | 66.6–72.1% | 240 | 47.9% |
| DeepSeek V4.1 Flash 1017 | 68.5% | 65.7–71.1% | 240 | 27.5% |
| Kimi K3.1 Pro | 65.3% | 62.5–68.0% | 240 | 22.5% |
| GPT-6 Luna | 64.5% | 61.7–67.4% | 240 | 32.9% |
| Claude Haiku 5.5 | 58.0% | 54.8–61.2% | 240 | 18.3% |
| Qwen3.8 Flash | 52.2% | 48.8–55.4% | 240 | 7.9% |
On the Findings page: by CEFR level, by evidence
Tutor accuracy
#The task. Check a learner's sentence: say whether it has an error, give the wrong words and the correction, and explain.
Marking. By rule: an item counts only when the error is found and fixed and nothing else changes, or a correct sentence is left alone. Two judge models mark the explanation separately.
Example item from the public sample blas-tutor-welsh-exd-bc34d0d6aa
Prompt
You are a Welsh language tutor checking a sentence written by a learner.
Language: Welsh
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Mae'r dŵr yn glân.
Reply with one JSON object and nothing else:
{"has_error": true|false, "error_span": "...", "correction": "...", "corrected_sentence": "...", "explanation": "..."}Expected correction
glân → lân Corrected sentence: Mae'r dŵr yn lân.
Results
Claude Fable 5.2 scores highest (88.5%) but is tied with Gemini 4 Hydrogen, GPT-6 Astra, Gemini 4 Argon and Kimi K3.1 Pro: their 95% ranges overlap.
Score of each model with its 95% range (the thin line). Marked by rule only; the judge-marked explanation score is separate, below.
View as table
| Model | Score | 95% range | Items | Consistency |
|---|---|---|---|---|
| Claude Fable 5.2 | 88.5% | 84.1–92.4% | 180 | 92.8% |
| Gemini 4 Hydrogen | 88.1% | 83.5–92.2% | 180 | 94.4% |
| GPT-6 Astra | 86.5% | 82.0–90.7% | 180 | 88.3% |
| Gemini 4 Argon | 84.1% | 78.9–88.9% | 180 | 91.1% |
| Kimi K3.1 Pro | 79.3% | 74.3–84.1% | 180 | 77.2% |
| DeepSeek V4.1 Flash 1017 | 76.1% | 70.6–81.3% | 180 | 81.7% |
| GPT-6 Luna | 71.3% | 65.4–77.0% | 180 | 79.4% |
| Claude Haiku 5.5 | 66.9% | 60.9–72.6% | 180 | 70.6% |
| Qwen3.8 Flash | 58.0% | 51.9–63.7% | 180 | 63.9% |
Tutor explanations
judge-marked, secondary
Tutor items also ask for a short explanation. Two judge models from other model families mark it. This is a secondary number: judge-marked, not human-validated, and not part of the score above.
| Model | Explanation | 95% range | Judges agree |
|---|---|---|---|
| Claude Fable 5.2 | 78.5% | 72.9–84.1% | 93.5% |
| Gemini 4 Hydrogen | 76.4% | 70.4–82.0% | 95.1% |
| GPT-6 Astra | 69.9% | 63.5–76.2% | 91.0% |
| Gemini 4 Argon | 75.9% | 70.1–81.7% | 95.2% |
| Kimi K3.1 Pro | 70.4% | 64.7–76.0% | 93.4% |
| GPT-6 Luna | 57.4% | 50.6–64.2% | 93.2% |
| Claude Haiku 5.5 | 59.6% | 53.1–65.8% | 95.2% |
Not shown, because the run used a different judge pair or was not judged: DeepSeek V4.1 Flash 1017, Qwen3.8 Flash. Their own pages list what happened.
On the Findings page: by CEFR level, hardest topics, step by step, by evidence
What the tests measure
Teanga Bench measures how well large language models handle six Celtic languages: Irish, Welsh, Scottish Gaelic, Manx, Breton and Cornish. Every model answers the same few hundred items per test, taken from the Blas course, dictionaries and parallel corpora: lesson exercises, word forms, translation, round-trip translation, and finding and fixing errors as a tutor would. Each item is asked several times. The score is the share of items answered correctly; translation tests give partial credit for closeness to reference translations.
- Lesson exercises. Multiple-choice, word-order and fill-the-gap questions from the Blas course, marked against the course's accepted answers. 360 items.
- Word forms. Give one inflected form, such as a verb tense, a plural or a prepositional pronoun, marked against a list of accepted forms. 360 items.
- Translation. Translate a sentence into or out of the language, scored by chrF++ similarity to one or more reference translations. 360 items.
- Round trip. Translate a sentence into another language, then back again in a fresh conversation, scored by chrF++ similarity to the original. 240 items.
- Tutor accuracy. Find and fix the error in a sentence, or leave a correct sentence alone; an item counts only when the error is found and fixed and nothing else is changed. 180 items.