Teanga Bench: Irish, Welsh, Scottish Gaelic, Manx, Breton and Cornish

Teanga Bench measures how well large language models handle six Celtic languages: Irish, Welsh, Scottish Gaelic, Manx, Breton and Cornish. Every model answers the same few hundred items per test, taken from the Blas course, dictionaries and parallel corpora: lesson exercises, word forms, translation, round-trip translation, and finding and fixing errors as a tutor would. Each item is asked several times. The score is the share of items answered correctly; translation tests give partial credit for closeness to reference translations.

Last updated 15 November 2026 · 1 model ranked · 5 tests

Leaderboard

Ranked by overall score, every test weighted equally. Cost is one run of this suite, judges included.

Rank	Model	Score	95% interval	Items	Cost per run	Cost per correct answer
1	openai/gpt-6-luna	75.6%	73.9% to 77.2%	1500	$0.78	$0.000224

How to read this

Score. The share of items the model got right, averaged over repeats. Translation tests give partial credit by chrF++ similarity to reference translations. The overall score weights every test equally.
Interval. The bar is the score; the thin line across it is the 95% interval, found by resampling the test items 2,000 times. When two models' intervals overlap, the gap between them may be chance.
Sources, not reviewers. Items and answers are taken from the sources named in the methodology. No one has reviewed them by hand, so a few gold answers may be wrong. Tutor explanations are marked by two judge models, not by people.
Private items. The full item sets are private so that models cannot be trained on them. 20 items per test are public, and the failure examples on the model pages come only from those.
Cost. What one run of this suite cost on OpenRouter at the prices listed on the day it ran, judge models included.

Read the methodology · Code and public samples on GitHub

Results by test

Lesson exercises

Multiple-choice, word-order and fill-the-gap questions from the Blas course, marked against the course's accepted answers.

Score with 95% interval; 360 items, each asked 3 times.

Word forms

Give one inflected form, such as a verb tense, a plural or a prepositional pronoun, marked against a list of accepted forms.

Score with 95% interval; 360 items, each asked 3 times.

Translation

Translate a sentence into or out of the language, scored by chrF++ similarity to one or more reference translations.

Score with 95% interval; 360 items, each asked 3 times.

Round trip

Translate a sentence into another language, then back again in a fresh conversation, scored by chrF++ similarity to the original.

Score with 95% interval; 240 items, each asked 3 times.

Tutor accuracy

Find and fix the error in a sentence, or leave a correct sentence alone; an item counts only when the error is found and fixed and nothing else is changed.

Score with 95% interval; 180 items, each asked 3 times. Rule-only headline; the judge-marked explanation score is on each model page.

By language

Score by language: the mean of the per-test scores, every test weighted equally. Darker cells are higher. Intervals are on the model pages.

Model	Irish	Welsh	Scottish Gaelic	Manx	Breton	Cornish
openai/gpt-6-luna	81%	83%	78%	67%	73%	70%

Score against cost

Overall score (with 95% interval) against the cost of one run of this suite. Up and to the left is better. Numbers are leaderboard ranks.

openai/gpt-6-luna: 75.6% for $0.78

The numbers on this page are also available as JSON. Full results files for every run are in the repository.