Teanga Bench

GPT-6 Astra

OpenAI · openai/gpt-6-astra · run 16 November 2026 · each item asked 3 times

Overall score
83.6%95% range 82.3–84.9%
Rank
3 of 9
Cost per run
$26.95this suite, judges included
Per 1,000 correct
$7.10cost of 1,000 full-score answers
Consistency
78.2%same score in all 3 asks, mean over tests

Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built

Scores by test

#

GPT-6 Astra places best in Word forms (2nd of 9, tied with the top) and worst in Round trip (4th of 9, 5.1 points below the top but tied with it); tied means the 95% ranges overlap.

Score on each test. Right: the score and its place among the ranked models. Select a test to compare every model.

  • This model, with its 95% range
  • Each other ranked model
  1. Lesson exercisesClaude Fable 5.2: 98.5%Gemini 4 Hydrogen: 97.3%Gemini 4 Argon: 98.2%DeepSeek V4.1 Flash 1017: 95.6%Kimi K3.1 Pro: 95.8%GPT-6 Luna: 93.6%Claude Haiku 5.5: 87.5%Qwen3.8 Flash: 90.6%GPT-6 Astra: 96.6%, 95% range 94.5% to 98.3%, 4th of 9
  2. Word formsClaude Fable 5.2: 94.1%Gemini 4 Hydrogen: 91.1%Gemini 4 Argon: 85.4%DeepSeek V4.1 Flash 1017: 87.0%Kimi K3.1 Pro: 82.1%GPT-6 Luna: 80.5%Claude Haiku 5.5: 65.3%Qwen3.8 Flash: 50.6%GPT-6 Astra: 92.5%, 95% range 89.9% to 94.8%, 2nd of 9
  3. TranslationClaude Fable 5.2: 75.1%Gemini 4 Hydrogen: 72.4%Gemini 4 Argon: 71.9%DeepSeek V4.1 Flash 1017: 68.8%Kimi K3.1 Pro: 69.4%GPT-6 Luna: 68.0%Claude Haiku 5.5: 57.9%Qwen3.8 Flash: 56.9%GPT-6 Astra: 73.1%, 95% range 70.5% to 75.7%, 2nd of 9
  4. Round tripClaude Fable 5.2: 74.3%Gemini 4 Hydrogen: 73.0%Gemini 4 Argon: 70.0%DeepSeek V4.1 Flash 1017: 68.5%Kimi K3.1 Pro: 65.3%GPT-6 Luna: 64.5%Claude Haiku 5.5: 58.0%Qwen3.8 Flash: 52.2%GPT-6 Astra: 69.3%, 95% range 66.6% to 72.1%, 4th of 9
  5. Tutor accuracyClaude Fable 5.2: 88.5%Gemini 4 Hydrogen: 88.1%Gemini 4 Argon: 84.1%DeepSeek V4.1 Flash 1017: 76.1%Kimi K3.1 Pro: 79.3%GPT-6 Luna: 71.3%Claude Haiku 5.5: 66.9%Qwen3.8 Flash: 58.0%GPT-6 Astra: 86.5%, 95% range 82.0% to 90.7%, 3rd of 9
View as table
TestScore and 95% rangePlaceHighestMedian model
Lesson exercises96.6% 94.5–98.3%4th of 9Claude Fable 5.2 (98.5%)95.8%
Word forms92.5% 89.9–94.8%2nd of 9Claude Fable 5.2 (94.1%)85.4%
Translation73.1% 70.5–75.7%2nd of 9Claude Fable 5.2 (75.1%)69.4%
Round trip69.3% 66.6–72.1%4th of 9Claude Fable 5.2 (74.3%)68.5%
Tutor accuracy86.5% 82.0–90.7%3rd of 9Claude Fable 5.2 (88.5%)79.3%

GPT-6 Astra places best in Word forms (2nd of 9, tied with the top) and worst in Round trip (4th of 9, 5.1 points below the top but tied with it); tied means the 95% ranges overlap.

One axis per test, with the same scale on every profile: 0% at the centre to 100% at the outer ring. Solid accent: this model; faint dashed polygon: median model score on each test; faint dotted outer polygon: best score on each test (not necessarily one model). Vertex labels show this model’s scores. Compare scores, not polygon area; no confidence intervals are shown.

GPT-6 Astra test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 96.6%, median 95.8%, best 98.5%; Word forms: 92.5%, median 85.4%, best 94.1%; Translation: 73.1%, median 69.4%, best 75.1%; Round trip: 69.3%, median 68.5%, best 74.3%; Tutor accuracy: 86.5%, median 79.3%, best 88.5%96.6%Lessonexercises92.5%Word forms73.1%Translation69.3%Round trip86.5%Tutor accuracy
View as table
TestThis modelMedianBest
Lesson exercises96.6%95.8%98.5%
Word forms92.5%85.4%94.1%
Translation73.1%69.4%75.1%
Round trip69.3%68.5%74.3%
Tutor accuracy86.5%79.3%88.5%

Items, consistency and cost by test

Consistency is the share of items that got the same score in every repeat ask.

TestScore95% rangeItemsConsistencyCostPer 1,000 correct
Lesson exercises96.6%94.5–98.3%36099.4%$2.48$2.38
Word forms92.5%89.9–94.8%36094.2%$5.24$5.25
Translation73.1%70.5–75.7%36061.1%$6.17$7.81
Round trip69.3%66.6–72.1%24047.9%$7.86$15.75
Tutor accuracy86.5%82.0–90.7%18088.3%$5.21$11.16

By language

Score, 95% range and number of items (n) for each test.

LanguageLesson exercisesWord formsTranslationRound tripTutor accuracy
Irish93.3% 86.7–98.3% · n=6091.7% 84.4–97.2% · n=6080.8% 75.6–85.5% · n=6077.7% 71.8–83.5% · n=4089.4% 82.2–95.6% · n=60
Welsh96.7% 91.7–100.0% · n=60100.0% 100.0–100.0% · n=6080.8% 76.3–85.1% · n=6079.6% 74.0–84.6% · n=4088.9% 82.2–95.0% · n=60
Scottish Gaelic98.3% 95.0–100.0% · n=6091.7% 83.3–98.3% · n=6077.0% 70.9–83.2% · n=6070.8% 65.4–76.1% · n=4081.1% 70.6–90.0% · n=60
Manx95.6% 90.0–100.0% · n=6087.2% 78.9–93.9% · n=6049.6% 44.0–55.8% · n=6056.8% 52.0–62.1% · n=40n/a
Breton95.6% 89.4–100.0% · n=6097.8% 93.3–100.0% · n=6071.6% 65.5–77.3% · n=6064.5% 58.3–71.4% · n=40n/a
Cornish100.0% 100.0–100.0% · n=6086.7% 78.9–93.3% · n=6078.9% 72.5–84.9% · n=6066.1% 58.3–74.1% · n=40n/a

By CEFR level

The CEFR level comes from the course content; items without one are listed as unspecified.

Score, 95% range and number of items (n) for each test.

CEFR levelLesson exercisesWord formsTranslationRound tripTutor accuracy
A197.1% 93.9–99.5% · n=12592.3% 87.8–96.3% · n=13476.6% 71.3–81.9% · n=11264.0% 55.8–72.5% · n=25100.0% 100.0–100.0% · n=22
A294.8% 91.1–98.5% · n=13590.7% 86.0–95.0% · n=14070.4% 65.4–75.3% · n=9173.7% 69.0–78.5% · n=8885.2% 71.6–96.3% · n=27
B197.7% 94.1–100.0% · n=7394.0% 87.2–98.3% · n=3965.9% 58.8–72.4% · n=4763.4% 58.5–67.9% · n=7985.7% 75.4–93.7% · n=42
B2100.0% 100.0–100.0% · n=2491.7% 79.2–100.0% · n=1665.1% 55.4–74.6% · n=3068.7% 62.0–76.0% · n=2776.8% 59.4–92.8% · n=23
C1100.0% 100.0–100.0% · n=3n/a86.3% 78.6–93.3% · n=1477.3% 69.2–85.8% · n=1570.0% 46.7–90.0% · n=10
C2n/an/a82.7% 75.4–90.6% · n=686.6% 79.3–94.5% · n=650.0% 33.3–66.7% · n=2
Unspecifiedn/a100.0% 100.0–100.0% · n=3176.3% 71.5–81.1% · n=60n/a90.7% 83.3–96.9% · n=54

Grammar topics

The topic is the grammar point or error type recorded on each item.

Lesson exercises

Strongest 5 topics.

TopicScoreItems
adjectives100.0%5
chapter overview100.0%11
daily life100.0%5
food100.0%5
grammar point100.0%10

Weakest 5 topics.

TopicScoreItems
syntax80.0%5
vocabulary85.7%7
narrative grammar86.7%5
conversation88.9%6
unit focus94.1%51

Only topics with at least 5 items are ranked.

Word forms

Strongest 5 topics.

TopicScoreItems
article gender mutation100.0%31
verb pluperfect100.0%5
verb preterite100.0%5
prepositional pronoun97.6%28
verb conditional97.1%23

Weakest 5 topics.

TopicScoreItems
number counting40.0%5
noun plural84.4%30
noun genitive85.2%9
verb future87.0%23
verb imperfect88.1%14

Only topics with at least 5 items are ranked.

Translation

All topics, highest score first.

TopicScoreItems
to english78.0%150
legal or administrative76.3%60
from english66.9%150

Tutor accuracy

All topics, highest score first.

TopicScoreItems
orthography doubled letter100.0%7
unit focus100.0%10
vocabulary91.1%15
guide90.0%20
grammar89.7%42
grammar point83.3%12
no error81.1%30
application66.7%11

Only topics with at least 5 items are ranked.

Tutor explanations

judge-marked, secondary

Tutor items also ask for a short explanation. Two judge models from other model families mark it. This is a secondary number: it is judge-marked and not human-validated.

Tutor accuracy

Headline score
86.5%: found the error, fixed it and changed nothing else (rule-only, no judge).
Parts of the headline
Found 86.7% · fixed 86.5% · no over-correction 89.4%
Explanation score
69.9% (95% range 63.5–76.2%), judge-marked, not human-validated. An explanation passes only when both judges pass it.
Judge agreement
91.0% of 523 judged answers
Unscored
17 answers a judge could not mark, after 17 failed judge calls; they are left out, not scored 0.
Judges
deepseek/deepseek-v4.1-flash-1017, qwen/qwen3.8-flash

Cost and prices

This suite
$26.95, of which $0.31 was judge spend
Whole run
$36.70 for both suites, against a cap of $110.00
Cost per correct answer
$0.0071 ($7.10 per 1,000): suite cost divided by the number of full-score answers
Run date
16 November 2026
Prices for openai/gpt-6-astra
$10.00 input, $50.00 output, $1.00 cache read, $12.50 cache write per million tokens. Retrieved 16 November 2026 from https://openrouter.ai/api/v1/models.
Judge deepseek/deepseek-v4.1-flash-1017
$0.15 input, $0.60 output, $0.0030 cache read per million tokens. Retrieved 16 November 2026 from OpenRouter.
Judge qwen/qwen3.8-flash
$0.15 input, $0.47 output, $0.0160 cache read per million tokens. Retrieved 16 November 2026 from OpenRouter.

Failure examples

12

The lowest-scoring items from the public samples, with the worst of the repeats. Private items are never shown. Model output is shown exactly as text.

Word forms

blas-morph:manx:number_ordinal:gv-num-12

Manx · number ordinal · score 0.00

Prompt

Language: Manx
Task: Give the ordinal numeral for the number below: the ordinal ('first', 'second', ...), including the definite article yn as it stands before the ordinal.
Number: 12 (twelve)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

yn nah jeig

Model output

ANSWER: yn nah yeig

blas-morph:scots-gaelic:copula:cop-gd-empha-04

Scottish Gaelic · copula · score 0.00

Prompt

Language: Scottish Gaelic
Task: Fill in the blank with the copula word (or copula particle) that the sentence needs. Write only the missing word(s).
Sentence: ___ do Dhia a bhuineas slàinte.
English: Salvation is God's affair.
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

'S ann

Model output

ANSWER: Is ann

blas-morph:scots-gaelic:number_counting:gd-num-2

Scottish Gaelic · number counting · score 0.00

Prompt

Language: Scottish Gaelic
Task: Give the counting numeral for the number below: the form used when counting aloud (not before a noun).
Number: 2 (two)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

dhà

Model output

ANSWER: a dhà

Translation

blas-translation-scots-gaelic-gd-sent-0223-from_english

Scottish Gaelic · from english · score 0.15

Prompt

Translate this English sentence into Scottish Gaelic.
Reply with the Scottish Gaelic translation only, on one line, with no notes or explanation.

English: He is a good carpenter.

Expected

Tha e na dheagh shaor.

Model output

’S e saor math a th’ ann.

blas-translation-scots-gaelic-gd-sent-0255-from_english

Scottish Gaelic · from english · score 0.25

Prompt

Translate this English sentence into Scottish Gaelic.
Reply with the Scottish Gaelic translation only, on one line, with no notes or explanation.

English: She used to drink beer.

Expected

B' àbhaist dhi leann òl.

Model output

Bhiodh i ag òl leann.

blas-translation-manx-gv-sent-parallel-15649-from_english

Manx · from english · score 0.41

Prompt

Translate this English sentence into Manx.
Reply with the Manx translation only, on one line, with no notes or explanation.

English: Have you made my bed?

Expected

Vel shiu er chiartaghey my lhiabbee?

Model output

Vel oo er n'yannoo my lhiabbee?

Round trip

blas-round_trip-cornish-kw-sent-tat-1830

Cornish · round trip · score 0.16

Prompt

Translate this Cornish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Cornish: Kas yw genev an traow na.

Expected

Kas yw genev an traow na.

Model output

Yma kas dhymm orth an taklow-na.

Pivot text

I hate those things.

blas-round_trip-manx-gv-sent-parallel-4693

Manx · round trip · score 0.20

Prompt

Translate this Manx sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Manx: Va mee sheiltyn dy darragh eh.

Expected

Va mee sheiltyn dy darragh eh.

Model output

Smooinee mee dy jig eh.

Pivot text

I thought he would come.

blas-round_trip-manx-gv-sent-parallel-6412

Manx · round trip · score 0.43

Prompt

Translate this Manx sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Manx: Nee eh costal queig keead stiagh as magh.

Expected

Nee eh costal queig keead stiagh as magh.

Model output

Nee eh costal mysh queig cheead.

Pivot text

It will cost about five hundred.

Tutor accuracy

blas-tutor-irish-exd-51c5e10877

Irish · communication · score 0.00

Prompt

You are a Irish language tutor checking a sentence written by a learner.
Language: Irish
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Fhan muid i mbrú.
Reply with one JSON object and nothing else:
{"has_error": true|false, "error_…

Expected

D'fhan muid i mbrú.

Model output

{"has_error":false,"error_span":"","correction":"","corrected_sentence":"Fhan muid i mbrú.","explanation":"The sentence is correct. “Fhan” is the past tense of “fan”, and “i” correctly eclipses “brú” to “mbrú”."}

blas-tutor-welsh-clean-sent-d6e9a5c833

Welsh · no error · score 0.00

Prompt

You are a Welsh language tutor checking a sentence written by a learner.
Language: Welsh
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Pleidlais, felly, ar welliant 12.
Reply with one JSON object and nothing else:
{"has_error": tru…

Expected

Pleidlais, felly, ar welliant 12.

Model output

{"has_error":true,"error_span":"Pleidlais","correction":"Pleidleisiais","corrected_sentence":"Pleidleisiais, felly, ar welliant 12.","explanation":"“Pleidlais” is the noun “vote”; “Pleidleisiais” is the verb form meaning “I voted”."}

blas-tutor-welsh-exd-d6fa3b01fb

Welsh · grammar · score 0.00

Prompt

You are a Welsh language tutor checking a sentence written by a learner.
Language: Welsh
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Byddai yna adar y tu allan er i fyddai merlod na defaid.
Reply with one JSON object and nothing …

Expected

Byddai yna adar y tu allan er na fyddai merlod na defaid.

Model output

{"has_error":true,"error_span":"er i","correction":"er na","corrected_sentence":"Byddai yna adar y tu allan er na fyddai merlod na defaid.","explanation":"Use 'er na' ('although … not') before the finite verb 'fyddai'. 'Er i' requires a different construction, with a verbal noun."}