Teanga Bench

Qwen3.8 Flash

Qwen · qwen/qwen3.8-flash · run 16 November 2026 · each item asked 3 times

Overall score
61.6%95% range 59.8–63.4%
Rank
9 of 9
Cost per run
$10.45this suite, judges included
Per 1,000 correct
$3.70cost of 1,000 full-score answers
Consistency
50.6%same score in all 3 asks, mean over tests

Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built

Scores by test

#

Qwen3.8 Flash places best in Lesson exercises (8th of 9) and worst in Word forms (9th of 9, 43.5 points below the top).

Score on each test. Right: the score and its place among the ranked models. Select a test to compare every model.

  • This model, with its 95% range
  • Each other ranked model
  1. Lesson exercisesClaude Fable 5.2: 98.5%Gemini 4 Hydrogen: 97.3%GPT-6 Astra: 96.6%Gemini 4 Argon: 98.2%DeepSeek V4.1 Flash 1017: 95.6%Kimi K3.1 Pro: 95.8%GPT-6 Luna: 93.6%Claude Haiku 5.5: 87.5%Qwen3.8 Flash: 90.6%, 95% range 87.9% to 93.1%, 8th of 9
  2. Word formsClaude Fable 5.2: 94.1%Gemini 4 Hydrogen: 91.1%GPT-6 Astra: 92.5%Gemini 4 Argon: 85.4%DeepSeek V4.1 Flash 1017: 87.0%Kimi K3.1 Pro: 82.1%GPT-6 Luna: 80.5%Claude Haiku 5.5: 65.3%Qwen3.8 Flash: 50.6%, 95% range 46.2% to 55.1%, 9th of 9
  3. TranslationClaude Fable 5.2: 75.1%Gemini 4 Hydrogen: 72.4%GPT-6 Astra: 73.1%Gemini 4 Argon: 71.9%DeepSeek V4.1 Flash 1017: 68.8%Kimi K3.1 Pro: 69.4%GPT-6 Luna: 68.0%Claude Haiku 5.5: 57.9%Qwen3.8 Flash: 56.9%, 95% range 54.2% to 59.9%, 9th of 9
  4. Round tripClaude Fable 5.2: 74.3%Gemini 4 Hydrogen: 73.0%GPT-6 Astra: 69.3%Gemini 4 Argon: 70.0%DeepSeek V4.1 Flash 1017: 68.5%Kimi K3.1 Pro: 65.3%GPT-6 Luna: 64.5%Claude Haiku 5.5: 58.0%Qwen3.8 Flash: 52.2%, 95% range 48.8% to 55.4%, 9th of 9
  5. Tutor accuracyClaude Fable 5.2: 88.5%Gemini 4 Hydrogen: 88.1%GPT-6 Astra: 86.5%Gemini 4 Argon: 84.1%DeepSeek V4.1 Flash 1017: 76.1%Kimi K3.1 Pro: 79.3%GPT-6 Luna: 71.3%Claude Haiku 5.5: 66.9%Qwen3.8 Flash: 58.0%, 95% range 51.9% to 63.7%, 9th of 9
View as table
TestScore and 95% rangePlaceHighestMedian model
Lesson exercises90.6% 87.9–93.1%8th of 9Claude Fable 5.2 (98.5%)95.8%
Word forms50.6% 46.2–55.1%9th of 9Claude Fable 5.2 (94.1%)85.4%
Translation56.9% 54.2–59.9%9th of 9Claude Fable 5.2 (75.1%)69.4%
Round trip52.2% 48.8–55.4%9th of 9Claude Fable 5.2 (74.3%)68.5%
Tutor accuracy58.0% 51.9–63.7%9th of 9Claude Fable 5.2 (88.5%)79.3%

Qwen3.8 Flash places best in Lesson exercises (8th of 9) and worst in Word forms (9th of 9, 43.5 points below the top).

One axis per test, with the same scale on every profile: 0% at the centre to 100% at the outer ring. Solid accent: this model; faint dashed polygon: median model score on each test; faint dotted outer polygon: best score on each test (not necessarily one model). Vertex labels show this model’s scores. Compare scores, not polygon area; no confidence intervals are shown.

Qwen3.8 Flash test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 90.6%, median 95.8%, best 98.5%; Word forms: 50.6%, median 85.4%, best 94.1%; Translation: 56.9%, median 69.4%, best 75.1%; Round trip: 52.2%, median 68.5%, best 74.3%; Tutor accuracy: 58.0%, median 79.3%, best 88.5%90.6%Lessonexercises50.6%Word forms56.9%Translation52.2%Round trip58.0%Tutor accuracy
View as table
TestThis modelMedianBest
Lesson exercises90.6%95.8%98.5%
Word forms50.6%85.4%94.1%
Translation56.9%69.4%75.1%
Round trip52.2%68.5%74.3%
Tutor accuracy58.0%79.3%88.5%

Items, consistency and cost by test

Consistency is the share of items that got the same score in every repeat ask.

TestScore95% rangeItemsConsistencyCostPer 1,000 correct
Lesson exercises90.6%87.9–93.1%36092.2%$0.74$0.75
Word forms50.6%46.2–55.1%36069.4%$4.13$7.56
Translation56.9%54.2–59.9%36019.7%$1.63$2.66
Round trip52.2%48.8–55.4%2407.9%$1.94$5.16
Tutor accuracy58.0%51.9–63.7%18063.9%$2.01$6.43

By language

Score, 95% range and number of items (n) for each test.

LanguageLesson exercisesWord formsTranslationRound tripTutor accuracy
Irish88.9% 81.1–95.0% · n=6062.8% 52.2–73.3% · n=6064.9% 58.7–70.6% · n=6065.5% 59.9–71.0% · n=4063.3% 53.3–73.9% · n=60
Welsh90.6% 83.9–96.7% · n=6053.9% 43.3–64.4% · n=6070.7% 66.4–74.8% · n=6075.3% 70.6–79.8% · n=4058.9% 48.9–68.9% · n=60
Scottish Gaelic94.4% 88.9–98.3% · n=6065.6% 55.6–74.4% · n=6074.9% 69.3–80.6% · n=6069.2% 63.4–75.2% · n=4051.7% 41.1–61.7% · n=60
Manx87.8% 79.4–95.0% · n=6046.1% 35.0–57.2% · n=6024.5% 21.4–27.9% · n=6021.7% 18.4–25.2% · n=40n/a
Breton93.3% 87.2–98.3% · n=6044.4% 33.9–55.0% · n=6058.3% 51.8–64.8% · n=6050.2% 44.4–56.5% · n=40n/a
Cornish88.3% 80.0–95.6% · n=6030.6% 21.1–40.0% · n=6048.3% 41.6–54.9% · n=6031.2% 25.2–37.0% · n=40n/a

By CEFR level

The CEFR level comes from the course content; items without one are listed as unspecified.

Score, 95% range and number of items (n) for each test.

CEFR levelLesson exercisesWord formsTranslationRound tripTutor accuracy
A190.1% 85.1–94.7% · n=12563.2% 56.2–69.9% · n=13461.6% 56.0–67.3% · n=11250.4% 39.1–61.8% · n=2578.8% 66.7–90.9% · n=22
A289.4% 84.7–93.8% · n=13545.5% 38.8–52.9% · n=14052.5% 46.9–58.0% · n=9154.4% 49.3–59.6% · n=8860.5% 45.7–74.1% · n=27
B192.7% 87.2–97.3% · n=7313.7% 7.7–20.5% · n=3952.5% 44.5–61.0% · n=4748.0% 42.5–53.8% · n=7954.8% 42.1–66.7% · n=42
B291.7% 80.6–100.0% · n=2427.1% 8.3–45.8% · n=1648.9% 37.4–58.9% · n=3044.8% 34.2–55.9% · n=2733.3% 18.8–47.8% · n=23
C1100.0% 100.0–100.0% · n=3n/a69.7% 58.1–79.3% · n=1465.6% 56.9–74.0% · n=1543.3% 20.0–70.0% · n=10
C2n/an/a68.6% 60.7–75.0% · n=681.1% 73.3–87.7% · n=616.7% 0.0–33.3% · n=2
Unspecifiedn/a77.4% 66.7–87.1% · n=3158.2% 53.7–62.8% · n=60n/a65.4% 55.6–75.9% · n=54

Grammar topics

The topic is the grammar point or error type recorded on each item.

Lesson exercises

Strongest 5 topics.

TopicScoreItems
adjectives100.0%5
chapter overview100.0%11
food100.0%5
health100.0%5
nouns100.0%6

Weakest 5 topics.

TopicScoreItems
mutations63.0%9
conversation77.8%6
numbers77.8%6
unit focus85.0%51
vocabulary85.7%7

Only topics with at least 5 items are ranked.

Word forms

Strongest 5 topics.

TopicScoreItems
copula79.6%31
article gender mutation77.4%31
number personal70.4%9
number cardinal63.3%30
number counting60.0%5

Weakest 5 topics.

TopicScoreItems
verb pluperfect0.0%5
verb preterite6.7%5
verb imperfect14.3%14
verb conditional14.5%23
verb past34.8%22

Only topics with at least 5 items are ranked.

Translation

All topics, highest score first.

TopicScoreItems
to english66.7%150
legal or administrative58.2%60
from english46.7%150

Tutor accuracy

All topics, highest score first.

TopicScoreItems
orthography doubled letter85.7%7
guide71.7%20
grammar62.7%42
unit focus60.0%10
no error48.9%30
grammar point47.2%12
vocabulary44.4%15
application39.4%11

Only topics with at least 5 items are ranked.

Tutor explanations

judge-marked, secondary

Tutor items also ask for a short explanation. Two judge models from other model families mark it. This is a secondary number: it is judge-marked and not human-validated.

Tutor accuracy

Headline score
58.0%: found the error, fixed it and changed nothing else (rule-only, no judge).
Parts of the headline
Found 58.9% · fixed 58.1% · no over-correction 64.3%
Explanation score
n/a (95% range n/a), judge-marked, not human-validated. An explanation passes only when both judges pass it. Not comparable with other models: this run was not judged by the current judge pair.
Judge agreement
n/a of 0 judged answers
Unscored
0 answers a judge could not mark, after 0 failed judge calls; they are left out, not scored 0.
Judges
deepseek/deepseek-v4.1-flash-1017, qwen/qwen3.8-flash

Cost and prices

This suite
$10.45, of which $0 was judge spend
Whole run
$10.98 for both suites, against a cap of $12.00
Cost per correct answer
$0.0037 ($3.70 per 1,000): suite cost divided by the number of full-score answers
Run date
16 November 2026
Prices for qwen/qwen3.8-flash
$0.15 input, $0.47 output, $0.0160 cache read per million tokens. Retrieved 16 November 2026 from https://openrouter.ai/api/v1/models.

Failure examples

14

The lowest-scoring items from the public samples, with the worst of the repeats. Private items are never shown. Model output is shown exactly as text.

Lesson exercises

blx-manx-gv_flow_ch09_u02_grammar_v2-sc1

Manx · unit focus · score 0.00

Prompt

This is a sentence-building exercise from the Manx course of a language-learning app. Arrange the tiles into the Manx phrase that means “fetch the cake”. Use the tiles in the right order (some tiles are decoys and do not belong in the phrase).

Tiles: cur | y | lhaih | lesh | verreen

Give the finished phrase.
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

cur lesh y verreen

Model output

blx-welsh-cy_ch31_lost_property_recall_v3-s1d4

Welsh · recap · score 0.00

Prompt

This is a fill-in-the-blank exercise from the Welsh course of a language-learning app. Supply the missing Welsh word(s) for the blank (____); the English in the exercise tells you what is meant.

Exercise: Fill in the question word: "___ o'r gloch yw hi?" (What's the time?)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

Faint

Model output

ANSWER: Beth

Word forms

blas-morph:breton:verb_present:br-verb-kregiñ:2p

Breton · verb present · score 0.00

Prompt

Language: Breton
Task: Give the present form of the verb for the person given. Write one word: the inflected verb form, with no pronoun, particle or auxiliary.
Verb: kregiñ
Meaning: to start
Tense: present
Person: 2nd person plural (several people addressed)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

krogit

Model output

ANSWER: kregit

blas-morph:cornish:number_cardinal:kw-num-17

Cornish · number cardinal · score 0.00

Prompt

Language: Cornish
Task: Give the cardinal numeral for the number below: the standalone numeral, the form used to name the number (SWF Main).
Number: 17 (seventeen)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

seytek

Model output

blas-morph:cornish:verb_imperative:kw-verb-bos:2p

Cornish · verb imperative · score 0.00

Prompt

Language: Cornish
Task: Give the imperative form of the verb for the person given. Write one word: the inflected verb form, with no pronoun, particle or auxiliary.
Verb: bos
Meaning: to be
Tense: imperative
Person: 2nd person plural (several people addressed)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

bedhewgh

Model output

Translation

blas-translation-cornish-kw-sent-tat-1356-from_english

Cornish · from english · score 0.06

Prompt

Translate this English sentence into Cornish.
Reply with the Cornish translation only, on one line, with no notes or explanation.

English: Do you like dogs?

Expected

Yw da genes keun?

Model output

A gar'ta ky?

blas-translation-manx-gv-sent-parallel-15295-from_english

Manx · from english · score 0.10

Prompt

Translate this English sentence into Manx.
Reply with the Manx translation only, on one line, with no notes or explanation.

English: I shall come back at ten o'clock at the latest.

Expected

Higym er-ash dyn s'anmee na jeih er y chlag.

Model output

Glooym ec feed ayns oyr.

blas-translation-manx-gv-sent-parallel-15649-from_english

Manx · from english · score 0.16

Prompt

Translate this English sentence into Manx.
Reply with the Manx translation only, on one line, with no notes or explanation.

English: Have you made my bed?

Expected

Vel shiu er chiartaghey my lhiabbee?

Model output

Vel oo er my gyrvel y ymyd?

Round trip

blas-round_trip-manx-gv-sent-parallel-4693

Manx · round trip · score 0.00

Prompt

Translate this Manx sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Manx: Va mee sheiltyn dy darragh eh.

Expected

Va mee sheiltyn dy darragh eh.

Model output

Pivot text

I thought it happened.

blas-round_trip-cornish-kw-sent-tat-0552

Cornish · round trip · score 0.14

Prompt

Translate this Cornish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Cornish: Mar pe dy’Sul hedhyw, ni a wrussa mos dhe byskessa.

Expected

Mar pe dy’Sul hedhyw, ni a wrussa mos dhe byskessa.

Model output

Os o Dehow hedhyu, gwynsen ni myghi.

Pivot text

If today were Sunday, we would go fishing.

blas-round_trip-cornish-kw-sent-tat-1830

Cornish · round trip · score 0.20

Prompt

Translate this Cornish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Cornish: Kas yw genev an traow na.

Expected

Kas yw genev an traow na.

Model output

My a gas an re na.

Pivot text

I hate those things.

Tutor accuracy

blas-tutor-irish-clean-sent-9f6457b394

Irish · no error · score 0.00

Prompt

You are a Irish language tutor checking a sentence written by a learner.
Language: Irish
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Tá sé tábhachtach nach sárófaí a gcearta daonna.
Reply with one JSON object and nothing else:
{"…

Expected

Tá sé tábhachtach nach sárófaí a gcearta daonna.

Model output

response was not a JSON object

blas-tutor-scots-gaelic-clean-ex-943a0622b9

Scottish Gaelic · relative clauses · score 0.00

Prompt

You are a Scottish Gaelic language tutor checking a sentence written by a learner.
Language: Scottish Gaelic
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Nach robh iad aig a' mhuir tron oidhche?
Reply with one JSON object and noth…

Expected

Nach robh iad aig a' mhuir tron oidhche?

Model output

response was not a JSON object

blas-tutor-scots-gaelic-exd-6e0581d3d2

Scottish Gaelic · grammar point · score 0.00

Prompt

You are a Scottish Gaelic language tutor checking a sentence written by a learner.
Language: Scottish Gaelic
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Cha tèid agam air deagh Fraingis a sgrìobhadh.
Reply with one JSON object an…

Expected

Cha tèid agam air deagh Fhraingis a sgrìobhadh.

Model output

{
  "has_error": true,
  "error_span": "deagh Fraingis",
  "correction": "deagh Fhraingis",
  "corrected_sentence": "Cha tèid agam air deagh Fhraingis a sgrìobhadh.",
  "explanation": "Scottish Gaelic lenites Fraingis after the adjective deagh, so it should be deagh Fhraingis."
}