Teanga Bench

DeepSeek V4.1 Flash 1017

DeepSeek · deepseek/deepseek-v4.1-flash-1017 · run 16 November 2026 · each item asked 3 times

Overall score
79.2%95% range 77.7–80.7%
Rank
5 of 9
Cost per run
$1.04this suite, judges included
Per 1,000 correct
$0.29cost of 1,000 full-score answers
Consistency
68.4%same score in all 3 asks, mean over tests

Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built

Scores by test

#

DeepSeek V4.1 Flash 1017 places best in Word forms (4th of 9) and worst in Tutor accuracy (6th of 9, 12.4 points below the top).

Score on each test. Right: the score and its place among the ranked models. Select a test to compare every model.

  • This model, with its 95% range
  • Each other ranked model
  1. Lesson exercisesClaude Fable 5.2: 98.5%Gemini 4 Hydrogen: 97.3%GPT-6 Astra: 96.6%Gemini 4 Argon: 98.2%Kimi K3.1 Pro: 95.8%GPT-6 Luna: 93.6%Claude Haiku 5.5: 87.5%Qwen3.8 Flash: 90.6%DeepSeek V4.1 Flash 1017: 95.6%, 95% range 93.6% to 97.2%, 6th of 9
  2. Word formsClaude Fable 5.2: 94.1%Gemini 4 Hydrogen: 91.1%GPT-6 Astra: 92.5%Gemini 4 Argon: 85.4%Kimi K3.1 Pro: 82.1%GPT-6 Luna: 80.5%Claude Haiku 5.5: 65.3%Qwen3.8 Flash: 50.6%DeepSeek V4.1 Flash 1017: 87.0%, 95% range 83.9% to 90.1%, 4th of 9
  3. TranslationClaude Fable 5.2: 75.1%Gemini 4 Hydrogen: 72.4%GPT-6 Astra: 73.1%Gemini 4 Argon: 71.9%Kimi K3.1 Pro: 69.4%GPT-6 Luna: 68.0%Claude Haiku 5.5: 57.9%Qwen3.8 Flash: 56.9%DeepSeek V4.1 Flash 1017: 68.8%, 95% range 66.2% to 71.4%, 6th of 9
  4. Round tripClaude Fable 5.2: 74.3%Gemini 4 Hydrogen: 73.0%GPT-6 Astra: 69.3%Gemini 4 Argon: 70.0%Kimi K3.1 Pro: 65.3%GPT-6 Luna: 64.5%Claude Haiku 5.5: 58.0%Qwen3.8 Flash: 52.2%DeepSeek V4.1 Flash 1017: 68.5%, 95% range 65.7% to 71.1%, 5th of 9
  5. Tutor accuracyClaude Fable 5.2: 88.5%Gemini 4 Hydrogen: 88.1%GPT-6 Astra: 86.5%Gemini 4 Argon: 84.1%Kimi K3.1 Pro: 79.3%GPT-6 Luna: 71.3%Claude Haiku 5.5: 66.9%Qwen3.8 Flash: 58.0%DeepSeek V4.1 Flash 1017: 76.1%, 95% range 70.6% to 81.3%, 6th of 9
View as table
TestScore and 95% rangePlaceHighestMedian model
Lesson exercises95.6% 93.6–97.2%6th of 9Claude Fable 5.2 (98.5%)95.8%
Word forms87.0% 83.9–90.1%4th of 9Claude Fable 5.2 (94.1%)85.4%
Translation68.8% 66.2–71.4%6th of 9Claude Fable 5.2 (75.1%)69.4%
Round trip68.5% 65.7–71.1%5th of 9Claude Fable 5.2 (74.3%)68.5%
Tutor accuracy76.1% 70.6–81.3%6th of 9Claude Fable 5.2 (88.5%)79.3%

DeepSeek V4.1 Flash 1017 places best in Word forms (4th of 9) and worst in Tutor accuracy (6th of 9, 12.4 points below the top).

One axis per test, with the same scale on every profile: 0% at the centre to 100% at the outer ring. Solid accent: this model; faint dashed polygon: median model score on each test; faint dotted outer polygon: best score on each test (not necessarily one model). Vertex labels show this model’s scores. Compare scores, not polygon area; no confidence intervals are shown.

DeepSeek V4.1 Flash 1017 test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 95.6%, median 95.8%, best 98.5%; Word forms: 87.0%, median 85.4%, best 94.1%; Translation: 68.8%, median 69.4%, best 75.1%; Round trip: 68.5%, median 68.5%, best 74.3%; Tutor accuracy: 76.1%, median 79.3%, best 88.5%95.6%Lessonexercises87.0%Word forms68.8%Translation68.5%Round trip76.1%Tutor accuracy
View as table
TestThis modelMedianBest
Lesson exercises95.6%95.8%98.5%
Word forms87.0%85.4%94.1%
Translation68.8%69.4%75.1%
Round trip68.5%68.5%74.3%
Tutor accuracy76.1%79.3%88.5%

Items, consistency and cost by test

Consistency is the share of items that got the same score in every repeat ask.

TestScore95% rangeItemsConsistencyCostPer 1,000 correct
Lesson exercises95.6%93.6–97.2%36095.3%$0.19$0.19
Word forms87.0%83.9–90.1%36090.8%$0.34$0.36
Translation68.8%66.2–71.4%36046.9%$0.06$0.08
Round trip68.5%65.7–71.1%24027.5%$0.34$0.69
Tutor accuracy76.1%70.6–81.3%18081.7%$0.11$0.26

By language

Score, 95% range and number of items (n) for each test.

LanguageLesson exercisesWord formsTranslationRound tripTutor accuracy
Irish92.2% 86.1–97.8% · n=6087.2% 79.4–93.9% · n=6076.9% 71.9–81.8% · n=6074.1% 68.4–79.9% · n=4076.1% 66.7–85.6% · n=60
Welsh96.1% 91.1–100.0% · n=6096.1% 91.1–100.0% · n=6076.3% 71.5–80.8% · n=6081.0% 75.8–86.1% · n=4083.9% 75.6–91.7% · n=60
Scottish Gaelic98.3% 95.6–100.0% · n=6091.1% 83.9–97.2% · n=6076.4% 70.6–81.9% · n=6074.8% 69.2–80.8% · n=4068.3% 57.8–77.8% · n=60
Manx93.9% 88.3–98.3% · n=6080.0% 70.6–88.3% · n=6046.2% 41.0–51.9% · n=6054.3% 49.1–59.7% · n=40n/a
Breton96.1% 91.7–99.4% · n=6085.6% 77.8–92.8% · n=6065.6% 59.3–71.6% · n=6061.4% 55.1–68.6% · n=40n/a
Cornish96.7% 91.7–100.0% · n=6082.2% 72.8–91.1% · n=6071.2% 64.6–77.5% · n=6065.3% 59.2–71.7% · n=40n/a

By CEFR level

The CEFR level comes from the course content; items without one are listed as unspecified.

Score, 95% range and number of items (n) for each test.

CEFR levelLesson exercisesWord formsTranslationRound tripTutor accuracy
A197.3% 94.7–99.5% · n=12589.3% 84.3–94.0% · n=13472.9% 67.5–77.9% · n=11264.4% 57.3–71.8% · n=2593.9% 86.4–100.0% · n=22
A293.6% 89.9–97.0% · n=13582.9% 76.9–88.3% · n=14065.0% 59.7–70.0% · n=9170.5% 65.9–75.2% · n=8877.8% 64.2–88.9% · n=27
B195.9% 91.8–99.1% · n=7392.3% 85.5–97.4% · n=3964.6% 57.6–71.4% · n=4765.1% 60.3–69.9% · n=7964.3% 50.8–76.2% · n=42
B295.8% 87.5–100.0% · n=2475.0% 54.2–93.8% · n=1665.1% 55.0–74.3% · n=3068.1% 61.7–75.1% · n=2769.6% 53.6–84.1% · n=23
C1100.0% 100.0–100.0% · n=3n/a81.6% 75.0–88.0% · n=1473.0% 64.8–81.3% · n=1576.7% 53.3–100.0% · n=10
C2n/an/a75.9% 66.5–86.1% · n=691.0% 82.7–98.0% · n=650.0% 0.0–100.0% · n=2
Unspecifiedn/a95.7% 88.2–100.0% · n=3168.3% 63.2–73.2% · n=60n/a80.9% 70.4–90.1% · n=54

Grammar topics

The topic is the grammar point or error type recorded on each item.

Lesson exercises

Strongest 5 topics.

TopicScoreItems
adjectives100.0%5
chapter overview100.0%11
food100.0%5
grammar point100.0%10
health100.0%5

Weakest 5 topics.

TopicScoreItems
syntax66.7%5
conversation83.3%6
vocabulary85.7%7
daily life86.7%5
unit focus91.5%51

Only topics with at least 5 items are ranked.

Word forms

Strongest 5 topics.

TopicScoreItems
number personal100.0%9
verb pluperfect100.0%5
verb preterite100.0%5
article gender mutation95.7%31
verb present94.2%23

Weakest 5 topics.

TopicScoreItems
noun genitive70.4%9
number counting73.3%5
noun plural76.7%30
verb imperative77.8%27
verb future79.7%23

Only topics with at least 5 items are ranked.

Translation

All topics, highest score first.

TopicScoreItems
to english74.0%150
legal or administrative68.3%60
from english63.7%150

Tutor accuracy

All topics, highest score first.

TopicScoreItems
orthography doubled letter100.0%7
unit focus86.7%10
grammar82.5%42
no error80.0%30
guide75.0%20
grammar point72.2%12
vocabulary64.4%15
application51.5%11

Only topics with at least 5 items are ranked.

Tutor explanations

judge-marked, secondary

Tutor items also ask for a short explanation. Two judge models from other model families mark it. This is a secondary number: it is judge-marked and not human-validated.

Tutor accuracy

Headline score
76.1%: found the error, fixed it and changed nothing else (rule-only, no judge).
Parts of the headline
Found 76.1% · fixed 76.3% · no over-correction 88.1%
Explanation score
n/a (95% range n/a), judge-marked, not human-validated. An explanation passes only when both judges pass it. Not comparable with other models: this run was not judged by the current judge pair.
Judge agreement
n/a of 0 judged answers
Unscored
0 answers a judge could not mark, after 0 failed judge calls; they are left out, not scored 0.
Judges
deepseek/deepseek-v4.1-flash-1017, qwen/qwen3.8-flash

Cost and prices

This suite
$1.04, of which $0 was judge spend
Whole run
$1.14 for both suites, against a cap of $1.50
Cost per correct answer
$0.000287 ($0.29 per 1,000): suite cost divided by the number of full-score answers
Run date
16 November 2026
Prices for deepseek/deepseek-v4.1-flash-1017
$0.0360 input, $0.50 output, $0.0100 cache read per million tokens. Retrieved 16 November 2026 from https://openrouter.ai/api/v1/models.

Failure examples

12

The lowest-scoring items from the public samples, with the worst of the repeats. Private items are never shown. Model output is shown exactly as text.

Word forms

blas-morph:cornish:verb_imperative:kw-verb-bos:2p

Cornish · verb imperative · score 0.00

Prompt

Language: Cornish
Task: Give the imperative form of the verb for the person given. Write one word: the inflected verb form, with no pronoun, particle or auxiliary.
Verb: bos
Meaning: to be
Tense: imperative
Person: 2nd person plural (several people addressed)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

bedhewgh

Model output

ANSWER: bydhowgh

blas-morph:manx:number_ordinal:gv-num-12

Manx · number ordinal · score 0.00

Prompt

Language: Manx
Task: Give the ordinal numeral for the number below: the ordinal ('first', 'second', ...), including the definite article yn as it stands before the ordinal.
Number: 12 (twelve)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

yn nah jeig

Model output

ANSWER: yn daa yeigoo

blas-morph:scots-gaelic:copula:cop-gd-empha-04

Scottish Gaelic · copula · score 0.00

Prompt

Language: Scottish Gaelic
Task: Fill in the blank with the copula word (or copula particle) that the sentence needs. Write only the missing word(s).
Sentence: ___ do Dhia a bhuineas slàinte.
English: Salvation is God's affair.
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

'S ann

Model output

ANSWER: Is

Translation

blas-translation-scots-gaelic-gd-sent-0255-from_english

Scottish Gaelic · from english · score 0.15

Prompt

Translate this English sentence into Scottish Gaelic.
Reply with the Scottish Gaelic translation only, on one line, with no notes or explanation.

English: She used to drink beer.

Expected

B' àbhaist dhi leann òl.

Model output

Bhiodh i ag òl lionn.

blas-translation-cornish-kw-sent-tat-1356-from_english

Cornish · from english · score 0.13

Prompt

Translate this English sentence into Cornish.
Reply with the Cornish translation only, on one line, with no notes or explanation.

English: Do you like dogs?

Expected

Yw da genes keun?

Model output

Osta fond a gŵn?

blas-translation-manx-gv-sent-parallel-15649-from_english

Manx · from english · score 0.41

Prompt

Translate this English sentence into Manx.
Reply with the Manx translation only, on one line, with no notes or explanation.

English: Have you made my bed?

Expected

Vel shiu er chiartaghey my lhiabbee?

Model output

Vel oo er n'yannoo my lhiabbee?

Round trip

blas-round_trip-cornish-kw-sent-tat-1830

Cornish · round trip · score 0.26

Prompt

Translate this Cornish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Cornish: Kas yw genev an traow na.

Expected

Kas yw genev an traow na.

Model output

My a gas an taklow na.

Pivot text

I hate those things.

blas-round_trip-cornish-kw-sent-tat-0552

Cornish · round trip · score 0.37

Prompt

Translate this Cornish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Cornish: Mar pe dy’Sul hedhyw, ni a wrussa mos dhe byskessa.

Expected

Mar pe dy’Sul hedhyw, ni a wrussa mos dhe byskessa.

Model output

Mara pe hedhyw dy' Sul, ni a dhi dhe bysketa.

Pivot text

If today were Sunday, we would go fishing.

blas-round_trip-irish-ir-sent-2746

Irish · round trip · score 0.42

Prompt

Translate this Irish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Irish: Ba sa bhliain 1976 a cuireadh pionós an bháis ar ceal.

Expected

Ba sa bhliain 1976 a cuireadh pionós an bháis ar ceal.

Model output

Cuireadh deireadh le pionós an bháis i 1976.

Pivot text

The death penalty was abolished in 1976.

Tutor accuracy

blas-tutor-scots-gaelic-exd-6e0581d3d2

Scottish Gaelic · grammar point · score 0.00

Prompt

You are a Scottish Gaelic language tutor checking a sentence written by a learner.
Language: Scottish Gaelic
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Cha tèid agam air deagh Fraingis a sgrìobhadh.
Reply with one JSON object an…

Expected

Cha tèid agam air deagh Fhraingis a sgrìobhadh.

Model output

{"has_error": true, "error_span": "deagh Fraingis", "correction": "deagh Fhraingis", "corrected_sentence": "Cha tèid agam air deagh Fhraingis a sgrìobhadh.", "explanation": "The adjective 'deagh' triggers lenition of the following noun, so 'Fraingis' becomes 'Fhraingis'."}

blas-tutor-scots-gaelic-clean-ex-943a0622b9

Scottish Gaelic · relative clauses · score 0.00

Prompt

You are a Scottish Gaelic language tutor checking a sentence written by a learner.
Language: Scottish Gaelic
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Nach robh iad aig a' mhuir tron oidhche?
Reply with one JSON object and noth…

Expected

Nach robh iad aig a' mhuir tron oidhche?

Model output

{"has_error": true, "error_span": "aig a' mhuir", "correction": "aig a' mhuir", "corrected_sentence": "Nach robh iad aig a' mhuir tron oidhche?", "explanation": "The sentence is actually correct as written; the erroneous substring field is left with the same wording because no change is needed."}

blas-tutor-scots-gaelic-exd-5821a3d892

Scottish Gaelic · unit focus · score 0.00

Prompt

You are a Scottish Gaelic language tutor checking a sentence written by a learner.
Language: Scottish Gaelic
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Tha biadh anns a' cidsin.
Reply with one JSON object and nothing else:
{"has…

Expected

Tha biadh anns a' chidsin.

Model output

{"has_error": false, "error_span": "", "correction": "", "corrected_sentence": "Tha biadh anns a' cidsin.", "explanation": "The sentence is correct."}