Teanga Bench

Kimi K3.1 Pro

Moonshot AI · moonshotai/kimi-k3.1-pro · run 16 November 2026 · each item asked 3 times

Overall score
78.4%95% range 76.9–79.9%
Rank
6 of 9
Cost per run
$36.20this suite, judges included
Per 1,000 correct
$10.14cost of 1,000 full-score answers
Consistency
65.3%same score in all 3 asks, mean over tests

Items and answers come from the named sources and have not been reviewed by people, so a few gold answers may be wrong. How the items are built

Scores by test

#

Kimi K3.1 Pro places best in Lesson exercises (5th of 9, tied with the top) and worst in Word forms (6th of 9, 11.9 points below the top); tied means the 95% ranges overlap.

Score on each test. Right: the score and its place among the ranked models. Select a test to compare every model.

  • This model, with its 95% range
  • Each other ranked model
  1. Lesson exercisesClaude Fable 5.2: 98.5%Gemini 4 Hydrogen: 97.3%GPT-6 Astra: 96.6%Gemini 4 Argon: 98.2%DeepSeek V4.1 Flash 1017: 95.6%GPT-6 Luna: 93.6%Claude Haiku 5.5: 87.5%Qwen3.8 Flash: 90.6%Kimi K3.1 Pro: 95.8%, 95% range 93.9% to 97.6%, 5th of 9
  2. Word formsClaude Fable 5.2: 94.1%Gemini 4 Hydrogen: 91.1%GPT-6 Astra: 92.5%Gemini 4 Argon: 85.4%DeepSeek V4.1 Flash 1017: 87.0%GPT-6 Luna: 80.5%Claude Haiku 5.5: 65.3%Qwen3.8 Flash: 50.6%Kimi K3.1 Pro: 82.1%, 95% range 78.4% to 85.6%, 6th of 9
  3. TranslationClaude Fable 5.2: 75.1%Gemini 4 Hydrogen: 72.4%GPT-6 Astra: 73.1%Gemini 4 Argon: 71.9%DeepSeek V4.1 Flash 1017: 68.8%GPT-6 Luna: 68.0%Claude Haiku 5.5: 57.9%Qwen3.8 Flash: 56.9%Kimi K3.1 Pro: 69.4%, 95% range 66.9% to 72.0%, 5th of 9
  4. Round tripClaude Fable 5.2: 74.3%Gemini 4 Hydrogen: 73.0%GPT-6 Astra: 69.3%Gemini 4 Argon: 70.0%DeepSeek V4.1 Flash 1017: 68.5%GPT-6 Luna: 64.5%Claude Haiku 5.5: 58.0%Qwen3.8 Flash: 52.2%Kimi K3.1 Pro: 65.3%, 95% range 62.5% to 68.0%, 6th of 9
  5. Tutor accuracyClaude Fable 5.2: 88.5%Gemini 4 Hydrogen: 88.1%GPT-6 Astra: 86.5%Gemini 4 Argon: 84.1%DeepSeek V4.1 Flash 1017: 76.1%GPT-6 Luna: 71.3%Claude Haiku 5.5: 66.9%Qwen3.8 Flash: 58.0%Kimi K3.1 Pro: 79.3%, 95% range 74.3% to 84.1%, 5th of 9
View as table
TestScore and 95% rangePlaceHighestMedian model
Lesson exercises95.8% 93.9–97.6%5th of 9Claude Fable 5.2 (98.5%)95.8%
Word forms82.1% 78.4–85.6%6th of 9Claude Fable 5.2 (94.1%)85.4%
Translation69.4% 66.9–72.0%5th of 9Claude Fable 5.2 (75.1%)69.4%
Round trip65.3% 62.5–68.0%6th of 9Claude Fable 5.2 (74.3%)68.5%
Tutor accuracy79.3% 74.3–84.1%5th of 9Claude Fable 5.2 (88.5%)79.3%

Kimi K3.1 Pro places best in Lesson exercises (5th of 9, tied with the top) and worst in Word forms (6th of 9, 11.9 points below the top); tied means the 95% ranges overlap.

One axis per test, with the same scale on every profile: 0% at the centre to 100% at the outer ring. Solid accent: this model; faint dashed polygon: median model score on each test; faint dotted outer polygon: best score on each test (not necessarily one model). Vertex labels show this model’s scores. Compare scores, not polygon area; no confidence intervals are shown.

Kimi K3.1 Pro test profileScale: 0% at the centre to 100% at the outer ring. Lesson exercises: 95.8%, median 95.8%, best 98.5%; Word forms: 82.1%, median 85.4%, best 94.1%; Translation: 69.4%, median 69.4%, best 75.1%; Round trip: 65.3%, median 68.5%, best 74.3%; Tutor accuracy: 79.3%, median 79.3%, best 88.5%95.8%Lessonexercises82.1%Word forms69.4%Translation65.3%Round trip79.3%Tutor accuracy
View as table
TestThis modelMedianBest
Lesson exercises95.8%95.8%98.5%
Word forms82.1%85.4%94.1%
Translation69.4%69.4%75.1%
Round trip65.3%68.5%74.3%
Tutor accuracy79.3%79.3%88.5%

Items, consistency and cost by test

Consistency is the share of items that got the same score in every repeat ask.

TestScore95% rangeItemsConsistencyCostPer 1,000 correct
Lesson exercises95.8%93.9–97.6%36096.9%$2.31$2.23
Word forms82.1%78.4–85.6%36087.5%$8.93$10.07
Translation69.4%66.9–72.0%36042.2%$8.55$11.40
Round trip65.3%62.5–68.0%24022.5%$9.15$19.45
Tutor accuracy79.3%74.3–84.1%18077.2%$7.26$16.96

By language

Score, 95% range and number of items (n) for each test.

LanguageLesson exercisesWord formsTranslationRound tripTutor accuracy
Irish93.3% 86.7–98.3% · n=6090.6% 82.8–96.7% · n=6078.0% 73.4–82.6% · n=6078.6% 73.0–84.3% · n=4077.2% 67.8–86.1% · n=60
Welsh95.6% 90.0–99.4% · n=6092.2% 85.6–97.8% · n=6078.9% 74.3–83.3% · n=6076.9% 70.9–82.4% · n=4081.7% 73.9–88.9% · n=60
Scottish Gaelic97.2% 92.2–100.0% · n=6087.8% 80.0–94.4% · n=6077.5% 71.9–83.1% · n=6069.3% 63.3–75.2% · n=4078.9% 70.0–87.2% · n=60
Manx94.4% 87.8–99.4% · n=6080.6% 71.7–88.9% · n=6046.2% 41.1–51.6% · n=6053.2% 48.1–58.8% · n=40n/a
Breton96.7% 91.7–100.0% · n=6067.2% 56.1–77.8% · n=6064.3% 57.6–70.7% · n=6056.6% 50.7–63.0% · n=40n/a
Cornish97.8% 95.0–100.0% · n=6074.4% 64.4–83.9% · n=6071.6% 65.4–77.8% · n=6057.5% 51.7–63.6% · n=40n/a

By CEFR level

The CEFR level comes from the course content; items without one are listed as unspecified.

Score, 95% range and number of items (n) for each test.

CEFR levelLesson exercisesWord formsTranslationRound tripTutor accuracy
A197.3% 94.4–99.5% · n=12585.3% 79.6–90.3% · n=13472.8% 67.6–77.7% · n=11266.8% 58.6–75.4% · n=2597.0% 92.4–100.0% · n=22
A293.8% 89.9–97.3% · n=13580.7% 74.5–86.4% · n=14066.7% 61.6–71.7% · n=9166.6% 61.9–71.2% · n=8874.1% 60.5–86.4% · n=27
B196.3% 91.8–99.5% · n=7368.4% 55.6–80.3% · n=3965.3% 58.3–72.3% · n=4759.1% 54.2–64.0% · n=7973.8% 61.9–84.9% · n=42
B297.2% 91.7–100.0% · n=2470.8% 50.0–87.5% · n=1663.8% 54.3–72.8% · n=3067.5% 60.5–74.4% · n=2775.4% 60.9–88.4% · n=23
C1100.0% 100.0–100.0% · n=3n/a80.1% 73.3–86.7% · n=1477.6% 70.7–84.9% · n=1573.3% 56.7–86.7% · n=10
C2n/an/a77.9% 69.2–87.7% · n=682.6% 79.0–86.5% · n=650.0% 0.0–100.0% · n=2
Unspecifiedn/a97.8% 93.5–100.0% · n=3169.9% 64.8–74.9% · n=60n/a82.7% 73.5–91.4% · n=54

Grammar topics

The topic is the grammar point or error type recorded on each item.

Lesson exercises

Strongest 5 topics.

TopicScoreItems
adjectives100.0%5
chapter overview100.0%11
daily life100.0%5
food100.0%5
grammar point100.0%10

Weakest 5 topics.

TopicScoreItems
syntax80.0%5
vocabulary85.7%7
unit focus92.8%51
conversation94.4%6
grammar95.2%21

Only topics with at least 5 items are ranked.

Word forms

Strongest 5 topics.

TopicScoreItems
number personal100.0%9
verb pluperfect100.0%5
article gender mutation97.8%31
verb preterite93.3%5
copula91.4%31

Weakest 5 topics.

TopicScoreItems
number counting46.7%5
verb conditional62.3%23
noun genitive66.7%9
verb past71.2%22
noun plural75.6%30

Only topics with at least 5 items are ranked.

Translation

All topics, highest score first.

TopicScoreItems
to english75.8%150
legal or administrative69.9%60
from english62.8%150

Tutor accuracy

All topics, highest score first.

TopicScoreItems
orthography doubled letter100.0%7
no error84.4%30
guide83.3%20
unit focus83.3%10
grammar point83.3%12
grammar73.8%42
vocabulary71.1%15
application63.6%11

Only topics with at least 5 items are ranked.

Tutor explanations

judge-marked, secondary

Tutor items also ask for a short explanation. Two judge models from other model families mark it. This is a secondary number: it is judge-marked and not human-validated.

Tutor accuracy

Headline score
79.3%: found the error, fixed it and changed nothing else (rule-only, no judge).
Parts of the headline
Found 79.6% · fixed 79.3% · no over-correction 88.0%
Explanation score
70.4% (95% range 64.7–76.0%), judge-marked, not human-validated. An explanation passes only when both judges pass it.
Judge agreement
93.4% of 529 judged answers
Unscored
10 answers a judge could not mark, after 10 failed judge calls; they are left out, not scored 0.
Judges
deepseek/deepseek-v4.1-flash-1017, qwen/qwen3.8-flash

Cost and prices

This suite
$36.20, of which $0.28 was judge spend
Whole run
$43.58 for both suites, against a cap of $50.00
Cost per correct answer
$0.01 ($10.14 per 1,000): suite cost divided by the number of full-score answers
Run date
16 November 2026
Prices for moonshotai/kimi-k3.1-pro
$0.24 input, $8.40 output, $0.20 cache read per million tokens. Retrieved 16 November 2026 from https://openrouter.ai/api/v1/models.
Judge deepseek/deepseek-v4.1-flash-1017
$0.0360 input, $0.50 output, $0.0100 cache read per million tokens. Retrieved 16 November 2026 from OpenRouter.
Judge qwen/qwen3.8-flash
$0.15 input, $0.47 output, $0.0160 cache read per million tokens. Retrieved 16 November 2026 from OpenRouter.

Failure examples

12

The lowest-scoring items from the public samples, with the worst of the repeats. Private items are never shown. Model output is shown exactly as text.

Word forms

blas-morph:breton:verb_present:br-verb-kregiñ:2p

Breton · verb present · score 0.00

Prompt

Language: Breton
Task: Give the present form of the verb for the person given. Write one word: the inflected verb form, with no pronoun, particle or auxiliary.
Verb: kregiñ
Meaning: to start
Tense: present
Person: 2nd person plural (several people addressed)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

krogit

Model output

ANSWER: kregit

blas-morph:cornish:verb_imperative:kw-verb-bos:2p

Cornish · verb imperative · score 0.00

Prompt

Language: Cornish
Task: Give the imperative form of the verb for the person given. Write one word: the inflected verb form, with no pronoun, particle or auxiliary.
Verb: bos
Meaning: to be
Tense: imperative
Person: 2nd person plural (several people addressed)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

bedhewgh

Model output

ANSWER: byewgh

blas-morph:manx:number_ordinal:gv-num-12

Manx · number ordinal · score 0.00

Prompt

Language: Manx
Task: Give the ordinal numeral for the number below: the ordinal ('first', 'second', ...), including the definite article yn as it stands before the ordinal.
Number: 12 (twelve)
Reply with the answer on one line, in this exact form:
ANSWER: <answer>

Expected

yn nah jeig

Model output

ANSWER: yn daa-yeigoo

Translation

blas-translation-cornish-kw-sent-tat-1356-from_english

Cornish · from english · score 0.25

Prompt

Translate this English sentence into Cornish.
Reply with the Cornish translation only, on one line, with no notes or explanation.

English: Do you like dogs?

Expected

Yw da genes keun?

Model output

A wredh hwi kara keun?

blas-translation-scots-gaelic-gd-sent-0223-from_english

Scottish Gaelic · from english · score 0.14

Prompt

Translate this English sentence into Scottish Gaelic.
Reply with the Scottish Gaelic translation only, on one line, with no notes or explanation.

English: He is a good carpenter.

Expected

Tha e na dheagh shaor.

Model output

Is saor math e.

blas-translation-manx-gv-sent-parallel-15649-from_english

Manx · from english · score 0.41

Prompt

Translate this English sentence into Manx.
Reply with the Manx translation only, on one line, with no notes or explanation.

English: Have you made my bed?

Expected

Vel shiu er chiartaghey my lhiabbee?

Model output

Vel oo er n'yannoo my lhiabbee?

Round trip

blas-round_trip-cornish-kw-sent-tat-1830

Cornish · round trip · score 0.17

Prompt

Translate this Cornish sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Cornish: Kas yw genev an traow na.

Expected

Kas yw genev an traow na.

Model output

My a gas an taklow-na.

Pivot text

I hate those things.

blas-round_trip-manx-gv-sent-parallel-6412

Manx · round trip · score 0.20

Prompt

Translate this Manx sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Manx: Nee eh costal queig keead stiagh as magh.

Expected

Nee eh costal queig keead stiagh as magh.

Model output

Coshtys eh mysh queig cheead.

Pivot text

It will cost roughly five hundred.

blas-round_trip-breton-br-sent-0288

Breton · round trip · score 0.32

Prompt

Translate this Breton sentence into English.
Reply with the English translation only, on one line, with no notes or explanation.

Breton: Hennezh n'eo ket ma c'hazh. Ma hini a zo gwenn e dreid.

Expected

Hennezh n'eo ket ma c'hazh. Ma hini a zo gwenn e dreid.

Model output

N’eo ket ma c’hazh. An hini din en deus pavioù gwenn.

Pivot text

That is not my cat. Mine has white paws.

Tutor accuracy

blas-tutor-welsh-exd-facf492c47

Welsh · grammar · score 0.00

Prompt

You are a Welsh language tutor checking a sentence written by a learner.
Language: Welsh
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: y llyfr mae ar y bwrdd
Reply with one JSON object and nothing else:
{"has_error": true|false, "e…

Expected

y llyfr sydd ar y bwrdd

Model output

{"has_error":true,"error_span":"y llyfr mae","correction":"mae'r llyfr","corrected_sentence":"mae'r llyfr ar y bwrdd","explanation":"Welsh statements normally use verb-initial word order, and mae + y contracts to mae'r."}

blas-tutor-irish-clean-sent-9f6457b394

Irish · no error · score 0.00

Prompt

You are a Irish language tutor checking a sentence written by a learner.
Language: Irish
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Tá sé tábhachtach nach sárófaí a gcearta daonna.
Reply with one JSON object and nothing else:
{"…

Expected

Tá sé tábhachtach nach sárófaí a gcearta daonna.

Model output

{"has_error": true, "error_span": "nach sárófaí", "correction": "nach sáraítear", "corrected_sentence": "Tá sé tábhachtach nach sáraítear a gcearta daonna.", "explanation": "Baineann an abairt le riail ghinearálta sa chúlchaint láithreach, mar sin is é an modh coinníollach ‘sárófaí’ atá mícheart. Ba cheart an fhoirm neamhphearsanta láithreach ‘sáraítear’ a úsáid: ‘nach sáraítear a gcearta daonna’."}

blas-tutor-welsh-exd-58b2c1cfa7

Welsh · grammar · score 0.00

Prompt

You are a Welsh language tutor checking a sentence written by a learner.
Language: Welsh
The sentence may contain one error, or it may already be correct.
If it has an error, change only that one error: give the exact erroneous substring and the replacement for it. If it is correct, set has_error to false, leave error_span and correction empty and repeat the sentence as corrected_sentence.
Sentence: Mae'n honni iddi cael ei magu yng nghanolbarth gorllewinol yr Unol Daleithiau.
Reply with one JS…

Expected

Mae'n honni iddi gael ei magu yng nghanolbarth gorllewinol yr Unol Daleithiau.

Model output

{"has_error": true, "error_span": "iddi cael", "correction": "iddi gael", "corrected_sentence": "Mae'n honni iddi gael ei magu yng nghanolbarth gorllewinol yr Unol Daleithiau.", "explanation": "Yn y patrwm 'honni i + goddrych + berfenw', mae'r berfenw yn cael y treiglad meddal, felly dylai 'cael' fod yn 'gael' ar ôl 'iddi'."}