How Anbani Bench measures Georgian

Anbani Bench measures how well a language model reads, writes and corrects Georgian. Every score is computed by code: no model grades another model, and no score depends on a person's judgement. The test items are private, so no model can be tuned on them. This page explains what is measured and shows examples written for this page; none of them is in the test.

Not validated against human judgement. The scores show how models differ on these tasks. They do not show how Georgian speakers would rank the same models.

What is measured

Every task is asked in Georgian. A multiple-choice task labels its options ა, ბ, გ and დ, in an order shuffled per item, and the model answers with one letter.

Reading comprehension

A short passage and a question about it, four options. Source: Belebele, human translations of FLORES passages.

თბილისის მეტროს პირველი ხაზი 1966 წელს გაიხსნა. დღეს ქალაქში ორი ხაზი მუშაობს.

როდის გაიხსნა მეტროს პირველი ხაზი? ა) 1956 ბ) 1966 გ) 1976 დ) 1986

Exam knowledge

Questions from Georgian school and professional exams, four options. Source: INCLUDE, questions written in Georgian.

რომელი მდინარე ჩაედინება შავ ზღვაში? ა) რიონი ბ) მტკვარი გ) არაგვი დ) ალაზანი

Everyday reasoning

A goal and two ways to reach it; the model picks the one that works. Source: Global PIQA, written natively in Georgian.

როგორ გავაშრო სველი ფეხსაცმელი სწრაფად?
ა) ჩავდო შიგნით გაზეთის ქაღალდი და დავტოვო თბილ ადგილას. ბ) ჩავდო მაცივარში მთელი ღამით.

Grammar: case and agreement

Two sentences that differ in one word; the model picks the grammatical one. Case pairs test Georgian's split-ergative case marking (source: the Georgian case-alignment sets built on the Georgian Language Corpus treebank). Agreement pairs test subject–verb agreement (source: MultiBLiMP).

ა) გოგონამ წერილი დაწერა. ბ) გოგონა წერილი დაწერა.

ა) ბავშვები ეზოში თამაშობენ. ბ) ბავშვები ეზოში თამაშობს.

Error correction

A sentence with zero, one or two errors; the model returns it corrected and otherwise unchanged. Errors are injected by code into treebank sentences: ejective and aspirate confusions, wrong case endings, a dropped person marker, a wrong postposition, a plural noun after a numeral, a negation written together with its verb, a split or doubled word, an old letter, a capital Mtavruli letter. One sentence in five has no error, to measure false corrections.

დედამ ორი ვაშლები იყიდა. → დედამ ორი ვაშლი იყიდა.

მე ტბილისში ვცხოვრობ. → მე თბილისში ვცხოვრობ.

Translation

One sentence from Georgian to English and one from English to Georgian. Source: FLORES-200 devtest, professional human translations.

ზაფხულში ზღვაზე წავედით. → In the summer we went to the seaside.

Tokenizer as served

How many tokens a model spends on Georgian text, read from each tool's own usage counter, minus the cost of the instruction alone. Reported as tokens per character and per word, and as the ratio to the same sentences in English. It is a cost profile, not a skill, and it does not enter the ranking.

How it is scored

• Accuracy. An answer counts only when it parses to one of the offered letters. An unparsable answer, a refusal or a timeout scores zero and is counted as a failure; the failure rate is shown beside every score.
• Error correction. The corrected sentence is compared word by word with the original, and the same comparison is made for the gold correction. An edit counts when its span and its replacement both match. F0.5 weighs precision twice as much as recall, because a wrong correction is worse than a missed one. The exact-fix rate and the false correction rate on error-free sentences are reported too.
• Translation. chrF++ (character n-grams plus word bigrams) over the whole test set, which suits Georgian's rich morphology better than word-level BLEU.
• Uncertainty. Every score carries a 95% bootstrap interval (2,000 resamples of the items). A multiple-choice score is marked when it is not significantly above chance.

How models are ranked

There is no single headline number. The board is a profile: one column per task.

The order follows the rank score used by EuroEval. Within a task, the best model has rank 1. Going down the list, a model keeps the previous rank unless a Welch t-test over item scores says it is significantly worse (p < 0.05); then it drops by the score gap divided by the spread of scores across models. The rank score is a model's mean rank over the ranked tasks, so 1 is best. With one prompt wording per item, item-level variance stands in for EuroEval's repeated few-shot runs. The board also shows each task's Gini coefficient across models: how much the task separates them.

How models are run

• Each model is called through its own command-line tool on a personal subscription, the way a person would use it: Claude Code, Codex, Antigravity, Grok and opencode. Open-weight models will join through a local server.
• Every call is read-only and tool-free: no web search, no code execution, no files, one answer. Where the tool supports it, the answer is constrained to a JSON schema.
• Reasoning effort is the tool's lowest setting unless the model's name says otherwise.
• The instructions and the system prompt are in Georgian. There are no examples in the prompt (zero-shot).

Caveats

• Public sources may be in training data. Reading, exam, reasoning, grammar and translation items come from published datasets, so a model may have seen them. Error correction items are generated privately for each suite version.
• Scores move with the serving stack. A tool update can change a model's answers or token counts; each run records the tool's version.
• Training on test items. Items pass through the vendors' consumer tools. The board marks which vendors' accounts have training on user data turned off.
• A suite version is frozen. Scores are comparable only within one suite version; a new version gets its own board.

Sources and licences

Belebele (CC BY-SA 4.0), INCLUDE (Apache 2.0), Global PIQA (CC BY-SA 4.0), Georgian case alignment sets (CC BY 4.0), MultiBLiMP (CC BY 4.0), Universal Dependencies Georgian GLC and GNC treebanks (CC BY-SA 4.0), FLORES-200 (CC BY-SA 4.0).