Benchmark

How we measure

Amada does not write text. It reads a text and a question, and returns probabilities: yes / no / "the text does not say" (noul), one option among a list (choice), or a level on a scale (score). So two things are measured:

• Accuracy: how often the most likely answer is the right one.
• Calibration (ECE): when Amada says 80 %, is it right 80 % of the time? Expected calibration error over 10 bins. 0 is perfect.

The test sets

Version by version

Same tests for every version, run on the same computer. 3.0.1 is 3.0 with new temperatures only, so its accuracy is identical; 3.1 is a short fine-tune of 3.0.

On these short tests, nothing has really moved since 2.0: differences of 0.1 to 0.5 points are within noise. Versions 3.0 and 3.1 were made for other things: long texts, long option lists, trap texts and near-identical labels. Those gains show in the public datasets (section 04), not here.

Calibration

This is where Amada changed the most. A model that says "95 %" and is wrong one time in four is dangerous to automate with. Since 3.0.1, temperatures depend on the length of the text and on the number of options.

Public datasets vs Jev

Seven public datasets with human labels, full test splits, zero-shot: Amada only gets the question and the names of the options. The Jev column is the score TypeSafe published for its closed model Jev 1.14; we did not run Jev ourselves.

From 2.0 to 3.1, Banking77 went from 0.337 to 0.611 and emotion from 0.369 to 0.471, thanks to long option lists (3.0) and near-neighbour labels (3.1). Jev is still ahead on every dataset. Amada runs on a laptop CPU; Jev is a hosted service.

Against other decision models

The system-one-benchmark (MIT) gives the same 300 examples per dataset and the same questions to every decision model. We ran Amada 3.1 on exactly the same rows and scored it with the same code. Numbers for the other models come from that repository.

† Kev's model card lists Banking77 and TREC among its training data: its scores on those two are not zero-shot. Laya, GLiNER2.5 and Von do not publish their training data; nothing in their numbers suggests they saw these tests. 300 examples per dataset: differences under ~5 points are within the margin of error.

Languages

Amada 3.1 on the short tests, all question types, per language. Training covered these 12 languages; Korean and Hindi (MASSIVE, section 05) were never in training and still reach 0.50 and 0.53.

Messy writing

Real messages are not clean: "jpeu pa venir demin jsui malade". We rewrote the same tests with keyboard typos (AZERTY, QWERTZ, QWERTY), swapped letters, lost accents, SMS spelling and homophones, without ever touching numbers or negations. Same questions, same answers.

On 20 hand-written trap messages, Amada 3.1 gets 17 right, 3 of them with little confidence. It misses, for example, "mé parents von au ciné ce soir mai moi jreste a la maison" → "Is the person going to the cinema?" (it says yes, 68 %), and the famous "Léa mange 4 gateau au fraise" → "léa mange t'elle des gateau ?" (yes, but only 70 %). This is the main target of Amada 4.

Where Amada is weak

Speed

Average time per text on a Windows laptop CPU, 4 threads, ONNX Runtime, on the short tests (several questions per text).

What comes next

Amada 4 is being prepared: messy writing (MASSIVE 1.1 and 88milSMS, both CC BY 4.0, plus half of the old data rewritten with typos), and the four weak spots of section 08. Starting point: the 3.1 weights.

Reproduce it

Questions or a number that looks wrong: liamdb.work@gmail.com.