The leaderboard has five columns. Which mistakes can you afford?

Read across the row

Suppose a small restaurant group wants an AI assistant to prepare a weekly purchasing summary. One model finds a delivery charge buried on page 18. Another keeps the totals in exactly the spreadsheet format the bookkeeper needs. A third notices that a supplier sent a quote instead of an invoice and leaves the amount blank. Which one is best depends on what happens when the assistant gets each of those jobs wrong.

The maintainers of SuperCLUE updated their Chinese general evaluation on November 5. It shows 19 models across agentic coding, mathematical and scientific reasoning, hallucination control, precise instruction following and long context. On the official board, GPT-6.1-Sol posts 69.23 in coding and 76.32 in reasoning; Gemini-4-Argon posts 83.19 in hallucination control and 95.63 in long context. They tie at 68.97 for precise instruction following. The board displays both for reference without numbered ranks. Among the entries the board labels open source, Darwin-180B-RSI-R3 records 73.68 in reasoning against Kimi-K3.1-Pro’s 63.16, while Kimi leads Darwin on long context, 87.25 to 30.32. Darwin is also shown for reference without a numbered rank. The board’s open-source designation does not establish license terms.

These are published benchmark scores. An 83.19 in hallucination control does not tell the restaurant how often its assistant will invent a charge. The comparisons point to a more practical use of the table: decide which capabilities deserve a closer look and build a small trial around the mistakes that matter.

Source: SuperCLUE: November 2026 Chinese general evaluation ↗ · NIST: Artificial Intelligence Risk Management Framework — Generative Artificial Intelligence Profile ↗

Write the work order before the scorecard

The restaurant’s weekly process has three steps. First, identify which documents count as invoices. Second, extract the vendor, invoice number, amount and currency. Third, turn the accepted records into a summary the bookkeeper can reconcile. The document set might include PDFs, email attachments and two slightly different copies of the same file. That mixture matters more to the trial than the broad description “understands documents.”

Give each step a condition a person can observe. For document identification, the assistant should leave the amount blank if a page is a quote or if the amount is unreadable. For extraction, the fields should match the invoice and link back to the relevant page. For the summary, the assistant should separate currencies and avoid counting repeated invoice numbers twice. Write these conditions before you know how any model responds.

Each condition suggests a different way a benchmark category could help. Long context is a reason to give a candidate a chance at a thick packet, but the local trial should ask whether it finds the right entry among near duplicates. Precise instruction following suggests a test of the required columns and blank fields. Hallucination control suggests including a packet for which the right answer is to say the evidence is missing. Coding and scientific reasoning scores might earn a place on another shortlist; this job has no reason to assign them equal importance.

NIST’s Generative AI Profile describes how a gap between laboratory testing and actual deployment can leave important risks unmeasured. For the restaurant, that gap is easy to picture: a benchmark evaluates a model under one set of conditions, while a bookkeeper will deal with the precise combination of files, tools, deadlines and staff review that the restaurant sets up. The useful trial reproduces enough of that combination to inform a decision.

Put expensive mistakes in their own column

Imagine an invented trial in which two candidates each complete nine of ten packets correctly. One fails because it puts the total in a field the spreadsheet can’t import. The other generates a believable amount for a document that never states it. Both might earn the same count of correct packets. The first failure creates an interruption that the bookkeeper is likely to notice; the second can move into a purchasing report unnoticed. Giving them the same penalty would hide the distinction the trial needs to reveal.

Build the decision in two layers. First, name the failures that block deployment for the specific workflow. A made-up charge, a wrong currency and a duplicate accepted as a new payable might belong here. Decide what response would be acceptable instead, such as marking the record unresolved and sending it to review. Then compare the candidates on ordinary operating costs: time to check the output, frequency of harmless formatting repairs, how often staff need to retry, and whether results arrive before the close of the week.

Resist the urge to add up both layers immediately. A candidate with fewer formatting repairs should not earn its way out of a failed critical case. Conversely, a cautious assistant that sends every document to a person will pass some safety checks while doing very little work. Record the rate of unnecessary escalations alongside the cases that deserved an escalation. That keeps “never guesses” from turning into a misleading label for “never helps.”

Decide what a small trial can show. One invented charge is enough to identify a real failure in the tested workflow; ten clean examples are not enough to establish a reliable real-world rate. A critical failure may lead you to change the workflow rather than change the model. For example, leave spreadsheet reconciliation in deterministic software and use the assistant only to point a reviewer to likely invoice fields. Then test that narrower product separately.

Make difficult cases by changing one fact

You can build a useful starter set without a large archive. Start with a clean, fictional invoice packet that resembles your work. Make a second version where a document is clearly labeled estimate. Make a third where the estimate label appears on the final page, while the opening page looks like an invoice. The expected outcome is clear in all three versions, but each change asks something different of the system.

For a long-document case, put an old price list near the beginning and the signed current price near the end. Then reverse their order in a paired example while leaving dates and document labels intact. The answer should follow the evidence about which price is current, regardless of its position. Pairing cases this way makes a failure easier to understand. A model might be able to locate a figure yet still select it using the wrong clue.

Add a case where the evidence stops short. A delivery receipt lists boxes and weights but no charges. Ask for the charge anyway and score an explicit “not stated” as correct. Give the answer key a reason for the abstention, so a reviewer can tell the difference between well-grounded restraint and an assistant that missed a figure printed on the next page. Include some ordinary clean cases so a model cannot do well by escalating everything.

Match the language and file quality that staff will encounter. If the restaurant receives records in multiple languages, include documents that were actually written in those languages or create realistic fictional versions with a fluent reviewer. Translating a single set can erase local number formats and paperwork conventions. Start with documents that contain no customer or staff identifiers; the trial can measure whether the assistant obeys an instruction to leave a blank without exposing real records.

Compare the system you would actually use

There are two legitimate comparisons and they answer different questions. You can ask which candidate handles a fixed prompt and identical extracted text best. That helps isolate differences in the model’s response under shared conditions. You can also compare two complete products, one with a better document parser and another with stronger spreadsheet checks. That helps a buyer decide which system is more useful. Keep the comparisons separate in your notes.

Use exact versions where you can identify them. Preserve the input files, prompt, tools and retry rule. If the workflow allows one repair attempt after a format error, record the first output as well as the repaired version. Count the time a person spends getting to a usable record. A product that routinely needs a second attempt may fit a weekly batch job and frustrate someone who needs an answer at a counter.

Choose a denominator that matches the budget. For this job, the unit might be one reconciled packet, not one model request. In a hypothetical comparison, a cheap request that returns three unusable rows may require several minutes of repair. A more expensive request could be the lower cost option once the bookkeeper’s time is counted. Keep the figures separate at first: request costs, staff minutes and the number of packets that could be reconciled. A narrow trial can show where effort went without implying that the numbers will hold across a year of purchasing.

Ask two reviewers to score a few outputs without seeing the model names. If they disagree, look at the rubric before declaring one reviewer wrong. “Links to the invoice” may mean a file name to one reviewer and a page number to another. Settle the definition and score the affected examples again. A clearer scorecard is useful even if it does not change which candidate goes forward.

If the first round reveals a fix, apply it and save some fresh cases for the next round. Editing the prompt until it solves the original examples can help diagnose a problem, but it gives little confidence that the repair will handle the next unusual invoice. A new packet with the same underlying complication is more informative. Keep a note that says which cases informed the change and which cases were used to check it afterward.

Decide what earns a place in the workflow

A decision record can be short. Write the job the system was tested for, the files it received, the outcomes that would prevent launch, and the parts where a human stays responsible. Include one or two concrete examples that explain the choice. “The review queue made unsupported amounts visible” tells the next person more than “Model A scored higher.”

The first decision can be narrower than a purchase. One candidate might be useful for identifying possible invoice pages while leaving the totals and currency checks to existing software. Another might work on clear digital files but fail on photographed documents with folded corners. Routing those photos directly to a person may be a reasonable trade if the rest of the process saves effort. Compare the resulting workload rather than assuming the model has to handle everything.

Set a trigger to reopen the decision. A new supplier, an unfamiliar language or a model update can change what reaches the assistant or how it behaves. Retain a handful of critical examples and repeat them before extending the workflow. If a staff member finds a new kind of failure, preserve an anonymized or fictional equivalent and add it to the next review. That gives the trial a way to keep learning from the work without turning every document into permanent evaluation data.

This is where a leaderboard earns its place. Its separate columns can reveal candidates that a glance at the total would miss. The restaurant still has to decide how much it cares about the location of a fact, the exact shape of a spreadsheet row or the cost of one plausible fabrication. The answer lives in the work the assistant is being asked to do and the evidence the buyer collects about that work.