More accents on the ASR board. Who still has to fix the transcript?

Two more accents become visible

An automatic transcript can look like an equal service because everyone gets the same button. That impression lasts until one person has to correct every third sentence while another can send the text straight on. The difference is part of the product’s workload. A useful evaluation should make that work visible.

On November 6, Hugging Face announced that the Open ASR Leaderboard had added private conversational datasets for Egyptian and Nigerian English through a collaboration with Liva AI. The Egyptian English split contains 1.34 hours and the Nigerian English split 1.55 hours. Both are described as spontaneous conversation, with punctuated, cased reference transcripts that retain disfluencies. The update also adds grouped comparisons in the leaderboard’s Private tab: US versus non-US English, and Inner Circle versus Outer and Expanding Circles, using Kachru’s framework. The announcement says private material is intended to reduce optimization against known evaluation audio.

More represented speech is a reason to examine the evidence, rather than a reason to declare accent fairness solved. These datasets cover particular recordings and speakers. They do not stand for every Egyptian or Nigerian English speaker, every microphone, or every purpose for which a transcript might be used.

Consider a fictional community group that records online workshops and publishes notes afterward. Its contributors use several varieties of English. The group wants to choose a transcription service without making the people least well served by the software responsible for fixing it. That is an achievable review question, and it leads to a different trial from simply asking which model tops an average score.

Source: Hugging Face: Egyptian and Nigerian English join the Open ASR Leaderboard ↗ · Open ASR Leaderboard documentation and scoring conventions ↗

Decide who carries the correction work

The group’s first meeting should concern the work after transcription. Who opens the draft? Who resolves a name the service misunderstood? Who checks whether a participant agreed to have a remark included in the public notes? If the answer is always “ask the speaker,” the organization has already assigned part of the cost to its contributors.

For a workshop, a reasonable arrangement might be that a paid editor prepares the draft and flags uncertain passages. Speakers can check the remarks attributed to them, but they should not have to reconstruct an entire recording before their contribution becomes usable. Let participants decline publication of their comments under the group’s recording policy. A transcription trial cannot replace that choice.

Define the output the group needs. It may want an accessible verbatim transcript for members and a separate edited summary for the public. Those are different documents. A summary can omit a repeated phrase while preserving the point; a verbatim transcript has a different responsibility to the speech. Asking reviewers to judge both without saying which they are looking at creates disagreement that has little to do with recognition quality.

Write a short review brief before collecting sample audio. For this group it could say: “We need a transcript that lets a reader follow each contribution, lets an editor locate uncertainty, and does not require one set of speakers to do most of the repair.” This makes participant effort an outcome of the review. It also gives the group permission to reject a fast service whose apparent efficiency depends on unpaid cleanup.

Build a sample with people’s permission

Use consented sample recordings collected for the trial, or material whose permission clearly covers this use. Explain which services will receive the audio and what will happen to the resulting files. If a candidate’s data handling is incompatible with that permission, exclude it before uploading anything. Calling the exercise an evaluation does not create permission to send recordings wherever a reviewer finds convenient.

Plan for variation within the group. A workshop contains short questions, longer explanations, interruptions and the occasional speaker far from a microphone. Select examples of the conditions that actually occur. Avoid giving one accent only clean, prepared speech and another only overlapping discussion. That design would bundle accent and recording conditions together, leaving the group unable to tell what drove a difference.

Do not assign accent labels by guessing from a voice. Where a breakdown is useful and participants agree to it, use descriptions they are comfortable supplying. Keep the categories modest. A person may have lived in several countries, use more than one language, and speak differently with different audiences. A single checkbox can be useful for organizing a small review without being a complete account of that speaker.

Keep a coverage note next to the results. It might say that the trial includes six volunteers in quiet rooms, several laptop microphones and no in-person workshop audio. That is an example of what to record, not a proposed minimum sample size. The note should also say which situations remain unexamined. A small, honest boundary is more helpful than a broad statement about accents that the recordings cannot support.

Separate recognition from editorial preference

The Open ASR Leaderboard documentation defines word error rate using substitutions, insertions and deletions relative to a reference transcript. Its scoring pipeline normalizes text before comparison to account for differences between model outputs and dataset transcriptions. That makes a standard numerical comparison possible; it also means a product review should ask which changes matter for the actual reader.

Create reference transcripts with reviewers who can understand the recordings. When a passage is unclear, mark it for discussion rather than inventing a polished sentence. Agree on conventions for pauses, repeated words and unfinished thoughts. If one reference retains “I, I think” and another reduces it to “I think,” a comparison can count a difference created by the transcription policy rather than a misunderstanding of the speaker.

For the community group, review three layers separately. First, check whether the words and attribution match the audio closely enough for the chosen transcript format. Second, ask whether a reader can follow the contribution without hearing it. Third, record the editor’s work to bring the draft to the agreed standard. A convenient total can hide a failure on one of those layers.

Suppose the software removes a participant’s repeated words and makes the sentence smoother. Whether that is acceptable depends on the document being produced. Suppose instead it substitutes a confident statement for an uncertain one. Even if the sentence reads well, it changes what the participant said. Give reviewers a place to record that distinction. The aim is to avoid rewarding fluency at the expense of a speaker’s meaning.

Reviewers should not “correct” a speaker into another variety of English merely because it sounds more familiar to them. If an expression is unfamiliar, listen again or consult an appropriate reviewer. The question is what was said and what the publication needs, rather than whether the speaker used the editor’s preferred phrasing.

Measure usable output and shared effort

Run candidate services on the same authorized recordings and preserve the drafts before editing. Record the settings and versions available to the group. Give editors the same brief and access to the same audio controls. If one service supplies timestamps or uncertainty cues, note whether those features actually help the editing workflow. A difference in the complete product can matter even when it is not a difference in raw recognition.

Use ordinary units that staff can understand: minutes of correction work per recording, passages still unresolved after review, and contributions that could be published under the agreed standard. Keep the original audio duration alongside editing time. Ten minutes of correction on a two-minute clip means something different from ten minutes on an hour-long workshop.

Look at the distribution before the overall average. In a hypothetical trial, two services could require the same total editing time while one concentrates nearly all of it on a few speakers. The group should inspect that concentration rather than celebrate the shared average. Ask whether the pattern follows microphone conditions, overlapping speech, a reviewer’s familiarity or several factors together. With a small trial, the result may point to the next question rather than identify a cause.

Have another reviewer examine a few passages, ideally without seeing which service produced them. When the reviewers disagree, use the recording and reference policy to resolve the issue. If both drafts are acceptable but one is easier to read, keep that as a usability judgment. If a passage remains genuinely unclear, report that limit instead of using an arbitrary reference choice to declare one model wrong.

Set a review point rather than treating the purchase as permanent. A change in the service, a different workshop format or a new group of contributors is a reason to repeat a small comparison. Preserve enough information to know which earlier result is being compared, but avoid retaining voice recordings indefinitely just because they once helped choose a product. Decide the retention period when participants give permission. If an audio sample is deleted, the evaluation record can still describe what was covered and when; it should not suggest that future reviewers can reproduce a check whose input is no longer available.

The group can then make a bounded decision. It might choose one service for quiet online sessions while arranging manual support for noisier events. Or it might decide that a better microphone and a dedicated editor matter more than switching models. Neither outcome diminishes the value of a public benchmark; the local review answers a more specific question.

Use private results without borrowing certainty

A private evaluation set can make it harder to tune a model directly to the particular recordings being scored. It also means an outside reader cannot necessarily inspect every audio clip and reference transcript. Those are different consequences of keeping test material held out. Neither makes the result useless or beyond scrutiny.

When reading the new leaderboard splits, check the dataset descriptions, the evaluation conventions and what the grouping combines. A non-US aggregate pools speech with different histories and recording conditions. A favorable average can be informative while leaving an important local gap unexamined. The community group should use the expanded coverage to ask better questions of vendors, then compare the answers with its own authorized sample.

Ask a vendor what evidence supports its claims about the participants the group actually serves. If it presents an accent score, request the task, sample description and transcript policy behind it. If it cannot provide them, keep the claim narrow. Do not convert a category label into a guarantee that a participant will be understood or that their editing burden will be equal.

After selecting a service, keep a simple route for reporting problems. A contributor should be able to flag a misattributed passage without having to diagnose the recognition system. Pay for the repair work the group needs. If a recurring problem emerges, create a new consented test example or a nonidentifying equivalent where feasible, and revisit the choice before expanding the recording program.

The November additions matter because they put more conversational English into view. A fairer workflow takes the next step: it notices whose words need the most repair, assigns that work deliberately, and lets participants influence how their speech becomes a public record. The scoreboard can start the conversation. The organization still has to make the service useful for the people pressing the record button.