A speech AI prompt should help it listen, not tell it what to hear

The new control is also a new variable

NVIDIA announced Nemotron 3.5 Transcribe and Granary v2 on October 28. Its open-weight speech model accepts natural-language instructions and supports transcription, speaker attribution, translation and questions about audio. NVIDIA documents different language coverage for different tasks, so a language appearing on a transcription list does not establish that every other operation works in it. Granary v2 supplies data and adaptation resources alongside the model.

That is the release news. The more interesting methods question is what happens when a transcription prompt contains information the recording does not. NVIDIA’s model card explicitly warns that supplied context can push output toward incorrect terms, and that speaker assignments and generated answers can be wrong. This article develops a proposed evaluation exercise from those limitations. We have not run the model or measured its performance.

Imagine a student club recording a discussion about an installation called Signal Garden. Several members say its name clearly. Someone else mentions an unrelated garden project, and another person briefly says signal processing. A glossary containing Signal Garden might help the transcription. It might also make the system overuse that phrase. You need a test that can distinguish those outcomes.

A familiar-looking proper noun is not automatically a more faithful transcript. The useful question is whether context helps the system recover audible words without making it invent additional appearances. Treat your prompt as a variable that needs evidence, rather than as a magic correction field.

Source: NVIDIA: Nemotron 3.5 Transcribe and Granary v2 ↗ · NVIDIA: Nemotron 3.5 Transcribe model card and limitations ↗ · W3C WAI: Transcribing Audio to Text ↗

Start with a listening contract

Before choosing settings, decide what the transcript is supposed to preserve. A readable event recap, a close record of an interview and searchable captions for a tutorial are different products. If you change that purpose halfway through an evaluation, you can mistake an editing choice for a recognition improvement.

For a small pilot, write a short contract in ordinary language. Specify whether hesitations remain, whether numbers use words or digits, how unidentified speakers are labeled and how uncertain passages appear. Keep the contract beside the recording. It should describe the output you need, without quietly supplying answers to passages that are difficult to hear.

W3C’s audio-transcription guidance distinguishes the information needed to make audio understandable, including relevant speaker identification and non-speech sound. Use that guidance to think about the reader’s task. A laugh, alarm or long interruption may matter to the meaning even when a polished summary would leave it out.

Then create a reference through careful human listening. For a learning exercise, choose a short recording you own and have permission to process. Mark ambiguous passages rather than pretending every syllable has an obvious answer. If two listeners disagree, preserve that disagreement as part of the reference notes.

This reference is a working aid, not a claim that your tiny sample represents every accent, microphone or room. Its job is to stop the AI’s confident output from becoming the answer key. Keep the original audio available throughout the comparison, because a transcript cannot verify itself.

Make the glossary earn its place

Use the same audio in three conditions. In the first, ask for transcription with your formatting contract and no glossary. In the second, add a short list of genuine names or specialist terms that occur in the clip. In the third, add those terms plus one plausible distractor that does not occur. Use harmless, invented material for this last condition.

The distractor is not an attempt to improve the final transcript. It is a way to investigate whether the system imports suggestions into the answer. For example, your recording might include the fictional project names Signal Garden and Paper Kite, while the extra glossary item is River Lantern. Listen for whether River Lantern appears despite being absent.

Do not change the recording, model version, decoding settings or formatting instructions between these conditions if the interface lets you control them. When a setting cannot be fixed, record that limitation. You are comparing prompt conditions, not staging a contest between three unrelated configurations.

Write down each term’s real occurrences before looking at the AI output. If a term occurs twice, recovery in two places is different from recovery in two places plus an invented third mention. Count both. A glossary that repairs one miss and introduces another consequential mistake deserves a mixed verdict.

Keep the glossary short enough that each entry has a reason to be there. If you cannot explain why a term belongs, remove it from the production prompt. More context creates more material to inspect. It should buy an observable improvement rather than merely make the instructions look thorough.

Separate the jobs inside the output

A transcript can get the words right and the speaker wrong. It can assign speakers correctly while shifting a timestamp past the sentence someone needs. It can translate a sentence smoothly while losing a qualification. Evaluate those jobs separately instead of letting one good-looking result hide another weak one.

For the club recording, make one pass for names and specialist terms. Make a second pass for speaker changes, especially interruptions. Make a third pass for statements containing dates, quantities, negation or uncertainty. These are suggested review lanes, not a complete benchmark or a substitute for checking the rest of the clip.

Suppose the recording says the group might open on Friday if the equipment arrives. A transcript that drops might and if changes the plan even if every project name is spelled correctly. Flag that as a meaning error. Do not bury it in a general count of harmless punctuation differences.

Similarly, label an unknown voice as unknown when your evidence does not establish identity. Speaker A and Speaker B are internal transcript labels; they do not prove who those people are. Mapping a label to a real name requires evidence beyond the system producing a neat sequence of turns.

NVIDIA’s model documentation distinguishes the supported tasks and lists failure modes. Our practical recommendation is to mirror that separation in your review sheet. Give each task its own outcome and unresolved questions. You can accept a model for draft transcription while deciding that its speaker labels still need full manual review.

Compare changes, not just final scores

A compact change log can reveal more than a single quality rating. For each difference between the baseline and glossary version, record the audio location, old text, new text and your listening judgment. Use categories such as corrected, worsened, cosmetic and unresolved. Keep examples brief enough to review together.

Here is a hypothetical outcome, not a test result: the glossary fixes three project-name spellings, introduces one name into an unrelated sentence and changes two commas. Calling that six improvements would be misleading. Calling it a failure without examining the introduced name would also throw away useful information.

The importance of an error depends on the job. An incorrect title in private brainstorming notes may be easy to repair. A wrong attribution in a published interview can change who appears to hold a view. Decide in advance which errors require withholding publication, and which can enter a routine editing pass.

Avoid manufacturing a universal score from a handful of examples. A short pilot can identify a reason to reject a prompt or investigate further. It cannot establish that a model is safe across all your future recordings. Report the sample’s limits in the same place you report any improvement.

After selecting a prompt, try a different short clip that you did not use to tune it. Include a new speaker or a different topic when possible. If you repeatedly revise the prompt after seeing every example, you are measuring how well you can fit those examples, not how well the prompt transfers.

Do not turn recognition hints into meeting conclusions

There is a second boundary worth keeping visible: information that helps recognize a word is different from information that determines what a meeting decided. A glossary entry can suggest how to spell a project. It should not silently supply an owner, deadline or approval that the audio never established.

For a proposed meeting-note workflow, produce the transcript first and review important passages. Build the action list in a separate step that points back to those passages. If a speaker proposes a deadline and nobody accepts it, leave its status unresolved rather than upgrading a suggestion into a commitment.

Ask the system to distinguish explicit statements from inferred summaries. Then check those distinctions yourself. An instruction to be accurate is a request, not proof that the output met the request. The evidence remains the recording and any separately confirmed decisions.

This matters when someone reads only the summary. A transcript error may be visible to a listener who follows along, while a tidy action table can circulate without its source. Give uncertain items a visible status and an easy route to the original moment, rather than hiding uncertainty in an introductory disclaimer.

For shared recordings, agree on access and retention before uploading. Use the processing environment you are actually authorized to use. An open-weight release does not by itself establish that a particular demo, hosted service or local deployment meets your group’s data requirements. Those are deployment questions with their own answers.

A manageable experiment for this week

Choose a two- or three-minute conversation recorded specifically for the exercise. Have participants mention two fictional project names, one ordinary phrase that sounds similar and a tentative decision. Keep the content mundane. Obtain agreement to record and process it, and avoid using anyone’s private meeting as convenient test data.

Prepare the listening reference and the three prompt conditions before generating output. If you cannot run Nemotron with suitable resources, you can still practice the comparison with a speech tool you already have. Be clear that the result then applies to that tool and configuration, not to NVIDIA’s model.

Set a modest question: does a short glossary improve the two fictional names without adding an absent one? Check the tentative decision independently. Preserve all outputs, even the unattractive baseline. A prompt that sounds more sophisticated is not entitled to win.

Spend your review time on the changed passages first, then listen through the complete clip once. The full pass catches errors unchanged by the glossary. Otherwise, you might celebrate a better proper noun while overlooking the same missing negation in every condition.

End with a decision you can explain in a paragraph. Perhaps you keep the short glossary, remove the distractor and require manual checks for names and commitments. Perhaps the baseline is equally useful and simpler. Either outcome is valuable if it follows the evidence rather than the visual polish of the transcript.

The release makes transcription more steerable. Your evaluation should make that steering visible. A useful prompt helps recover what is present in the audio, leaves room for uncertainty and produces an output whose important changes you can account for.