Your AI found the PDF. Did it find the right evidence inside it?

Finding a document is the beginning of an answer

A document assistant returns a convincing response and links to the right PDF. You open it and discover that the answer came from the wrong year, the wrong table row, or a nearby paragraph that only sounds relevant. The citation was real. The connection between the citation and the claim was the problem.

NVIDIA's October 21 Nemotron 3.5 retrieval article describes models for three different jobs: Parse extracts structured content from document images, Embed helps retrieve relevant material, and Rerank reorders candidates against a query. The write-up includes text-and-image retrieval and reports agent evaluations with the surrounding task setup held fixed while retrieval configurations change. Those are vendor-reported results, not measurements we reproduced.

The separate model card describes Parse outputs including layout classes, locations, and reading order. It also lists an October 17 release date for that model, so the October 21 article should not be read as the first availability date of every component. The useful news is the collection's expanded retrieval workflow and the explanation of how its parts fit together.

Our proposed exercise is an evidence trail through those parts. Instead of judging only the final answer, inspect what the system extracted, what it found, what it prioritized, and what it ultimately claimed. A small document project can make those stages visible without requiring an enterprise-scale installation or an invented benchmark score.

Source: NVIDIA: Nemotron 3.5 Embed, Rerank, and Parse retrieval announcement ↗ · NVIDIA: Nemotron 3.5 Parse model card and evaluation scope ↗ · NVIDIA: Nemotron 3.5 Embed 1B BF16 model card ↗

Create a tiny archive with a known answer

Build an exercise from a few documents you own or are permitted to process. For example, invent three programs for a fictional games festival. Give each a different year, a session table, and a short cancellation note. Make the answer to one question require both the table and the note. Keep the original documents available for visual inspection.

A useful question might ask which sessions remain available on the final afternoon. One session can appear in the printed schedule but be canceled in a note on the next page. That setup tests whether the answer connects relevant pieces rather than simply copying the first matching row. Every event and detail in this example is fictional.

Before using an AI tool, write your expected evidence by hand. Identify the document, page, table row, and note that support the answer. Also write what the documents do not establish. If capacity is not stated, the system should not invent a number. If one year lacks a cancellation note, absence of a note should not become proof of a policy you never supplied.

Add a few decoys deliberately. Another year can contain the same session title at a different time. A nearby paragraph can use similar words while answering a different question. These are useful for distinguishing topic similarity from actual support.

Keep the exercise small enough to inspect completely. You are not establishing the quality of a model across all document types. You are learning how to identify the point where your particular application loses the evidence it needs.

Look at the extracted page before blaming search

Start with the representation that enters your index. Does the extracted content preserve which values belong under which headers? Does a two-column page read in a sensible order? Is a cancellation footnote still attached to the session it modifies? If the representation already changes those relationships, the search stage cannot reliably recover facts that are no longer represented correctly.

For your fictional program, place the original page beside the extracted result. Mark each field needed for the question. If the tool exposes locations or structured blocks, use them to navigate back to the source. If it only produces plain text, inspect whether the text still carries enough context to interpret the values.

A model card is useful here because it names the intended output and evaluation scope. NVIDIA's Parse card cautions that basic OCR evaluation on simple synthetic renderings does not guarantee accuracy on complex scans, handwriting, or degraded images. That is a reason to include the kinds of pages your project actually contains, rather than extrapolating from a broad language count.

Do not assume that every extracted table is correct because its formatting looks neat. Read a couple of rows across and down. Check units, dates, headings, and any spanning cells relevant to the question. Preserve the image or PDF page needed for later verification.

When you find an extraction problem, save that example and describe it precisely. The useful report is that a cancellation marker became detached from a named session, not that the AI cannot read PDFs. A narrow description suggests what to inspect next and prevents you from changing unrelated components.

Check the candidate set before the final order

Once extraction is understandable, ask whether retrieval returns the evidence at all. In a small project, inspect the candidate documents or passages before reranking. If the needed cancellation note never appears in the candidate set, a later ranking stage cannot promote it from nowhere.

Compare the query with the form of the information. Your question may say final afternoon while the program uses a date and time. A useful system has to connect those expressions, but your evaluation should make the expected connection clear. Try a carefully specified query as a diagnostic and keep the original question too. Do not replace all realistic questions with perfectly matching keywords merely to make the demo look successful.

When the evidence is present but buried, examine the ranking stage. Ask whether the top result directly supports the requested fact or merely discusses the same festival. A passage from the wrong year can be highly similar and still be wrong for this task. Keep the year in the query and in the evidence shown to the answer generator.

NVIDIA's article distinguishes embedding-based retrieval from a reranker that evaluates query–document pairs. The practical implication we draw is that you should measure whether a new ordering promotes genuinely useful evidence, not just whether the final response becomes more fluent. A different order can improve one question and hurt another.

Record a small set of candidate lists before and after a change. Label the supporting items by hand for this exercise. That gives you a concrete way to see whether a change affected discovery, ordering, or neither. It also makes a disappointing result easier to investigate.

Keep the answer attached to the actual support

Now inspect the final response. Highlight each factual claim and point to the specific evidence behind it. A single document link at the end should not excuse several unsupported details in the middle. In the fictional festival example, a statement about an available session needs the relevant year's schedule and any qualifying cancellation information.

Ask the system to distinguish a supported answer from an unresolved one. If the materials do not establish whether a session requires registration, leave that question open. A response can be useful by saying what the archive contains and what you would need to check elsewhere. Completion should not require inventing a missing policy.

Watch for a change in meaning during summarization. A schedule might say a session is planned; the response might say it is guaranteed. A note might apply to one room; the response might apply it to the whole event. These are answer-stage problems even if retrieval returned the correct passages.

Keep source identifiers and document versions with the response when designing your project. A reader should be able to reopen the same evidence rather than land in a later file with a changed page order. The exact implementation depends on your system, but the user need is straightforward: make the source of the answer inspectable.

For a first demonstration, keep a human review step and use a non-sensitive archive. Good evidence retrieval does not automatically make a system appropriate for every decision. Start with a task where a mistaken answer is easy to detect and correct while you learn the failure patterns.

Change one stage and keep the question fixed

An appealing new model collection can tempt you to replace the whole pipeline at once. For a personal experiment, keep a working reference and change one stage where practical. Compare the same saved questions and source versions. If you change extraction, retrieval, ranking, and prompting together, describe it as a new overall system rather than attributing every improvement to one component.

Measure useful endpoints. Did the supporting material enter the candidate set? Did it appear high enough to reach the answer stage? Did the final claim preserve its meaning? Did the response make clear when evidence was missing? These checks can be recorded in a simple sheet without inventing an elaborate score.

Keep failures alongside successes. One beautifully answered question is not a reason to discard a case involving a difficult table. If you refine the system around your initial examples, reserve a different small set for the next comparison. Otherwise you are measuring familiarity with your own development cases.

Also note the practical work required to run the system: preparing the documents, handling failures, reviewing output, and maintaining versions. A vendor's throughput number on specified hardware does not tell you the full experience of your application. This article makes no prediction that a named model will fit or perform well on your laptop.

If you proceed to installation, use the current official model cards and deployment documentation, including their terms and environment requirements. Do not treat this conceptual walkthrough as a ready-to-run configuration.

The benefit of separate components is not only another chance to improve a benchmark. It is a better vocabulary for debugging. You can say the page was extracted incorrectly, the evidence was not retrieved, the useful result was ranked too low, or the final answer overclaimed. Each description points toward a different next step.

Make the small project teach you something reusable

At the end, write a short account of one success and one failure. Include the question, the expected evidence, the intermediate outputs you inspected, and the final response. A friend should be able to follow the account without access to your memory of the experiment.

For the fictional program, the most useful discovery may be that a date qualifier gets lost before indexing. Or you may find that the evidence reaches the model correctly and the wording of the answer needs attention. Either result is a meaningful diagnosis. It is more informative than calling the whole project accurate or inaccurate after a few prompts.

Keep the exercise artifacts small and portable. The original files, extracted output, candidate lists, and your notes can form a reference set for future changes. When a later version behaves differently, you will have a concrete comparison instead of a vague recollection that it used to work.

NVIDIA's new write-up offers more tools for constructing that pipeline. Your durable skill is knowing which stage has to preserve which part of the evidence. A correct PDF link is a good start. A useful answer lets the reader trace the actual claim all the way back to the right place inside it.