Before the AI Draws a Conclusion, Check the Measurement Trail

The missing step between a file and a conclusion

An AI assistant receives a spreadsheet, a photograph and a handwritten note. It produces a clean graph and a confident explanation. The graph might be arithmetically correct while answering the wrong question: perhaps the photograph shows a different sample, the spreadsheet contains an earlier run, or the note describes a setting that changed halfway through. A result becomes useful when someone can follow the path from the evidence to the claim, including the places where that path is incomplete.

OpenAI’s November 6 announcement introduces BioBasicsBench, developed with Ligo, to evaluate agents on 100 tasks involving routine wet-lab analysis. The tasks use raw experimental evidence and expert-written grading criteria. The announcement emphasizes recognizing errors and situations where the evidence cannot support a conclusion. Its headline scores are averages of the available rubric credit, not the proportion of experiments an agent could independently carry out successfully. OpenAI also states that higher scores do not yet establish reliable lab judgment.

That news offers a broader lesson for anyone asking AI to analyze measurements. A polished answer should preserve the relationship between the files, their labels, the calculation and the interpretation. You do not need a biology experiment to practice checking that relationship. A small fictional classroom exercise about colored water is enough to reveal several common mistakes.

The exercise below is entirely invented for explanation. No experiment was performed, and the numbers are not evidence about a dye, camera or instrument. Imagine that a class receives already recorded light-intensity readings for colored-water samples, plus photographs and a worksheet. The job is to describe what the records support. It is a lesson in handling supplied data, rather than instructions for making samples or conducting a laboratory procedure. The useful output is a short analysis that another person can audit, with a clear account of what remains unknown.

Source: OpenAI: Measuring how AI agents handle everyday wet-lab science ↗ · NIST: Metrological Traceability, Frequently Asked Questions and Policy ↗

Build an evidence map before a chart

The fictional folder contains a spreadsheet with rows labeled A, B and C; a photograph showing three containers labeled 1, 2 and 3; and a note saying that labels were changed before the second recording. The spreadsheet gives a time for each reading. The photograph has no visible clock, and its filename says only “final.” A quick analysis might assume A corresponds to 1 and combine everything into one story. That would turn an unresolved identity into a hidden premise.

Start by making an inventory. For each file, record its name, what it contains, the date or time that is actually present, and which sample identifiers appear. Keep the original file unchanged. A working copy can contain convenient names and extracted values, but those names should point back to the supplied labels. If the photo is called “sample_A” in the analysis, a reviewer needs to know whether that name came from the original evidence or from an analyst’s guess.

Ask the assistant to list links it can establish and links it cannot. The spreadsheet may establish that row A has three intensity readings. The photograph may establish that container 1 appears darker than container 2 in that image. Neither observation establishes that container 1 produced row A. The note introduces a reason to question the mapping, but it does not provide the missing mapping itself. The analysis can proceed on the spreadsheet alone while treating the photograph as unresolved evidence.

This small discipline changes the questions you ask. Instead of asking why sample A looks darkest, ask whether the records connect sample A to any photographed container. Instead of requesting a corrected table, ask which corrections have documentary support. If the teacher later supplies a mapping sheet, add it as new evidence and record which conclusions change. The assistant should be able to explain the revision without overwriting the earlier uncertainty. That history matters when an attractive chart has already made an assumption feel like a fact.

Name the measured quantity

Suppose row A contains the readings 62, 63 and 61, and row B contains 48, 49 and 47. The worksheet calls the column “sensor output” but gives no unit. An assistant can calculate means of 62 and 48 and state that the average recorded output for A exceeds B by 14 units of the supplied scale. It cannot honestly rename those values as dye concentration, light absorption or percentage purity merely because one of those quantities would fit a familiar classroom story.

Ask what relationship the supplied records establish between the instrument’s output and the quantity of interest. If a reference sheet defines the output as an arbitrary intensity scale, use that name. If another document supplies a conversion, identify the document and state the assumptions under which it applies. A formula in a generated answer is not evidence that the conversion belongs to this data. A convincing-looking axis title can carry an unsupported claim just as easily as a paragraph.

NIST’s measurement guidance makes a useful distinction here: metrological traceability belongs to a measurement result, supported by a documented chain of calibrations, rather than to an instrument in the abstract. NIST also cautions that traceability alone does not establish fitness for a particular purpose. Our fictional classroom records do not establish such a chain. Keeping their provenance is useful, but it should not be described as formal metrological traceability or certified measurement.

Give the assistant a narrow reporting rule: preserve the original quantity name and units, and mark an absent unit as missing. If it proposes a derived quantity, require it to identify the input fields and the supplied relationship. A graph can still be useful when its vertical axis says recorded sensor output in unspecified units. That title is less impressive than concentration, but it lets readers understand exactly what was compared and what further information they would need for a stronger claim.

Separate variation from an explanation

The three readings for A sit close together, as do the three for B. That tells you something about the spread of those recorded values. It does not, by itself, show that the measurement process was accurate, that the samples differed only in dye content, or that the same relationship would hold on another day. Repetition answers a narrower question than people often give it credit for.

Look at what was repeated. Were the three values readings of the same container, records from three separate containers, or copies of one value entered during a spreadsheet exercise? Those cases have different meanings. If the worksheet does not say, the analysis should retain that ambiguity. Nine rows can represent nine independent observations, three objects measured three times, or a mixture. Counting rows before understanding their origin can exaggerate how much evidence the dataset contains.

Imagine the fictional notes also say that the room lights were adjusted between the records for A and B. This is a possible alternative explanation for a change in recorded output. It is not proof that lighting caused the difference. A careful summary would report the difference and the documented setting change, then explain that the available records do not separate sample differences from measurement conditions. That is more informative than either ignoring the note or confidently blaming the lights.

Ask the assistant to place three sentences next to each other: what the records show, which interpretation is being considered, and what would distinguish it from a competing interpretation. The first sentence might report the observed means. The second might propose that the samples differ in how much light reaches the sensor. The third might identify the missing evidence about the recording conditions. This makes uncertainty actionable without inventing a new experiment or pretending the existing data can settle the issue. The task is to identify the information gap, then let the responsible teacher decide how to address it.

Preserve inconvenient evidence

Now add a fictional row C with readings 40, 41 and 72. The third value is conspicuous. An assistant might remove it automatically, call it an outlier and present a tidier chart. That choice would determine much of the conclusion before anyone had established whether the value was an error. A surprising observation is something to inspect, not permission to erase a row.

Keep the original series visible. Ask the assistant to check for documentary reasons the value might be invalid: a note identifying a transcription mistake, a file that contains the original reading, or a stated rule about excluded records. A value can be unusual and still be correctly recorded. Conversely, a value can fit the expected pattern and be wrong. Agreement with the story you hoped to tell is a weak criterion for accepting data.

If there is no documented correction, the analysis can show how much the summary depends on the value. For example, the mean of the three supplied readings is 51, while the first two average 40.5. Those calculations do not justify choosing the smaller figure. They show that excluding the last reading would materially alter the summary. State that any two-reading result is a conditional comparison, and explain which supplied record was omitted. A reviewer can then see the effect without mistaking it for the preferred answer.

Apply the same care to photographs. An image may show glare, a cropped label or containers at different positions. Describe visible features before attaching a cause. If a bright patch could be a reflection, the image alone may not establish that a sample changed color. If the assistant cannot read a label, it should report the uncertainty rather than completing the sequence from neighboring labels. Keep the uncropped image available alongside any enlarged region. A crop is useful for inspection, but it can remove the context that explains an apparent anomaly.

This approach also helps when sources conflict. Keep both readings and the reason they disagree in the record. The worksheet may use an old name while the photograph uses a new one. Resolving that conflict requires evidence about identity, not a vote among files. Multiple copies of the same mislabeled worksheet do not become independent confirmation simply because the assistant found them in several places.

Make the final analysis reconstructable

A good report lets a classmate reproduce the numerical summary and understand why the wording stops where it does. Begin with the question the supplied evidence can answer: how the recorded sensor outputs compare within this dataset. State which files were used and which were left unresolved. Then give the calculations with enough detail to check them. For a small table, that may mean naming the input rows, the number of readings and the treatment of missing or questionable entries.

Keep arithmetic checks separate from interpretation checks. Confirm that the means use the intended rows and that the graph plots the values the caption names. Then inspect the claims in the prose. Does the summary quietly turn recorded output into concentration? Does it treat three repeated readings as three different samples? Does a sentence about consistency overlook the unexplained 72? An answer can pass every arithmetic check while failing one of these interpretation checks.

The final paragraph should distinguish observations from inferences. An observation can state that A’s supplied mean is higher than B’s. An inference might suggest that the samples differed in an underlying property, provided the recording conditions were comparable. The report should say whether that condition is documented. If it is not, preserve the numerical comparison while withholding the stronger explanation. This is a useful conclusion, not an unfinished version of a more confident answer.

Save the analysis instructions and the working table alongside the report so another person can follow the same choices. If an AI assistant revised a label, excluded a row or applied a conversion, the reviewer should be able to locate that decision. A compact change log is enough: what changed, which evidence justified it and which outputs were affected. Avoid presenting a generated explanation as a record of something the class actually observed.

BioBasicsBench’s focus on raw evidence is a reminder that interpretation begins before the final calculation. In everyday analysis, the useful habit is to keep asking which file supports a statement, which relationship was assumed and which uncertainty survived the summary. That habit turns a fluent answer into work someone else can inspect. It also makes an honest limit easier to recognize: sometimes the measurements support a clear comparison, while the explanation people want remains unresolved.