Counting AI sessions is easier than measuring scientific progress

A session tells you what someone tried

Imagine a small astronomy group studying whether the night sky near town grew brighter after a streetlight replacement. A researcher opens an AI assistant to help convert observation timestamps. Another asks for a draft figure caption. A third spends an evening checking a proposed change to the analysis. By morning the group has logged a dozen conversations. It might also have discovered that the observations are too sparse to answer the question. Both things can be true.

On November 5, Anthropic published a study of 207,049 sessions from Claude consumer plans between August 2 and September 30. The sample includes Chat, Cowork and Claude Code sessions that a classifier assigned to a scientific field and task. The company says its research team analyzed the labels rather than retaining the content of the conversations for this report. Data analysis accounted for roughly a third of sessions in Chat and Cowork and nearly half in Claude Code. In a separate survey of 528 scientists who actively use AI, respondents rated generating ideas and gathering data as the most important categories for producing important research results. Those categories accounted for smaller shares of the observed sessions.

The contrast is useful because it separates where an assistant gets used from what a scientist thinks moves a project forward. Anthropic also modeled scenarios for research acceleration. Those are conditional projections, not gains observed in the conversations. A local team can take a more modest next step: measure whether AI helped it reach a result it can defend, and identify what work was required to get there.

Source: Anthropic: AI for scientific discovery: How researchers use Claude ↗ · National Academies: Reproducibility and Replicability in Science, Executive Summary ↗

Start with what the instrument cannot see

Anthropic’s unit is a session. A person who opens a new conversation for every plotting question produces a different trail from someone who keeps one thread open for a month. Neither the number of sessions nor the number of people can be inferred from the other without more information. Even two sessions on the same task can mean opposite things: a quick check that saved effort or a second attempt after the first answer failed.

The sample is also bounded by product and plan. It tells readers about sessions that the study observed and classified. It cannot see everything researchers did with other assistants, institutionally managed products, local software or no AI at all. The accompanying survey recruited active adopters and primarily includes life and health scientists. That makes their priorities useful evidence about the respondents, with no need to make them stand in for every scientist.

Now apply the same reasoning to the astronomy group. Its volunteers may open the assistant most when they are working on unfamiliar analysis code. The people who already know the code might use it for unusually difficult problems only. Comparing the average duration of their assisted and unassisted tasks would mix the effect of the tool with the reasons each person picked it up.

Ask what the log omits before adding another metric. The assistant sees a question about matching observation files. It may not see the two hours someone spent examining a sensor that was pointed at a new location, or the meeting where the group revised its research question. An honest measurement plan names these missing stages. It does not interpret their absence from a chat history as evidence that they are unimportant.

Choose a scientific decision to measure

For the fictional project, define a milestone the team actually needs: decide whether the observations before and after the lighting change are comparable enough to estimate a difference. The first useful deliverable might be an inventory of observation nights with the station, sensor, timestamps and reasons any record was excluded. A publishable result may still be months away.

That milestone makes the work visible at the right scale. A well-formatted table is helpful if it correctly aligns the records and exposes missing information. It is much less helpful if the table fills a gap with a guessed timestamp. A proposed analysis might save effort by exposing a problem early, even if it leads the group to abandon a planned comparison. Count the defensible decision to stop as progress.

Decide what evidence a colleague would need to reach the same decision. They might need the original files, a script that produces the inventory, a note on timestamp conventions and a documented rule for flagging records. The National Academies’ report on reproducibility distinguishes rerunning a computation with the same inputs and methods from asking whether another study with new data reaches a consistent result. A colleague rerunning the inventory checks the first kind of claim. It does not establish that another town would show the same change in brightness.

For example, the club might discover that most of its before observations came from the park and most of its after observations came from a school roof. That fictional imbalance would give it a reason to look for measurements from the same site on both sides of the change. It would also explain why an attractive plot showing two different averages might answer a different question from the one the group intended. A useful assistant could help expose the imbalance by summarizing the inventory. Whether the sites can reasonably be combined is a decision for researchers who can explain what makes the observations comparable.

Write a definition of completion the team can apply without knowing how the work was done. For example: a colleague can rerun the inventory, trace a few rows to their source and explain why an ambiguous observation was set aside. Measure how long it takes to get there. The first draft is an intermediate event, however quickly it appears.

Compare the work without confusing it with the worker

A small team can begin with a deliberately narrow comparison. Use an archived or synthetic observation packet whose expected structure is already understood. Ask two colleagues to produce an observation inventory with the same required fields, data and review procedure. Give one access to the assistant for that packet and give the other the usual scripts and documentation. On a second comparable packet, switch conditions. Keep the exercise separate from the final scientific analysis.

This design can reveal differences worth following up on. It does not erase differences in people’s skills, and the second task may benefit from what they learned on the first. Vary the starting condition if you have more participants. Make sure neither packet is simply a copy of the other with new filenames. Then record which tasks each person actually did and whether familiarity with a particular instrument might explain a result.

Evaluate the work with the same checklist. Is every accepted file traceable? Does the script leave uncertainty visible? Do exclusion reasons match the supplied records? If possible, have a colleague review the outputs without being told which workflow produced them. The formatting might still give clues; the purpose is to keep reputation and expectations from being the main basis for judging a result.

Record failed and incomplete attempts. If a participant abandons an assistant-generated script and starts again, both periods belong to the assisted workflow. If an unaided participant reuses a reliable library routine, that belongs to the usual workflow. A trial designed to compare the team’s choices should include the tools its members would realistically use. Removing ordinary aids from one condition creates a comparison the team cannot apply.

Treat the result as a local pilot. If one workflow looks promising, repeat it on another appropriate task before deciding that it generalizes. An assistant that helps translate file formats may contribute little when someone needs to decide whether a change in instrument placement invalidates a comparison.

Follow the saved time to the next stage

Time saved has a destination. Suppose the assistant drafts a script in minutes, but a collaborator must spend the afternoon tracing an unnoticed assumption about which timestamps are local. The draft came quickly. The time to a checked inventory did not necessarily fall. Track preparation, review and repair separately so the group can see where the work moved.

A reasonable record has a row for each completed or abandoned work item. Include the starting condition, the artifact it produced, the time to an acceptable review, a note on any significant repair and what the result allowed the group to decide. If a researcher used an assistant on one small part of a longer task, note that scope. An entire analysis should not be counted as automated because a caption or import script was assisted.

The group should also track what was waiting on something else. If no one can collect additional observations until a sensor is serviced, accelerating the plot code will not move that appointment. The time saved can still matter. It might free a colleague to document the instrument history or prepare a cleaner handoff. Describe that gain where it occurred instead of automatically converting it into days off the whole project.

Look for changes in evidence quality, too. A workflow that takes the same time but leaves a script another person can rerun may be a good choice. A workflow that encourages the team to check alternative explanations sooner can reveal why an apparent pattern should not be reported. On the other hand, a workflow that produces many plausible plots may create more decisions about what deserves attention. Keeping a record of which plots answer the agreed question helps separate exploration from progress.

Keep the result narrow enough to be useful

At the end of a trial, write what the assistant helped with and how the team knows. A defensible conclusion might read: “In our practice comparison, the assistant helped draft a file inventory. The output was accepted after a colleague checked it against the source records and reproduced it from the saved script.” That sentence is an example of useful wording, not an account of testing conducted for this article. It leaves the brightness question open and says nothing about results at another site.

Include conditions that would make the decision worth revisiting. A new sensor, a different file format or a task that requires unpublished observations may change what the team is willing to send to an assistant and what review is feasible. Keep the minimum notes needed for the comparison. There is no reason to turn the group’s record of work into a permanent archive of private conversations, unpublished ideas or full prompts.

Make the next decision practical. The astronomy group might use the assistant to draft import routines while keeping calibration choices with people familiar with the instruments. It might decide to have another person review inventories regardless of how they were prepared. It might abandon the tool for this stage if it mainly creates rework. Each decision can follow from evidence about one project without claiming to settle whether AI speeds up science in general.

Anthropic’s report gives researchers a map of activities that already bring some scientists to Claude. The project team needs a map of its own decisions and the evidence required to make them. Connect the two at a small, reviewable milestone. That is where a conversation can become useful scientific work, or show the team what still has to happen before it can say so.