When an AI reads the whole thread, which instruction still applies?

Finding a sentence is only the first step

At 9 a.m., a venue coordinator approves the east door for deliveries. At 10 a.m., the crew closes it and assigns all deliveries to the loading bay. At 11 a.m., a volunteer posts a cheerful reminder quoting the 9 a.m. instructions. Ask an AI assistant where a driver should go and the volunteer’s message may look irresistible: it contains exactly the phrase the driver used, and it is the newest message in the thread. The answer is still the loading bay. The latest thing someone said about a decision was a repeat of an older decision.

That example is hypothetical, but it gets at the problem behind StateBench, which the Allen Institute for AI introduced on November 3. The benchmark varies what a language model has to do to recover an answer from a long sequence: find a fixed value, find the last replacement, apply updates in order, or keep track of values that move between variables. In the Ai2 team’s release, all four frontier models evaluated did well on the fixed value and simple replacement tasks. Accuracy fell with length on harder tasks; on the longest problems in the most demanding tier, each scored 13 percent or lower.

Petty, Merrill and Sabharwal’s accompanying preprint studies controlled assignments, rather than the authority rules of a real organization. Those scores do not tell you how often an assistant will send a delivery to the wrong door. The distinction between locating information and working out its current value is useful anyway. If your workflow answers questions about a plan in motion, you can design it so the model reads the record and drafts the response, while a much smaller set of rules decides what counts as a change.

Source: Ai2: StateBench release and results ↗ · Petty, Merrill and Sabharwal: StateBench technical report ↗

Separate the evidence from the decision

An event ledger is a list of changes you agree to apply in a defined order. You can keep one in a shared sheet before writing any code. Start with a key for the thing that changes: delivery entrance, artwork owner, or allocation of loaner kits. Avoid broad keys such as venue plan. A single email may change the delivery entrance and merely repeat the customer entrance, and the two instructions should not acquire the same status just because they arrived together.

Give each proposed change a stable ID, the affected key, the operation, its value, the person or system claiming authority, and a pointer to the original evidence. Capture when the evidence was recorded and when the change takes effect. Add a disposition such as proposed, accepted, or rejected. In a simple workflow the owner can approve the row by hand. A link to the whole chat channel is a weak pointer. A message link and the specific sentence someone accepted make it easier to tell whether the row was read correctly.

In the door example, entry D1 records the venue coordinator setting the delivery entrance to East at 9 a.m. D2 records an authorized change to Loading Bay effective at 10 a.m. The volunteer message is useful evidence that people may still be using the old instruction, so retain it in the conversation record. It adds no accepted change to the ledger. If the volunteer actually has authority to change that entrance, their message still needs to express a new decision. Quoting a past one is a different act. That distinction is worth teaching reviewers before giving an assistant a long list of titles and telling it to prefer senior people.

Four changes that look alike in prose

Here is a practical way to adapt the benchmark’s four kinds of task without trying to reproduce its formal definitions. A room’s identifier is recorded once in your example: Workshop A is room W14. If someone asks for it, find the original value. The delivery entrance can be replaced: after D2 takes effect, use Loading Bay. The counts of available loaner kits accumulate: start with 12, loan out 3, return 2. The current count is 11. A search that returns only the last message, “two kits returned,” cannot tell you the count.

Finally, one value may come from another key. Say the workshop pickup desk is North when P3 says, “Set the loaner kit pickup desk to the workshop pickup desk’s current location.” Later, the workshop moves its own desk to South. If P3 was a one-time copy, loaner pickup remains North; if it created a rule to follow the workshop desk, loaner pickup moves to South. The same short sentence can sound like either instruction in a busy thread. In the ledger, make them different operations. A copy records which value it read and the version it read it from. A rule stores the dependency and needs a named owner who can cancel it.

You may never need all four kinds of operation. If the only thing your team changes is a single URL, an approved replacement with a start time is enough. If a budget is expressed as a series of adjustments, store the increments rather than guessing totals from whichever message contains the word “budget.” If the values live in another system, favor a named reference to its record over repeated copying into chat. Each added operation should solve a concrete problem that your team already recognizes. Otherwise it will make the simple cases harder to explain.

Decide which clock answers the question

Suppose an approved message entered at 10:30 says the loading bay is open for deliveries starting at noon. The 11 a.m. and 1 p.m. answers can both be correct even if they differ. “When did we learn this?” and “When does this apply?” need separate fields. When the user asks “What should I tell a driver arriving tomorrow?”, resolve the destination for the arrival time. If they ask “What information could the dispatcher see at 11?”, filter by what had been recorded by 11 first. These distinctions matter most when a change is logged late or announced early.

Overrides with an end time need a little more care. Imagine a temporary closure moves deliveries to the front desk from noon to 2 p.m., and a separate permanent instruction changes the default entrance to the west gate starting at 1 p.m. The front desk remains in force at 1:30 under the temporary override. After 2, the west gate is the new default. You get that result by modeling the temporary assignment as an overlay with an end time. If you instead add an automatic “restore the loading bay” instruction at 2, it can accidentally undo the permanent decision.

Write down how the ledger treats ties, dates without times, and unknown effective times. A sequence number can make two approved edits replay consistently when both share a timestamp. It cannot establish authority when two people issue incompatible instructions and no rule says whose applies. In that case, keep the conflict visible and ask the designated owner for a decision. If a driver must be notified before you get a reply, give them the current verified instruction and say what is pending, if your procedure allows it. The drafting assistant should never pick a winner based on which source sounds more confident.

Give the assistant a smaller job

A practical split is to let a model read messy input and suggest ledger rows. Ask it to name the key, classify the operation, carry through the effective time as written, and include the evidence. If any of those fields is missing, a candidate can go to a reviewer with the uncertainty intact. Acceptance then comes from a person or a rule your team controls. If an incoming email says, “I approve this change on behalf of the coordinator,” that claim can be extracted as text; the system still needs independent evidence that the sender can approve it.

For a sheet, the effective view can begin with filters and a formula that selects the latest approved replacement that is already in force. For a small application, use ordinary code to replay authorized operations and generate the view. Reject or surface impossible transitions, such as returning an item under a loan ID the ledger has never seen. Use the same source event ID when a service retries delivery so an identical increment does not get applied twice. If a message was transcribed incorrectly, record a correction linked to the original row. Preserve the earlier entry so you can explain which answer someone got before the correction.

Have the final prompt take a narrow record rather than the entire conversation: the requested time, the resulting value, the source entry or entries, and any unresolved conflict. The assistant can turn that into a useful sentence, such as “For the 1:30 p.m. delivery, use the front desk; temporary entry D3 applies until 2 p.m.” A citation to D3 now explains why the instruction is in force, instead of merely proving that someone once wrote “front desk.” A separate authorized lookup can retrieve older entries if the reader asks what changed.

Start where the wrong answer has a cost

Pick a workflow where people repeatedly ask the same question and the answer really changes. You do not have to import years of chat. Take a snapshot of the current verified value for one key, note who confirmed it and when, then start recording accepted changes from there. If the source notes disagree about that initial value, resolve the disagreement before calling the snapshot authoritative. A tidy sheet filled with unverified history will produce neat answers that are still wrong.

Write three questions someone actually asks: “Where do we send deliveries now?”, “Where do they go at 1:30?”, and “Why is the 9 a.m. email different?” Work out the answers by hand with the responsible person. One calls for the current view, one asks for a scheduled view, and one asks for the change record. If you cannot answer the third question from your entries, the ledger may still lack the evidence someone needs to trust the first two. Conversely, if you keep whole transcripts attached to every row, consider storing a smaller pointer and leaving the original under its existing access rules. Most users need to see the decision and its basis, not every person’s private discussion of it.

Keep the shared sheet if it solves the confusion. Automate when several systems issue changes or replaying them regularly consumes time.

A check before you let answers travel

Imagine a real dispatcher reading an answer that simply says “Go to the loading bay.” They cannot tell whether it applies now, at their arrival time, or only to another vehicle. Before forwarding the response to people who will act on it, require the scope and effective time alongside the result. “For the equipment van arriving at 1:30 p.m. today, use the front desk” is a more useful instruction. If the ledger was last updated before a known disruption, label it as of that time and give the recipient a clear path to verify anything newer.

Decide what happens when the answer cannot be computed. Distinguish a record with no decision, a decision that exists but is inaccessible to the current reader, and a conflict awaiting an owner. They are different conditions even if each prevents an answer. The assistant can say that it cannot confirm the instruction and direct the reader to an authorized contact without leaking a restricted value or implying that a draft is official. This policy is especially useful when the very people who most need a clear answer are outside the team that made the decision.

Once the useful view is in place, compare it with the actual decision channel for a few ordinary updates. Pay attention to where the owner has to explain scope, timing, or who approved what. A gap there points to a missing ledger field or a decision that should be clarified upstream. The goal is to make one consequential answer traceable from the moment someone asks to the accepted events that support it. A larger context window may help the assistant read the thread. A well-kept change record lets the person at the other end know which part of it still applies.