An AI Answer Can Be Accurate and Still Fail the Help Desk

The answer can be true and still fail

A resident asks a help desk where to collect the key for an evening meeting at a community center. The assistant gives the correct office hours. It leaves out that evening organizers collect the key from a different desk. The reply contains a true fact and still sends the resident to the wrong place. Now imagine that the assistant includes the correct pickup instructions but also prints an access code reserved for staff. Those two failures need different responses. Calling both answers simply “bad” hides what went wrong.

On November 4, Google shared a white paper describing how it evaluates generative AI for health use cases. The framework distinguishes accuracy, safety and usefulness, and combines closed-question benchmarks, automated rating and expert human review. The authors begin with specific user journeys and consider evaluation at different stages of a system’s life. The paper describes an evaluation approach. It does not establish a medical outcome, and the example here applies the distinction to a fictional community center help desk, not to care.

The distinction matters because evaluation questions tend to collapse into whatever is easiest to count. A help desk can count whether the hours match a published page. It takes a different check to see whether the answer exposed a private detail, and another to see whether a resident could complete the handoff. A team that reports one answer score may discover too late that it was measuring the easiest part of the interaction.

Start with a specific service and a specific moment. In this example, the assistant can explain published room-booking rules, directions and opening hours. Staff handle access exceptions, changes to an existing reservation and requests about another group’s booking. Those boundaries give reviewers something concrete to assess. The question is no longer whether a model is good at customer service in general. It is whether this proposed help desk can answer the questions it is meant to answer and handle the rest responsibly.

Source: Google: Our framework for building safer and more helpful AI for health ↗ · Montgomery-Taylor et al.: Towards Multi-Method Evaluation of AI in Health (arXiv preprint, v2) ↗ · NIST AI RMF Playbook: Measure ↗ · NIST AI RMF Playbook: Manage ↗

Give each dimension its own question

Accuracy is about whether the answer matches the evidence available for the case. For the community center, reviewers can attach the dated public policy, the site schedule and any special notice that applies. A reply that gives Saturday hours for a Friday event is inaccurate even if it is friendly and easy to follow. A reply that gives the regular Friday hours but ignores a posted closure is also inaccurate for that case. The reference material needs the relevant exception as well as the ordinary rule.

Safety asks what could go wrong if someone follows the reply or if the system takes an action. At this help desk, one concern is disclosure of restricted information. Another is telling someone that a reservation changed when the assistant has no authority to change it. These checks require more than a factual comparison. A real access code is accurate and still unsuitable for a public answer. A false claim that a room was moved can be consequential even if a human can eventually repair the record.

Usefulness asks whether the person has a workable next step. An answer can be cautious and factually correct but send someone through an unnecessary loop. If the policy says people can reserve a public room through a form, pointing them only to a long policy page makes them do work the assistant was supposed to save. If an exception needs a human, a usable answer can name the relevant desk and explain what information the organizer should have ready, without making promises on behalf of staff.

Score the same case on all three dimensions. Suppose a resident asks whether the center has a room available and wants a booking moved. A good response may explain the public availability process and route the change to staff, while acknowledging that it cannot confirm the new slot. A blanket refusal might avoid making an unauthorized change, but it gives up the part of the request the system could answer. A rating form that lets reviewers mark safety and usefulness separately makes that lost value visible. It also leaves room for a legitimate handoff to count as useful.

Make the case set match the job

Build an evaluation set from the kinds of tasks the desk is responsible for, including the moments when people get stuck. A straightforward question about a room’s location belongs in the set. So do a person who names the wrong branch, a request that arrives near closing, and a message that contains both a public question and a private request. People do not usually format questions as clean units of work. The messy combinations reveal whether the assistant can answer the permitted part and route the rest.

Keep the expected behavior alongside each case. Write down what the assistant may say, what it must avoid and what counts as an acceptable next step. For a request about someone else’s reservation, reviewers can agree that a response must not confirm the booking. They can allow more than one useful response, such as directing the requester to the organizer or explaining the center’s public booking process. That makes the rule checkable without forcing every good answer to use identical phrasing.

Vary one detail at a time in a few cases. Compare a message asking for public entrance hours with one that adds a request for the staff door code. Compare an inquiry about switching rooms with a similar message that says staff already approved the switch. The second message does not give the assistant independent proof of approval. These paired cases help reviewers see where a change in wording should alter the response and where it should leave the policy intact.

Separate examples for improving the system from examples for checking the revision. If the team repeatedly edits instructions to solve the same evening pickup prompt, a passing result on that prompt may reflect familiarity with the example. A new case could describe a weekend rehearsal or phrase the pickup question indirectly. Reserve some cases for a reviewer who did not write the prompt guidance. Keep everyday traffic and deliberately difficult cases labeled. Both are useful, but a collection built to expose problems should not be presented as a sample of how often those problems occur in the world.

Keep failures visible when you summarize

Imagine a fictional evaluation set with 100 cases: 90 routine questions and 10 that require a handoff. The help desk handles the 90 routine questions and misses three handoffs. A single combined count would show 97 cases handled as expected. A separate count shows three misses in the 10 cases that needed staff. The figures describe the same hypothetical run. Only the second count tells the person responsible for the queue where the problem lives. Neither count establishes a live failure rate.

Keep denominators next to the results. Report the restricted-information cases separately from ordinary accuracy checks, and show whether each category contained a handful of examples or a substantial set. A zero in a small category means the team saw no failures there under those conditions. It does not establish that the risk is zero. If the answer can vary across runs and the team repeats a case, count cases and attempts separately. Three failed attempts on one prompt are not three distinct user scenarios.

Agree before grading on what happens when a severe failure appears. In the community center example, the team could set a rule that any observed disclosure of a restricted code blocks the relevant workflow from proceeding to the next stage of the pilot. A clumsy but accurate directions answer may instead enter a backlog for improvement. The point is to choose a response that fits the consequence. A long list of correct directions cannot offset a disclosure that requires staff action.

NIST’s AI Risk Management Framework Playbook distinguishes measuring risks from deciding how to respond, and calls for prioritizing treatment with attention to impact and likelihood. Applied here, the summary should help an owner make a decision. Record the failure category, what it affected, who owns the response and what evidence will justify reconsidering the decision. If the team changes which documents the assistant can see after a disclosure, it can repeat the original case and test nearby variants. A cleaner score on unrelated public questions would say little about whether that particular problem was addressed.

Evaluate the handoff as part of the answer

A handoff is an action with a beginning and an end. In a fictional evening pickup scenario, the assistant should identify that a staff decision is needed, explain the next step and route the request to an actual owner if it is authorized to do so. A chat bubble saying someone will be in touch cannot prove that a ticket arrived. Reviewers should check what reached the receiving queue, how much information was shared and what the resident sees if nobody is available.

Map the handoff to the help desk’s real operating assumptions before testing it. Does the fictional evening desk have somebody on duty, or does a message wait until morning? Is the assistant allowed to submit a request, or can it only provide contact details? What does the user see if the queue rejects the submission? Reviewers can model those conditions with a test queue and fictional records. If they only inspect the text, they should label the result as a review of the handoff message rather than of the complete process.

Measure errors on both sides. A missed handoff leaves someone with the impression that the assistant resolved something it could not resolve. An unnecessary handoff sends simple work to staff and may make service slower for people whose requests actually need judgment. Split the queue cases into handoff required, handoff acceptable and answer expected. That middle category lets the team examine tradeoffs without treating every transfer as a mistake. For instance, an ambiguous branch name might be resolved by a clarifying question or by routing to a staffed desk, depending on what the requester can provide.

Reviewers can also check what the transfer asks of the resident. If they have already named the event and branch, a handoff should not require starting from a blank description unless there is a good privacy reason. At the same time, the queue should receive only the information it needs. Score the tradeoff concretely: did the transfer arrive, did staff have enough to act, and did the message avoid forwarding unnecessary details? Those checks reveal more than a general rating for politeness.

Match the reviewer to the question

A fixed answer check works well when the policy gives a single clear fact. A simple comparison can tell whether a reply named the public entrance specified in the fixture. A separate automated check may flag a fictional restricted code if the string is present. These checks are cheap enough to rerun after changes, and their limits are easy to explain. They cannot judge whether an ambiguous message promised a booking change in language the team did not anticipate.

An automated language-model rater can help sort a larger set of open-ended responses, but compare its judgments against human ratings before trusting the summaries. Try cases with awkward but valid replies and polished replies that quietly ignore an exception. If the rater consistently rewards the polished versions, the summary may measure writing style more than service quality. Use staff who understand booking rules for disputed authority cases, and ask someone unfamiliar with the internal policy to read the next steps as a resident would.

When reviewers disagree, capture the reason. Perhaps the person who works the desk knows an informal workaround that the public policy never mentions. That disagreement may reveal a documentation problem rather than a defect that can be fixed in the model. Resolve it with the service owner and update the expected behavior before rerunning the case. If multiple responses are legitimate, describe what they have in common instead of reducing the rubric to a preferred sentence.

A useful evaluation record ends with decisions people can revisit. It states which workflows were examined, where each kind of failure appeared, what remains untested and who can stop or narrow the pilot. For the community center, that could mean continuing with published directions while keeping changes to reservations with staff. The record is stronger when it explains why those permissions differ. A reassuring average would leave the most important part unsaid: whether the assistant knows where its useful work ends and somebody else’s responsibility begins.