When AI picks the models, measure the work that reaches you

The model picker is becoming a workflow picker

GitHub announced Copilot Fusion’s general availability on October 28. It describes a system that selects and orchestrates models, with user-selectable effort levels and organization controls over the available model pool. Its documented execution patterns include a single-model route, escalation through a cascade and critique followed by revision. Billing reflects the models used.

Those release facts matter because the object being evaluated has changed. Choosing a routing system is different from choosing one fixed model. GitHub’s performance comparisons are vendor-reported results, not a promise about your codebase, so this guide does not turn them into expected savings.

Our proposed evaluation begins with the work that reaches a human reviewer. Did the task get completed? Was the change easy to inspect? How much repair did it require? The following is an original pilot design, not a hands-on review of Fusion.

Imagine a small team maintaining a community events app. Its work includes straightforward copy changes, bug fixes and occasional features that cross several files. A router could potentially reduce the need to choose a model manually for every task. The team still needs to decide whether that convenience improves its own work rather than merely changing which name appears below an answer.

Source: GitHub: Route and orchestrate models with GitHub Copilot Fusion ↗ · GitHub Docs: About GitHub Copilot Fusion ↗ · Git: git diff documentation ↗

Give the pilot a job description

Before enabling a routing system as a default, write down why you are considering it. Reducing time spent choosing models is a different goal from improving complex bug fixes. Lowering direct usage charges is another goal, and it can conflict with the first two.

For the fictional events team, suppose the objective is to finish routine maintenance with fewer interruptions while preserving review quality. That objective suggests a pilot containing routine tasks, not just the most difficult issue anyone can remember.

Choose tasks from a recent, ordinary slice of work. Include one text-only change, one narrow bug, one change with a clear test and one task whose requirements need clarification. This is an illustrative mix, not an official Fusion evaluation suite.

Write an acceptance note for every task. “Update the event description” should specify which wording is approved and which files are in scope. “Fix the date bug” should include a reproducible example and the expected result. A router cannot compensate for a pilot that has no shared definition of done.

Use an isolated branch or copy of the project so experimentation does not quietly become production work. Give each trial the same permitted tools and the same starting material. If one route receives a prepared explanation while another has to discover the bug from scratch, record that difference rather than presenting the comparison as equivalent.

Finally, decide who will review the outputs and how much time the pilot is allowed to consume. A small evaluation that answers one decision is more useful than an open-ended tournament whose rules keep changing whenever a favorite approach loses.

Track the whole task, including the human repair

A response can be quick while the task remains unfinished. A proposed fix may require a reviewer to locate an unrelated edit, correct a mistaken assumption or explain the requirements again. Those costs belong in the evaluation.

For each pilot task, record the initial request, the first usable output, the final accepted result and the human interventions in between. You do not need a complicated dashboard. A short note can distinguish “accepted after one review” from “required two clarifications and a manual repair.”

Keep direct tool cost separate from human time. Do not convert a few minutes into a precise dollar saving unless your team has deliberately chosen a defensible valuation. The useful early observation may simply be that one approach reduced paid usage but created more review work.

Also preserve tasks that fail. If a route never reaches an acceptable result within your chosen limit, record that outcome and the time spent. Dropping failures from the comparison would make the remaining successes look more efficient than the workflow actually was.

Use clear timing boundaries. Start when the trial receives the agreed task, and stop when a reviewer accepts the result or ends the attempt. You can record model response time separately if available, but it should not replace the complete task duration.

This is especially relevant when models review other models. A critique step might prevent a mistake, or it might produce commentary a human still has to investigate. Evaluate the final effect on the task. Counting internal steps alone does not tell you whether the review became easier or the change became safer to accept.

Consider invented numbers for a pilot of ten tasks. Suppose one approach uses ten units of paid usage and produces eight accepted results, while another uses eight units and produces five. The second has lower total usage, but it spends 1.6 units per accepted result compared with 1.25 for the first.

These are hypothetical arithmetic examples, not Fusion prices or performance figures. They illustrate why a report should keep accepted work beside consumption. A lower bill can be a poor bargain if too many tasks remain unfinished.

Now add review time. Imagine the first approach needs forty minutes of human repair across its accepted results and the second needs fifty. Keep those minutes visible instead of blending them into an unexplained efficiency score. A team may value predictable review effort more than a small reduction in tool usage.

Look at the unaccepted tasks too. They may be the most important tasks in the batch, or they may be work your team routinely handles manually. The decision cannot be made from a fraction alone. Use the numbers to identify the tradeoff, then explain the practical consequence in words.

This is why the pilot needs task records as well as totals. A denominator helps prevent a misleading comparison, while the individual examples explain whether the difference matters to the people doing the work.

Read the diff as a statement of scope

In the fictional events app, a request to fix a date label might also produce a broad formatting change. The code may still run, but a reviewer now has more material to inspect. That extra scope should appear in the pilot notes.

Git’s diff documentation provides tools for viewing changes between working trees, commits and other references. For this exercise, use a readable diff to compare the proposed change with the task’s acceptance note. Keep the technical command choice consistent with your team’s existing workflow.

Ask three ordinary questions. Does every changed file serve the requested outcome? Is the main behavior easy to find? Can a reviewer explain why the change is sufficient? These are our suggested review questions, not claims that a particular route will perform better.

When an output includes a useful refactor, decide whether to accept it as separate work. A clever improvement outside the request should not automatically earn the original task a better score. The team may value it, but it has changed the amount of work being reviewed.

Record missing scope as well. A fix that changes the visible label but leaves another instance inconsistent may be incomplete. The acceptance note should make that judgment possible without asking the model whether it considers itself finished.

For a non-coding task, use an equivalent comparison. Review the changed paragraphs, configuration entries or spreadsheet formulas. The principle is that the accepted artifact should make its relationship to the original request visible. Model routing happens upstream; the human still needs a clear account of what changed.

Treat effort as a hypothesis

It is tempting to use the highest effort setting for everything and call that the strongest evaluation. That would answer only one question: how that configuration behaves on those tasks. It would not tell you whether the additional effort is worthwhile for ordinary maintenance.

Try a predeclared rule. For example, the fictional team could begin routine tasks at its everyday setting and reserve a stronger setting for a defined class of difficult work. Write the rule before the pilot so you do not selectively rescue one approach whenever a task goes badly.

When escalation occurs, record what triggered it. Was there a failed check, a reviewer concern or simply a feeling that the task deserved a bigger model? That distinction helps the team decide whether the escalation rule can be used consistently by other people.

GitHub’s documentation describes Fusion as choosing models and execution patterns at runtime. It also explains that its pool can change. Treat the selected effort level as one part of the configuration, not as a permanent identity for a fixed underlying model.

For a pilot report, note the client, date, available settings and any model information the interface actually exposes. Do not invent a detailed internal route if the product does not show it. If visibility is limited, say that the result belongs to an observed configuration with incomplete internal detail.

An ordinary user should not need to reverse-engineer the router to benefit from it. Still, a team deciding on a default needs enough information to understand what it evaluated and when. That record makes future changes easier to interpret without pretending that automatic routing is static.

Distinguish agreement from useful review

A multi-model workflow can sound reassuring because more than one system is involved. The relevant question is what the additional review catches and what it misses. Agreement is not itself evidence that a proposed change satisfies the task.

For a harmless rehearsal, create a small seeded mistake in a toy project. The mistake might be an inconsistent label or a calculation with an obvious boundary condition. Define the expected behavior before asking any assistant to work on it.

Observe whether the final result fixes the issue and whether its explanation points to the relevant evidence. Do not infer that an internal critic identified the problem unless the interface actually shows that information. You are evaluating an output, not reading the system’s mind.

Then include a task where no change is needed. A useful assistant should be able to explain that the reported issue is not reproduced under the supplied conditions, or ask for missing information. A pilot containing only real bugs can accidentally reward unnecessary edits.

Keep the rehearsal small and detached from production security claims. Finding a seeded inconsistency in a toy app does not establish that a routing system will catch vulnerabilities or prevent incidents. It simply gives you a readable case for discussing how a review affected the result.

If the final explanation is confident but the check still fails, count the failure. If the explanation is modest and the artifact meets the acceptance criteria, count the accepted result. This keeps evaluation centered on evidence instead of the emotional comfort of a polished review narrative.

Include one task that should stop

A routing pilot should include a request that lacks an essential decision. In the fictional events app, perhaps someone asks to change the cancellation policy without providing approved wording. The task is not ready for a final implementation.

Write the expected behavior: identify the missing policy decision and produce a bounded draft or clarification. The assistant should not quietly invent a rule that changes what attendees are promised. This is a useful case even if the application has no financial transactions or sensitive data.

Observe whether the route stops at the right boundary. A system that produces more code is not necessarily completing more of the legitimate task. Sometimes the best output is a clear explanation of what the human owner must decide.

Compare the cost of that interaction with the baseline workflow. Did routing reduce the effort needed to identify the missing requirement, or did it generate a large change that a reviewer had to reject? Record the answer without generalizing from a single example.

This case also tests the acceptance note. If reviewers disagree about whether the assistant should have implemented a provisional policy, the team needs to clarify its own rules. The pilot has discovered a management decision, not merely a model weakness.

After resolving that ambiguity, add the agreed behavior to your normal task instructions. The value of an evaluation can extend beyond picking a tool: it can reveal which expectations were previously carried only in one teammate’s head.

At the end of the pilot, group results by task type. A routing system might be a useful default for narrow maintenance while needing more review for requirements-heavy changes. Another team might find the opposite. These are possible outcomes, not findings from our testing.

Avoid announcing a universal winner from a handful of tasks. Say what the pilot supports: for this small collection, under this configuration, the team observed these benefits and these problems. Keep the underlying task records available so the conclusion can be challenged.

Choose a review date or a clear trigger for revisiting the decision. A major product change, a new repository or a repeated class of failures could justify another bounded evaluation. Constantly rerunning the same comparison without a reason can consume the attention the tool was supposed to save.

Preserve an easy way to select a different approach for a task that needs it. A default should reduce routine decisions without making exceptions mysterious. The person doing the work should understand when to ask for help or change the configuration.

Then explain the default in practical terms. The fictional team’s note might say that everyday maintenance begins with a selected routing setting, every change still receives a diff review and policy decisions remain with the owner. It should not promise that automatic model selection eliminates human judgment.

The useful shift is from asking which model sounds most powerful to asking which workflow produces acceptable work with a manageable review burden. Copilot Fusion makes that shift explicit. A thoughtful pilot can make it measurable enough to guide a real decision.