An AI usage chart is not a report card on your team

More complete traffic is still traffic

GitHub's October 23 Copilot update adds a client-level view of usage from tools that were previously difficult to capture through editor telemetry. The new totals_by_client data includes successful model requests, sessions, token usage, and model breakdowns in supported reports.

GitHub explicitly distinguishes model calls from prompts: one prompt may cause several calls. It also describes client identification as a best-effort grouping rather than verified identity. These qualifications belong close to any chart built from the data.

Our interpretation is that the update can help organizations understand tool support and resource use, while creating a risk of overreading the numbers. More observed activity does not, on its own, establish more useful work. Less observed activity does not establish that a person contributed less.

This is an ethics and measurement explainer, not a report that we accessed an employer's telemetry or tested a company's evaluation process. The examples below are fictional and use invented numbers to show how interpretations can diverge.

Imagine a small development team deciding which AI integrations to support. A new report shows more requests from one client than another. That is a useful operational observation. The mistake would be to skip directly from that observation to a ranking of the people using the tools.

Before making a chart, write the decision it is supposed to help. Supporting a popular integration, investigating an unexpected usage increase, and evaluating the quality of a software change are different jobs. They need different evidence. A number becomes informative when its meaning matches the question, not merely when it is easy to count.

Source: GitHub: Usage metrics API adds tools such as OpenCode ↗ · GitHub Docs: REST API endpoints for Copilot usage metrics ↗

Write the unit beside the number

For a proposed dashboard, put the unit directly in the label. Model requests, sessions, tokens, prompts, completed tasks, and reviewed changes are not interchangeable. If the source only measures one of them, do not rename it to sound closer to the outcome you wish you had measured.

Consider an invented example. One interaction triggers twelve model calls while another triggers three. Both produce a single draft change. The first may involve a more complex workflow, a different tool design, or repeated attempts. The count alone does not tell you which draft is better.

The same reasoning applies to tokens. A longer context can include useful source material, repeated boilerplate, or irrelevant content. An output can be brief because it is concise or because it did not complete the job. Volume is a clue to investigate, not an automatic quality judgment.

For each field, write a plain-language definition and one tempting interpretation it does not support. For example, model requests describe calls, not the number of human ideas. Sessions describe a grouping of activity, not necessarily a standard-sized unit of effort.

Keep missing or unknown categories visible. GitHub's announcement includes other and unknown cases for some classifications. Do not quietly redistribute them into a preferred category to make a chart appear complete.

A useful review question is whether someone unfamiliar with the API could read the label and understand what was counted. If the answer requires a hidden caveat, improve the label or the accompanying note.

This small habit prevents an ethical problem before it becomes a dispute. People should not have to discover after a decision that the metric's name promised more than its underlying definition could deliver.

Separate tool adoption from work quality

Suppose the fictional team's first question is which integrations need onboarding support. Client-level activity can help identify where people are using the model service. Pair the report with a short conversation about unmet needs rather than treating the most active client as automatically best.

A different question concerns whether an AI tool improves completed work. For that, our proposed assessment would look at appropriate task outcomes: whether a change met its requirements, whether review found important defects, and what effort remained for the team. Those observations still require careful interpretation.

Do not compare two groups with different jobs as if they received the same assignment. One group might explore unfamiliar code while another maintains a stable service. Even a meaningful outcome measure can be misleading without context about the work.

A useful evaluation plan states the intended use of the numbers before collecting them. If the purpose is infrastructure planning, keep the conclusion about infrastructure. If the purpose changes, revisit whether the available evidence supports it.

Also ask what success looks like when the tool is not used. A developer may solve a task quickly from prior knowledge. Another may appropriately avoid sending sensitive context to an external service. A dashboard should not make thoughtful restraint look like failure simply because restraint produces fewer calls.

These are proposed interpretation principles, not findings about a specific employer. They help keep an adoption study from becoming an accidental incentive to generate more activity.

The practical distinction is simple: usage tells you something about a tool's presence in a workflow. Work quality concerns whether the workflow achieved its purpose. The first can inform the second, but it cannot replace it.

Include review effort in the discussion. If an AI-generated change requires extensive correction, a high activity count may coexist with a poor result for the team. If a small suggestion prevents a recurring defect, a low count may accompany useful work. In the fictional exercise, write these as separate cases and ask which evidence would distinguish them. That forces the evaluation to consider the work surrounding the model, including the human review that a traffic chart does not directly describe.

A fair review should survive a change in tool design

Imagine the fictional team changes an integration so that one user request now triggers several internal model calls. Human behavior and completed work remain similar, but the request chart rises. If the organization rewards the rise as improved productivity, its judgment depends on an implementation detail rather than the outcome it cares about.

Now imagine the tool becomes more efficient and uses fewer calls for the same result. A careless dashboard could make that improvement look like declining engagement. This is an invented scenario, but it illustrates why the unit matters.

For a proposed reporting process, record relevant changes to integrations, model selection, telemetry coverage, and report definitions. Keep those notes with the time series. A new data source can change the observed total even when the underlying work changes little.

Do not splice incompatible measurements into a seamless trend without explanation. If the report becomes more complete after an update, mark the boundary. Readers should know whether a jump reflects newly captured activity, changed behavior, or an unresolved combination.

Ask a counterfactual question before using a metric in a consequential decision: if the tool performed the same useful work with half as many requests, would our interpretation improve or worsen? If it worsens, the metric may be rewarding activity that is only indirectly related to the goal.

This does not make request counts useless. They can be valuable for capacity, adoption, and troubleshooting questions. It means the organization should preserve the relationship between the measurement and the decision.

A chart becomes fairer when it can explain changes in its own measuring process. Otherwise the people represented by the chart can be judged for differences they did not cause and cannot reasonably control.

Make access and explanation part of the process

GitHub's documentation restricts usage-report access to authorized roles and permissions, with the relevant policy enabled. The existence of a report endpoint does not make its contents appropriate for every audience or every purpose.

For the fictional team, our proposed practice is to state who can inspect the report, why they need it, and how the results will be discussed. Use the least detailed view that answers the operational question. An aggregate support-planning question may not require a public list of individual activity.

Explain the intended use to the people whose work is represented. Give them a way to correct a factual misunderstanding about the workflow or data. A person may know that an integration changed, a task was unusually complex, or an activity category is missing.

Do not present a small group's metric as an anonymous guarantee if the context makes individuals obvious. This is a general caution for the proposed reporting design, not a legal determination or a claim about GitHub's privacy guarantees.

Also separate observation from accusation. An unexpected spike is a reason to investigate. It is not by itself evidence of misuse. An empty report is not automatically evidence that someone did no work. The documented field definitions and coverage should guide the first questions.

A useful meeting format begins with what the report measures, what it omits, and what decision is under discussion. That gives participants a shared basis for interpretation instead of asking them to defend themselves against a number whose meaning is unclear.

The ethical objective is a process people can understand and challenge. Transparency does not require exposing every record to everyone. It requires a clear account of the evidence and how it is being used.

Use the update to ask a better operational question

The most constructive first use of the new metrics may be modest. Which client needs documentation? Where should the team investigate unusually large contexts? Are people using an integration that the support team did not know about? These questions fit traffic data more naturally than a universal productivity score.

For a proposed exercise, build a fictional report with three clients and a small number of fields. Write two useful questions the data can answer and two conclusions it cannot support. Then ask another reader whether the chart labels communicate that boundary.

Add a hypothetical integration change and see whether your reporting note explains the resulting shift. Add an unknown category and keep it visible. Add a completed task with no model calls and decide how your broader assessment would recognize its value.

You do not need access to an organization's real data to practice this reasoning. Invented examples are sufficient to reveal whether the dashboard's interpretation depends on hidden assumptions.

If you do use the API in an authorized setting, follow the current documentation for report scope and permissions. Preserve the report dates and definitions. Do not substitute a visually similar field from another dashboard without checking whether it measures the same thing.

GitHub's update improves visibility into a wider range of model traffic. The durable responsibility belongs to the people reading it: distinguish a counted event from a valued outcome, and choose an interpretation that the evidence can bear.

A good AI dashboard should help a team make a better decision. It should not pressure people to produce a larger number merely because that number is now available. The right question is what the information helps you understand and improve.