Step-by-Step Playbook: Audit a GitHub Copilot App Rollout After July's Metrics Updates

GitHub's Copilot app moved from interesting demo to something engineering teams can actually audit in late July 2026. Three changes matter together: on July 17 GitHub added Copilot app activity to the usage metrics API, on July 27 it split the app into its own admin policy and extended managed-settings.json guardrails to the app, and on July 28 it pushed Copilot app activity down into user, feature, model, and language rollups.

That combination changes the rollout playbook. Before July 17, you could see app activity only as a coarse total. Before July 27, app access was tied to the CLI policy. Before July 28, you still could not reliably answer which teams were using the app, how often, or what kinds of code activity it was generating.

If you run an engineering org and want a clean Sunday task for next week, use this six-step audit loop.

1. Separate access from excitement

Start with policy, not prompts. As of July 27, 2026, the Copilot app has its own policy under AI Controls, with three states: Enabled everywhere, Disabled everywhere, and Let organizations decide. That means you no longer need to piggyback on the Copilot CLI policy just to let a small group test the desktop app.

The lazy rollout is a pilot org, not an enterprise-wide flip. Pick one or two teams with active pull-request flow, enable the app there, and leave the rest alone until the metrics say the app is helping.

2. Push the same guardrails into every client

On the same day, GitHub extended enterprise managed settings to the Copilot app and Copilot cloud agent. If you already manage VS Code or Copilot CLI with a .github-private repo, keep using the same copilot/managed-settings.json file. The point is consistency, not novelty.

Use that file to lock the basics:

which plugins are allowed
which plugin marketplaces are allowed
whether approval prompts can be bypassed
whether new conversations default to auto model selection

This is where many app rollouts fail. Teams measure adoption before they normalize policy, then end up comparing one locked-down IDE surface against one loosely governed desktop agent surface. That is a bad comparison.

3. Pull the 28-day report, not just the 1-day report

GitHub's docs now expose Copilot app metrics in the same reporting family as the rest of Copilot usage. For rollout decisions, the 28-day window is the right default because it smooths out launch-day curiosity and catches repeat behavior.

The fields to watch first are simple:

used_copilot_app
totals_by_copilot_app.session_count
totals_by_copilot_app.request_count
totals_by_copilot_app.prompt_count
totals_by_copilot_app.token_usage.avg_tokens_per_request

A quick NDJSON spot check is enough:

bash

jq 'select(.used_copilot_app == true) | {user_login, used_copilot_app, totals_by_copilot_app}' copilot-users-28-day.ndjson

You do not need a custom dashboard on day one. A filtered export already tells you who came back, who only tried it once, and whose sessions turned into high-request, high-token loops.

4. Stop treating the app as a black box

The July 28, 2026 update is the real unlock. Copilot app activity now appears in totals_by_feature, totals_by_model_feature, totals_by_language_feature, and totals_by_language_model under the copilot_app feature value. Top-level code activity, lines added, lines deleted, and daily_active_users also now include app-only users.

That means you can finally ask useful rollout questions:

Are developers using the app mostly for one language or repo type?
Are they burning tokens on long agent loops without corresponding code acceptance?
Is the app mostly complementing IDE chat, or replacing it for a small set of heavy users?

If the app shows high request volume but weak code activity, tighten prompts, shrink task scope, or move some work back into the IDE. If it shows concentrated wins in one language or team, expand there first instead of forcing a broad rollout.

5. Compare adoption to output, not to hype

GitHub's impact dashboard and usage docs now tie adoption cohorts to pull-request output. The dashboard tracks cohort distribution, an adoption multiplier, and recommendations; the API also now includes review velocity fields such as avg_pull_requests_minutes_to_review and avg_pull_requests_review_cycles by adoption phase.

This is the test that matters: after you enable the Copilot app for a pilot group, do their engaged cohorts merge faster or with fewer review cycles than passive users? If the answer is no after a few weeks, you do not have a rollout problem. You have a workflow-fit problem.

6. Keep the success bar boring

A good rollout memo for August 2026 should be short:

policy enabled for a narrow pilot
managed-settings.json applied consistently
returning app users identified from 28-day reports
code activity compared against token usage
adoption cohorts checked against PR velocity
expansion approved only where the data is positive

That is enough. You do not need a giant internal scorecard yet. GitHub's late-July changes made the Copilot app measurable. Your job is to use those new metrics to decide where the app belongs, and just as importantly, where it does not.