The UK Department for Business and Trade (DBT) has published the results of a three-month pilot of Microsoft 365 Copilot, and the findings are a masterclass in cautious optimism. 72% of users said they were satisfied or very satisfied, and neurodiverse staff reported meaningful accessibility improvements. Yet, when the department drilled into actual productivity metrics, the numbers were far more sobering: time savings were small, concentrated in text-based tasks, and frequently offset by inaccuracies that demanded rework. The pilot, which ran from October to December 2024 across 1,000 licences, stands as one of the most rigorously measured public-sector evaluations of generative AI to date—and it should compel Windows admins and IT leaders to rethink how they measure value in the age of Copilot.
Inside DBT’s Methodology: Diaries, Telemetry, and Observed Tasks
DBT did not rely on vibes or vendor promises. The evaluation mixed three data streams: telemetry from Microsoft’s official Copilot dashboard, detailed diary logs from participants (32% response rate), and a series of observed, timed tasks to ground-truth the self-reported claims. Standard statistical tests—Chi-squared and Mann-Whitney U—were applied to check for representativeness and significant subgroup differences. Crucially, the team adjusted time-saving calculations to exclude “novel” tasks: work that users only attempted because the tool made it possible, and outputs that were never reused. This conservative approach immediately set DBT’s methodology apart from broader, survey-heavy studies.
Yet the pilot was not without limitations. The short three-month window, partially disrupted by the year-end holidays, meant long-term behaviour change couldn’t be assessed. And while observed tasks tempered self-reporting bias, the diary logs still captured perceived rather than clocked efficiencies. DBT acknowledged that time-saving claims must be taken with a grain of salt.
Satisfaction High, But Only in Certain Corners
The headline user satisfaction figure of 72%—and a Net Promoter Score of 31, which DBT labelled “good”—was driven almost entirely by document drafting, editing, and email summarisation. Users who spent their days in Word, Outlook, and Teams found Copilot a handy, time-saving companion. Outside that comfort zone, the picture turned patchy. Scheduling meetings and generating images with Copilot actually took longer, either because outputs were too low quality to use or because the tool encouraged experimentation with tasks users wouldn’t have otherwise attempted.
A standout finding was the inclusion dividend. Neurodiverse staff and non-native English speakers reported disproportionately higher satisfaction, praising Copilot’s ability to reduce friction, improve clarity, and help them express ideas more confidently. This equity dimension, DBT noted, is often overlooked in standard return-on-investment analyses but matters immensely for workforce accessibility.
Time Savings: Real, Modest, and Use-Case Dependent
When DBT looked at the stopwatch, the gains were measurable but hardly transformative. Drafting, rewriting, and summarising documents produced the largest time savings—yet even these were “small” in magnitude and failed to translate into noticeable department-wide productivity improvements. For data-heavy tasks, the story reversed. In Excel analytics, Copilot-assisted work scored lower on average accuracy than manual methods. PowerPoint slides generated with the assistant were significantly less accurate in observed trials. The one bright spot: manually summarised reports that used Copilot as a drafting aid came out more accurate and were completed faster than entirely manual equivalents. This suggests Copilot works best as a junior co-pilot rather than an autonomous pilot, demanding human judgment at every turn.
The contrast with the UK Government Digital Service’s (GDS) cross-government experiment is instructive. That trial, involving roughly 20,000 civil servants across multiple departments, reported an average 26 minutes saved per user per day. DBT’s more conservative, task-level metrics expose how much methodology matters: broad self-reported survey aggregates tend to inflate savings, while ground-truthed observation yields a clearer—if less spectacular—picture.
The Hallucination Elephant in the Room
Generative AI’s tendency to confabulate did not skip DBT’s trial. Participants regularly saw “hallucinations”—confident, completely fabricated statements—in Copilot outputs. In a government department where decisions have real-world consequences, this was a serious flag. The risk multiplies when users forward AI-generated text without verification, potentially embedding errors deeper into processes. DBT’s report rightly insists that any deployment of Copilot in regulated or high-stakes settings must pair with mandatory human review, acceptable use policies, and robust audit trails.
Environmental Anxiety Surfaces
One of the more striking disclosures was employee pushback on environmental grounds. Interviewed participants voiced concerns about the carbon footprint of large language models, with some saying they were less willing to use the tool because of it. DBT did not conduct a life-cycle assessment or estimate emissions, but it flagged the gap as urgent: any large-scale adoption should be preceded by quantified environmental impact analyses and vendor transparency on data-centre energy use. For Windows admins managing corporate sustainability targets, this is a conversation that can no longer be deferred.
Transatlantic Context: US Takes a Different Bet
While the UK piloted cautiously, the US federal government chose speed and scale. In September 2025, the General Services Administration (GSA) announced a OneGov agreement with Microsoft, making M365 Copilot available at no cost for up to 12 months for eligible G5 customers. It projected roughly $3.1 billion in first-year savings through blended cloud and productivity discounts. The contrasting approaches—one a rigorous, bottom-up pilot; the other a top-down, incentive-laden procurement—will offer a natural experiment in AI governance. Critics warn that large discount deals can create vendor lock-in and suppress competitive options, precisely the sort of dynamic that DBT’s procurement teams are trained to resist.
Critical Analysis: What DBT Got Right—and Where the Risks Still Lurk
DBT’s mixed-method design and its adjustment for novel, unused outputs set a new standard for how to evaluate AI copilots. Instead of asking “How much time did you save?” and stopping there, the team asked whether the output was actually usable, whether the task would have been done otherwise, and whether the quality held up. This is the kind of evidence-first piloting that IT leaders should emulate.
On the risk side, three areas demand immediate governance:
- Hallucination guardrails: Outputs used for decision-making, finance, legal, or policy must be flagged for mandatory human review. Without that, errors will seep into official documents.
- Data creep: Embedded AI assistants can tempt users to surface sensitive or personal data. Tenant-level controls, data-loss prevention policies, and read-only connectors are not optional.
- Vendor lock-in: Aggressive discounts and multi-year deals can create switching costs that stifle future flexibility. Procurement should insist on data portability, audit logging, and exit clauses.
DBT’s approach also exposes the illusion of simplistic ROI. Self-reported time savings often ignore verification and correction overheads. By adjusting for unused outputs, DBT produced a net value estimate that, while unglamorous, is more actionable.
Recommendations for Windows-Focused Organisations
Based on DBT’s experience, Windows admins and enterprise IT planners can take five concrete steps:
- Pilot with surgical precision: Don’t shower licences across the organisation. Target administrative, communications, and policy-drafting teams where text-heavy workflows promise the greatest upside. Avoid data-intensive analytical teams until the model’s accuracy on domain data is validated.
- Measure like a scientist: Use a combination of telemetry, diaries, and observed tasks. Adjust for outputs that get discarded or require major rework. Track not just time saved but time reallocated to higher-value work.
- Build governance from day one: Draft acceptable use policies that explicitly forbid using unverified AI outputs in high-risk contexts. Enforce through tenant controls and periodic audits.
- Invest in prompting skills and verification habits: DBT found that self-led training correlated with higher satisfaction. Hands-on, role-specific guidance makes the difference between a productivity booster and a time-wasting toy.
- Demand environmental transparency: Include energy and carbon reporting requirements in vendor contracts. The absence of such data in DBT’s own pilot should be a wake-up call.
What Comes Next
The DBT evaluation is a snapshot from late 2024, and the Copilot platform is evolving. Upcoming features like Copilot Agents and deeper Excel/Power BI integration may address some of the accuracy gaps flagged in the pilot. Public-sector procurement watchdogs will likely scrutinise large government-wide deals, such as the US GSA agreement, for their long-term competitive impact. And the DBT’s call for quantified environmental assessments will likely catalyse new reporting frameworks for AI workloads.
For Windows-focused enterprises, DBT’s report is more than a policy document—it’s a blueprint. The lesson isn’t that Copilot is doomed or that it’s a silver bullet. It’s that disciplined measurement, tight governance, and role-aware deployment can convert AI hype into genuine, if modest, value. Anything less risks turning incremental time savings into hidden costs—and giving Copilot a black eye it may not deserve.