Staff at the UK's Department for Business and Trade (DBT) shaved over an hour off document drafting tasks with Microsoft 365 Copilot, but the department itself saw no measurable productivity improvement—a paradox that emerged from one of the most rigorous public-sector AI pilots to date. The three-month trial, ending December 2024, gave 1,000 employees access to the generative AI assistant and combined telemetry, diary studies, and observed tasks to gauge real-world impact. Its headline finding: 72% of users were satisfied, yet those individual time savings failed to translate into department-level gains during the short evaluation window.
The DBT report, published alongside a wider cross-government experiment by the Government Digital Service (GDS), offers a sober, evidence-based checkpoint for any organization considering Microsoft 365 Copilot. It underscores that likable, time-saving tools do not automatically boost organizational productivity without deliberate workflow redesign, governance, and rigorous measurement that looks beyond self-reported minutes.
Inside the DBT's mixed-method evaluation
The DBT pilot distributed 1,000 Microsoft 365 Copilot licenses from October to December 2024. About 70% of participants volunteered, while the remaining 30% were randomly assigned to improve representativeness. The evaluation avoided relying on a single data source. Instead, it triangulated:
- Telemetry from Microsoft's M365 Copilot dashboard to track active users and app usage patterns.
- A diary study during one week in November, where staff logged tasks, satisfaction, accuracy, and estimated time without Copilot. The study had a 32% response rate.
- Observed tasks and interviews with a smaller sample to validate self-reports and inspect output quality and verification overhead.
Crucially, the team adjusted time-savings calculations. They removed outputs that were never used (drafts abandoned) and subtracted time spent on “novel” work—tasks staff performed only because Copilot made them possible, which might have inflated apparent gains. Statistical checks tested for representativeness and subgroup differences. This conservative approach aimed to surface net effects rather than first impressions, a move the report says is essential because short pilots and self-reporting notoriously overstate impact.
Where Copilot shined—and where it stumbled
User satisfaction was strong. The DBT recorded a Net Promoter Score of 31, which the report deemed “a good outcome for a new digital service.” Satisfaction clustered around text-heavy tasks: drafting, editing, and summarizing. Neurodiverse staff and non-native English speakers reported statistically higher satisfaction, citing improved meeting accessibility and comprehension—a real, if non-productivity, win.
Diary study data, after adjustments, revealed chunky per-task time savings for some activities:
| Task | Mean Time Saved (Hours per Task) |
|---|---|
| Drafting written documents | ~1.3 |
| Summarising research | ~0.8 |
| Transcribing/summarising meetings & searching for information | ~0.7 |
Yet other tasks took longer. Scheduling added an estimated 0.6 hours per task, and image generation also burned more time after unused and novel outputs were accounted for. Observed sessions confirmed that Copilot often accelerated email drafting and meeting note summarization, but Excel data analysis sometimes became slower and less accurate—a direct counterexample to blanket productivity claims.
The productivity gap: why hourly wins didn't add up
Even with these per-task gains, DBT found “no robust evidence” that they translated into department-level productivity improvements during the pilot. Colleagues outside the pilot noticed no visible change in participants' overall output. The evaluation warned that small per-task savings don't automatically compound into organizational gains without changes in workflows, governance, and long-term measurement.
Several factors explain the gap:
- Verification overhead. Hallucinated outputs forced users to spend time fact-checking, eroding net savings. Where outputs fed decisions without review, operational risk rose.
- Novel-task paradox. Copilot made some work so easy that staff performed additional, previously uncompleted tasks—increasing workload rather than reducing it. The DBT's adjustments for this materially shrank apparent time savings.
- Task-specific degradation. Observed Excel analysis sessions were sometimes slower and less accurate, dragging down aggregate gains.
- Short pilot horizon. Three months is insufficient to form new habits, redesign workflows, or measure systemic change.
Why the headlines diverge: the GDS cross-government experiment
Alongside the DBT work, the GDS ran a parallel experiment with roughly 20,000 civil servants across 12 organizations. That study reported an average saving of 26 minutes per user per day. The contrasting numbers—from “no department productivity gain” to “two weeks saved per year”—stem from methodological differences:
- Scale and sample composition. The GDS aggregated broad survey data across many departments and roles, smoothing variance. DBT focused on a single department with role-specific analysis.
- Metric choice. GDS emphasized daily minutes saved from self-report surveys; DBT used adjusted per-task hours from diaries and insisted on observed validation. Self-report typically overstates gains.
- Task mix. Copilot's strongest wins lie in templated writing, summarization, and meeting notes. Organizations heavy in data analysis or specialized workflows may see smaller or negative net effects.
Media summaries that cherry-pick a single figure—like “26 minutes a day”—simplify a complex evidence base. Both official documents must be read to understand the boundaries of each claim.
Industry claims and real-world context
Microsoft CEO Satya Nadella has said that up to 20–30% of Microsoft's code is now written by AI in some projects, a claim reported by CNBC and the Financial Times. Such figures reflect engineering contexts with highly structured tasks, not general office work. The DBT pilot shows that claims of universal productivity uplifts should be treated with caution: context, measurement method, and task type are everything.
Risks, hidden costs, and governance gaps
Beyond the productivity debate, the DBT report flags concrete risks for any organization deploying Copilot at scale:
- Hallucinations and verification drain. Confidently wrong outputs force human checking, and the effort can erase time savings.
- Excel and data analysis pitfalls. Observed sessions showed Copilot can slow or degrade some data-heavy tasks.
- Environmental blind spots. Pilot participants raised concerns about the energy footprint of large language models. DBT noted the lack of quantified environmental impact assessment as a procurement risk.
- Vendor transparency and contracts. Both DBT and GDS recommended contractual clarity on data use, model training, retention, and environmental disclosures before large-scale procurement.
Practical guidance for IT leaders and Windows administrators
The DBT and GDS evaluations converge on actionable advice for public-sector IT and any organization considering Copilot-style deployments:
- Pilot deliberately and restrictively. Start with high-volume, low-risk tasks like meeting notes, templated emails, and document summarization.
- Measure the right things. Combine telemetry with timed observed tasks, not surveys alone. Convert minutes saved into financial terms for targeted pay bands, and factor in training and verification time when modeling ROI.
- Invest in training and human-in-the-loop workflows. Hands-on prompt engineering and verification training matter; DBT found self-led training correlated with higher satisfaction.
- Insist on vendor transparency and contractual safeguards. Demand explicit clauses about whether tenant data may be used to improve vendor models, along with data residency, retention, DLP guarantees, and environmental disclosures.
- Segment rollouts by role and workflow. Deploy where verification overhead is low and benefits are concentrated; defer or restrict use in sensitive data-analysis roles until behavior is validated.
What remains unproven and needs longer measurement
DBT's cautious conclusion is not a rejection of Copilot's potential. It is a call for stronger, longer, and more narrowly designed measurement. The following require follow-up before organizations treat Copilot as a proven productivity multiplier:
- Longitudinal behaviour change: Habit formation, prompt literacy, and deeper UX adaptation occur over many months, not three.
- Organisation-level ROI modelling that converts per-task minutes into net financial outcomes after verification, training, and remediation costs.
- Quantified environmental impact and life-cycle assessments for large-scale LLM usage.
- Independent third-party audits of hallucination rates and failure modes in mission-critical workloads.
When managers or vendors tout single headline numbers, procurement teams should ask: “Which metric, which sample, and what adjustment method produced this number?” Different answers imply very different expected outcomes.
The verdict: measured optimism, not hype
The DBT pilot offers a pragmatic reality check. Copilot is useful and, for many people, satisfying. It produces measurable time savings on specific tasks. But those savings do not automatically add up to department-level productivity gains without follow-through on governance, measurement, and workflow redesign. For IT leaders and M365 administrators, the immediate imperative is operational discipline: pilot with purpose, measure conservatively, invest in verification and training, and require vendor transparency. When those levers are pulled correctly, the per-task wins can compound into real capacity improvements. When they are not, outputs that feel faster can generate hidden work and risk.
The debate now is not whether Copilot can help people—it can—but exactly how organizations will convert those human-level wins into durable, accountable operational value. The DBT evaluation stands as an evidence-based checkpoint on that path, one that any enterprise evaluating Copilot would do well to study.