Anthropic’s head of Claude Code, Boris Cherny, has a blunt message for enterprises drowning in AI dashboards: stop counting tokens and start counting hours of engineering work avoided. In a series of posts on X on July 16, Cherny argued that usage metrics show activity, not real-world return on investment. The question that matters, he wrote, is whether a task would have required manual engineering effort—and how much time that effort would have cost.
Cherny’s framework arrives as companies grapple with spiraling AI costs and pressure to demonstrate tangible productivity gains. The same week, OpenAI published a post echoing the theme: organizations should evaluate AI based on useful work completed, the cost of a successful task, correctness, and whether spending scales output. Together, the two messages signal a pivot away from “tokenmaxxing” toward measuring actual engineering throughput.
They’re not the only ones talking about it. JPMorgan Chase CEO Jamie Dimon told CNBC that AI costs are surging and require a hard-headed approach. OpenAI’s Sam Altman, speaking at a conference in Sun Valley, called cost reduction and value extraction the top concern for business leaders. For Windows developers and IT operations teams that are already using coding agents, the shift in how success is measured could reshape tool evaluation, budget requests, and daily workflows.
A new yardstick for AI coding tools
Cherny’s proposal is part of a four-step organizational adoption playbook he shared. While the first two steps focus on initial rollout and getting AI into everyday use, the third step is where the rubber meets the road: once employees are working with the tool, companies must measure returns—not just consumption.
“Monitoring usage via a dashboard is a great step, but it shows you activity, not ROI,” Cherny explained. His recommended metric: For any task completed with AI, ask how many engineering hours would have been needed without it, then compare that to the actual time spent. The difference is the avoided cost.
That avoids the trap of treating token counts as a proxy for productivity. A tool that burns a massive context window to generate a single correct PowerShell remeadiation script may be far more cost-effective than a cheaper model that spits out plausible but broken code an engineer must rewrite. Conversely, a “high-usage” team might be spinning its wheels prompting, reviewing, and patching subpar output.
OpenAI’s July 17 guidance concurred. Organizations, it said, should measure the tradeoff between spend and output—specifically, whether each additional dollar of AI expenditure produces more completed work. The company stressed that sheer volume of generated text is not the point; the point is whether tasks get done correctly and efficiently.
Why this matters for Windows pros
For Windows sysadmins, cloud engineers, and internal development teams, AI coding assistants are increasingly used to draft scripts, configure infrastructure, patch .NET applications, and troubleshoot deployment failures. Cherny’s message has direct implications for how these teams defend their tool choices and manage costs.
Consider a real-world scenario: An IT admin uses Claude Code to generate a script that automates user provisioning across a hybrid Active Directory environment. A naive dashboard might report 10,000 tokens consumed and a successful code generation. But what if the script had a subtle permission bug that required two hours of debugging? Or what if it worked perfectly, replacing a manual task that normally takes six hours? Token count alone tells you none of that.
The sensible comparison is end-to-end task cost: AI interaction time plus human review, testing, and remediation versus the effort the task would have required without AI. Only that calculation tells you whether the tool earned its keep.
The same logic applies to developers building Windows desktop apps, services, or internal tools. A coding agent that speeds up writing boilerplate for a COM interop layer may save hours even if its per-query token count is high. An agent that churns out reams of plausible-looking C# that fails unit tests is not saving time; it’s creating work.
For managers, the “hours saved” metric requires a defensible baseline—otherwise it’s vulnerable to overcounting. Teams must compare against similar historical work, documented estimates, and, crucially, confirm that AI-assisted output was actually shipped or put into production. A change that’s generated but never merged saves nothing.
How we got here: from token hype to ROI scrutiny
Token-based measurement emerged naturally because it’s easy: every API call logs consumption, dashboards populate automatically, and finance teams can directly map tokens to spend. As AI coding tools moved from experimental pilots to line-item budget categories, cracks appeared. Enterprises noticed that higher token usage didn’t always correlate with happier, faster teams.
The term “tokenmaxxing” captured the initial gold rush—maximizing token throughput as if it were a standalone good. But a series of public statements this week mark a turning point. Cherny’s X thread, Business Insider’s reporting on it, and OpenAI’s own post collectively argue that the industry must mature beyond counting calls.
Cost pressures are accelerating the shift. Dimon’s CNBC remarks reflect a boardroom sentiment that AI spending cannot grow indefinitely without clear, defensible returns. Altman’s comments signal that even model builders see an urgent need to help customers extract more value from each dollar.
For Windows-focused IT departments, this evolution mirrors earlier conversations about cloud consumption. Years ago, raw compute hours were the metric; today, FinOps disciplines look at cost per transaction, cost per business outcome, and unit economics. AI measurement is following the same arc.
What to do now: building an hours-saved measurement
Translating Cherny’s principle into your own workflow doesn’t require heavy tooling, but it does require rigor. Here’s a practical starting point for Windows teams:
-
Establish a baseline. Before measuring savings, agree on how long the task would have taken without AI. Use prior similar work, historical ticket resolution times, or structured estimates from senior staff. Avoid round-number guesses; anchor in data.
-
Track end-to-end time. When an engineer uses AI to tackle a task, log the total clock time: prompt engineering, waiting for model output, reviewing the result, testing, and any rework. If the task fails, capture that time too—it counts against the AI’s net impact.
-
Verify production delivery. A generated change only saves time if it ships. Require that tasks be marked complete only when code is merged, deployment succeeds, and monitoring stays green. Include time spent on any incident triggered by the AI change.
-
Calculate net savings per task. Net engineering time saved = (manual baseline hours) – (AI-assisted end-to-end hours). Only then aggregate across tasks to get a team-level metric. This naturally rewards tools that produce correct, deployable output quickly.
-
Watch for second-order effects. Some AI tools reduce repeat tickets or support calls. If an AI-generated script fixes a recurrent WMI error that used to generate 20 helpdesk tickets per month, track that reduction as extended savings. Conversely, if AI output introduces new bugs that surface later, account for those too.
-
Pair AI usage data with outcome metrics. Keep token consumption and seat cost as financial inputs, but present them alongside completed tasks, net hours saved, and quality metrics in reviews. This tells a complete story: what was spent and what engineering capacity it freed.
Cherny acknowledged in subsequent posts that the bigger win comes when maintenance and fixes happen in the background, letting engineers focus on creation. For Windows shops, that might mean one platform engineer automating patch compliance with Claude Code while nine others tackle feature work. The hours saved aren’t just the one engineer’s time—they’re the cumulative freed capacity.
The road ahead
Expect AI vendor dashboards to evolve. As Cherny and OpenAI give voice to what enterprise customers are demanding, tools will likely start surfacing outcome-oriented metrics natively—time saved, tasks completed, and quality scores. Until then, the burden is on internal teams to connect the dots.
The underlying message is healthy: AI coding tools are maturing from novelty to utility, and measurement should follow suit. Windows professionals who adopt an hours-saved mindset now will be better positioned to justify spending, select the right tools, and—most importantly—spend their own time on work that actually moves the needle.