The Copilot Audit: A Thought Experiment for SMBs

Jim Smith AI Strategy

In my SMB AI guide, I called company-wide Copilot rollouts “the safest way to waste $36,000.” That line got more responses than anything else I’ve written.

The pushback was predictable: “But our team loves it.” “Productivity is up.” “Everyone uses it.”

Let’s run a thought experiment. Imagine you could see exactly how your Copilot licenses are actually being used. What would you find?

What the Research Actually Shows

Before we dive into the thought experiment, let’s look at what the data tells us.

A July 2025 study by the National Bureau of Economic Research and OpenAI analyzed over a million ChatGPT conversations. The findings are revealing: 70% of usage is non-work related. Of work-related use, writing is the most common task—but two-thirds of those writing requests are edits, not new content. Programming? Just 4.2% of all messages.

Meanwhile, Microsoft 365 Copilot adoption has been described as “diabolically bad”. As of August 2025, only 8 million of Microsoft’s 440 million M365 subscribers have active Copilot licenses—a 1.81% conversion rate. Only 1% of organizations have completed a full Copilot deployment. A Gartner survey found just 3% of respondents said M365 Copilot provides “significant” value.

Tim Crawford, a former IT executive who advises CIOs, put it bluntly: “Am I getting $30 of value per user per month out of it? The short answer is no.”

This isn’t speculation. It’s pattern recognition. And it’s why the following thought experiment probably looks a lot like your company.

The Thought Experiment: Acme Anvil Distribution

Picture a typical SMB: Acme Anvil Distribution, 96 employees across 19 departments. Six months ago, they rolled out Microsoft 365 Copilot company-wide at $30/user/month. “Everyone gets it.” Annual cost: $34,560.

(Side note: Microsoft dropped the price to $21/user/month for SMBs in December 2025—a tacit admission that adoption has been disappointing. But even at the lower price, the fundamental problem remains: most licenses go unused.)

Now imagine the CFO asks a simple question: “Is this working?”

Here’s what an honest audit would probably reveal—and I’d bet good money your company looks similar.

The Audit Framework

What to Measure

Adoption Rate: What percentage of licensed users actually activated Copilot? This is different from “has a license.” Many employees never click the button.

Active Usage: Of those who activated, how many used it in the last 30 days? This separates the curious from the committed.

Feature Distribution: Which Copilot features actually get used? Email drafts? Excel formulas? PowerPoint generation? Teams meeting summaries? The answer matters for ROI.

Time Saved: Self-reported estimates are almost always inflated. “I save 30 minutes a day” usually means “I used it once and it was fast.”

Output Quality: Are people using AI drafts as-is, or editing them heavily? Heavy editing suggests the time savings are illusory.

How to Get This Data

Microsoft 365 Admin Center: Navigate to Reports → Usage → Microsoft 365 Copilot. You’ll see adoption rates, active users, and feature breakdown by app.

Simple User Survey: “How many times did you use Copilot this week?” Five questions, anonymous, takes two minutes. You’ll be surprised how honest people are when it’s anonymous.

Department Interviews: Talk to managers. Ask: “If I took away Copilot tomorrow, what would break?” The answer is usually “nothing.”

The Findings: What Would Acme Anvil Discover?

Department Licenses Active Users (30 days) Primary Use Case ROI Assessment
Finance 6 5 Financial modeling, forecasting High
Accounting 5 4 Excel reconciliation, reports High
Sales 15 4 Email drafts Low
Marketing 10 6 Content drafts, presentations Medium
Product Management 4 3 Specs, roadmap docs Medium
Operations 10 2 Meeting summaries Very Low
HR 6 2 Policy drafts Low
Legal/Compliance 1 1 Contract review, policy language Low
IT - Desktop Support 3 1 Ticket responses, documentation Low
IT - System Admin 1 1 Script drafts, runbooks Medium
IT - Networking 1 1 Documentation Low
IT - SW Developer 1 1 Code snippets High
Customer Service 10 3 Response templates, FAQ drafts Low
Admin/Support 8 3 Email, basic documents Low
Procurement 4 1 RFP drafts, vendor correspondence Low
Executive 5 5 Meeting summaries, email Medium
Reception 2 0 — None
Janitorial 2 0 — None
Building Maintenance 2 0 — None
Total 96 43    

45% active usage. And that’s being generous—”active” means used at least once in 30 days, not daily. Annual cost: $34,560.

Look at the bottom three rows. Reception, janitorial, and building maintenance—6 licenses, zero active users. These employees don’t sit at desks writing emails. They’re not creating PowerPoints. But when you roll out “company-wide,” they get licenses too. That’s $2,160/year in licenses that will never be used.

And it’s not just the obvious cases. In this scenario, Customer Service has 10 licenses and 3 active users. Procurement has 4 licenses and 1 user who touches it occasionally. Operations bought 10 licenses for meeting summaries that nobody reads. Does this sound familiar?

Patterns You’ll Probably See

The 20-40% Rule: Across SMB deployments, roughly 20-40% of licenses typically see regular use. Acme’s 45% is actually on the high end. Many companies are worse. Remember: only 1% of organizations have completed a full Copilot deployment—most are still in “pilot” mode years after launch.

Finance Always Wins: Excel integration with Copilot is genuinely useful. Complex formulas, data analysis, pivot table creation—this is the one area where ROI is consistently positive. Acme’s finance team? Worth every penny.

IT Is a Mixed Bag: Look at those four IT rows. Desktop support and networking barely touch Copilot—they’re troubleshooting hardware and configuring switches, not writing documents. System admins get some value for scripts and runbooks. Developers use it, but they’d be better served by purpose-built coding tools. M365 Copilot is a word processor with AI bolted on; IT needs something that understands code.

Meeting Summaries Are Oversold: Everyone thinks they want Teams meeting summaries. Almost no one changes their behavior because of them. Operations has 10 licenses and 2 active users. Those summaries aren’t being read. This tracks with Gartner’s finding that only 3% of respondents said M365 Copilot provides “significant” value.

Email Drafts Get Edited to Death: Sales has 15 licenses, 4 active users. The ones who do use it spend so much time editing the AI output that they’re not saving time. They’ve just shifted the work from writing to editing. The NBER study confirms this pattern: two-thirds of AI writing requests are edits or modifications, not new content generation.

Legal Needs Context, Not a Better Model: Acme has one in-house counsel using Copilot for contract review. She’s using it—but she shouldn’t be. For legal work, the model matters far less than the context. A generic AI drafting contract language is dangerous; it doesn’t know that Smith v. Jones was overturned last month or that your state just passed new data privacy regulations. Legal AI needs access to current case law and statutes, updated daily. That’s not what Copilot provides. A specialized legal AI like OpenCase ($79/month), which integrates directly with Cornell LII’s U.S. Law Database and ingests federal and state court opinions daily, is both cheaper and dramatically more useful than a $21 Copilot license that hallucinates precedents.

The Hidden Costs

The $36,000 line item is just the beginning.

Training Time: How many hours did your team spend in “Introduction to Copilot” sessions, watching Microsoft tutorials, and experimenting with prompts? At Acme, rough estimate: 200 hours across the company. At an average loaded cost of $50/hour, that’s $10,000 in productivity.

Prompt Iteration: The first Copilot output is rarely usable. Users iterate. “Make it shorter.” “More professional.” “Actually, go back to the first version.” Each iteration takes time. Nobody tracks this.

Over-Reliance Risk: Junior employees using AI-drafted emails without fully understanding the content. At Acme, one sales rep sent a proposal with a pricing error because they trusted Copilot’s Excel formula without checking. Cost: one lost deal, roughly $15,000.

Security Review: IT spent 40 hours reviewing Copilot’s data handling, updating policies, fielding employee questions about what’s safe to paste in. That’s not free.

The “Good Enough” Trap: Mediocre AI output that technically works but isn’t good. Press releases that sound generic. Emails that are correct but lack personality. Presentations that hit all the points but inspire no one. Hard to quantify, but it’s real.

The Right-Sizing Framework

Here’s what Acme should have done—and what you can still do.

The 6-License Approach

Instead of 100 licenses at $36,000/year, start with 6:

  1. Identify actual power users: Not who says they want it. Who will actually use it daily for tasks that matter?

  2. Match tool to task:
    • Excel-heavy work? M365 Copilot is the right choice
    • Coding? GitHub Copilot ($19/user/month) is better
    • Writing and research? Claude Pro ($20/month) or ChatGPT Plus ($20/month) often outperforms M365 Copilot for pure text work
    • Sensitive data? Consider local models with no cloud dependency

  3. Set expectations and measure: “I expect you to use this at least 3 times per day. We’ll check in after 30 days.”

  4. Quarterly review: Expand only when you have documented ROI. Not “feels faster.” Actual metrics.

When to Scale

Scale your licenses when:

  • Power users are demonstrably more productive (measured, not guessed)
  • Other employees are asking for access based on peer success
  • You’ve identified specific workflows that clearly benefit

Don’t scale because:

  • “Everyone else has it”
  • Microsoft’s sales team is persuasive
  • You need something to report to the board

The Alternatives

Before renewing that company-wide contract, consider matching tools to roles:

Claude Code ($20/month for Claude Pro, free with Max): For IT—especially system admins and developers—this is transformative. It reads your codebase, writes scripts, automates infrastructure tasks, and actually understands what it’s looking at. A sysadmin with Claude Code becomes a team of five. M365 Copilot can’t touch this for technical work.

GitHub Copilot ($19/user/month): Purpose-built for developers. Better than M365 Copilot for pure coding. If your software developers are using M365 Copilot for code snippets, they’re using the wrong tool.

Google Gemini ($20/user/month): For marketing teams, Gemini’s integration with Google Workspace and its longer context window make it surprisingly capable for content work. Campaign briefs, SEO content, research synthesis—worth testing against M365 Copilot for your marketing workflows.

Claude Pro/Team ($20-25/user/month): For writing, research, and analysis tasks, Claude often outperforms Copilot. Better at nuance, less corporate-speak.

ChatGPT Team ($25/user/month): Similar strengths to Claude. Some users prefer the interface. Good for general-purpose AI assistance.

Local Models: For sensitive data or specific workflows, local deployment (via Ollama, LM Studio, or similar) gives you full control with zero data exposure. Higher setup cost, lower ongoing risk.

OpenCase ($79/month): For legal work, context beats model quality every time. OpenCase integrates with Cornell LII’s U.S. Law Database, ingests federal and state court opinions daily, and works directly inside Microsoft Word. Your in-house counsel gets AI that actually knows the law—not one that confidently cites cases that don’t exist.

Nothing: The underrated option. Some tasks don’t need AI. Some teams work better without it. If your Operations department has 2 active users out of 10, maybe those 10 licenses should just go away.

The Five-Question Audit

Run this audit on your own deployment:

  1. What percentage of licenses were active in the last 30 days? (Check Microsoft 365 Admin Center)

  2. Which departments have the highest and lowest usage rates? (The gap will be larger than you expect)

  3. What’s the primary use case in each department? (If the answer is “email drafts,” you’re probably not getting ROI)

  4. What would you lose if you cut 50% of licenses tomorrow? (If the honest answer is “not much,” you have your answer)

  5. Are your heaviest users in roles where AI assistance provides measurable value? (Finance and engineering: yes. Admin and operations: probably not)

What Should Acme Do Next?

If this thought experiment played out, here’s how a rational CFO would respond:

  • Finance + Accounting: Keep all 11 licenses. Clear ROI on Excel-heavy work.
  • IT - SW Developer: Switch to GitHub Copilot. Purpose-built for code.
  • IT - System Admin: Switch to Claude Code. Scripts, automation, infrastructure—this is where Claude Code shines.
  • IT - Desktop Support & Networking: Cancel. They’re not using it.
  • Legal/Compliance: Switch to OpenCase. Context-aware legal AI beats generic Copilot for contract work.
  • Product Management: Keep 3 licenses. Spec writing benefits.
  • Marketing: Switch 4 power users to Google Gemini. Better for content workflows with Google Workspace.
  • Executives: Keep 5 licenses. They claim to use them.
  • Sales, Customer Service, HR, Operations, Admin, Procurement: Cancel. These teams weren’t using it.
  • Reception, Janitorial, Building Maintenance: Cancel. Should never have had licenses.

New setup (at current December 2025 pricing):

  • 19 M365 Copilot licenses @ $21/month: $4,788/year
  • 1 GitHub Copilot license @ $19/month: $228/year
  • 1 Claude Pro license (for Claude Code) @ $20/month: $240/year
  • 4 Google Gemini licenses @ $20/month: $960/year
  • 1 OpenCase license @ $79/month: $948/year
  • Total: $7,164/year

Previous cost: $34,560/year

Savings: $27,396/year

And productivity in this scenario? The teams that kept licenses would use them more effectively because they’re the right people with the right tools. IT would actually be more productive—Claude Code for the sysadmins would automate work that M365 Copilot couldn’t touch. Marketing would have a tool that fits their Google-centric workflow. Legal gets AI that actually knows current case law instead of hallucinating citations. The teams that lost licenses wouldn’t notice.

This is a thought experiment. But I’d encourage you to run the real audit. The numbers at your company might be better than Acme’s—or they might be worse. Either way, you should know.

Sources

Need help running this audit for your company? Get in touch. We’ll tell you what you actually need—which is usually a lot less than what you’re paying for.

Tags: