How to tell whether the automation actually worked
Unmeasured automation is indistinguishable from theatre. Take the baseline before you build, and know the difference between activity and outcome metrics.
Unmeasured automation is just theatre
Something can run every single day, look impressive in a demo, and still be doing nothing useful for the business. Without measurement, there’s no way to tell the difference between an automation that’s genuinely working and one that’s just busy. This is the step almost everyone skips — not because it’s hard, but because it isn’t exciting, and by the time anyone thinks of it, the “before” has already vanished.
Take the baseline before you build anything
This is the part that can’t be fixed after the fact. Once a new process is running, the old numbers are gone — nobody kept a record of how slow the old way was, because the old way was just how things were done. If you want to know whether an automation helped, you need the “before” picture written down before a single thing changes.
The good news is that a baseline doesn’t need to be sophisticated. It’s usually a handful of simple counts that someone could pull together in an afternoon with a notebook and a bit of patience:
- How long does it take, on average, from an enquiry landing to the first reply going out?
- How many quotes went out later than they should have, over a typical month?
- How many invoices are currently sitting older than thirty days?
- How many hours a week does someone spend on this task, roughly?
None of these need a spreadsheet wizard. They need someone to actually look, count, and write the number down before the automation goes live.
Activity metrics versus outcome metrics
Once something is built, it’s tempting to measure how busy it is. “It ran four hundred times this month” sounds like proof of value. It isn’t — it’s an activity metric, and activity metrics only tell you the thing switched on, not that it did anything worth doing.
An outcome metric asks a harder, more honest question: did the result actually change? Did enquiries get replied to faster? Did more quotes turn into booked work? Did the overdue invoice pile actually shrink? This is the number that matters, and it’s the number that takes more effort to track, which is exactly why it gets skipped.
Suppliers love activity metrics because they’re easy to produce and hard to argue with. “It processed everything you sent it” is true and also tells you almost nothing about whether your business is better off. Ask for the outcome number instead, every time.
The honest failure signals
An automation doesn’t usually fail with a dramatic crash. It fails quietly, and the signs are worth watching for on purpose rather than discovering by accident:
- People start quietly working around it — doing the task the old way on the side because the new way doesn’t quite fit reality.
- The rate of manual corrections creeps up over time, rather than settling down as everyone gets used to it.
- The thing gets switched off, or quietly stops running, and nobody notices for weeks. If nobody notices, that tells you something important about how essential it actually was.
Any one of these is worth a conversation. All three together mean the automation solved a problem nobody actually had, or solved the right problem in the wrong way.
Did the saved time go anywhere?
This is the hardest question, and the one owners are least likely to ask themselves. Say the automation genuinely does save someone two hours a week on chasing invoices. Where did those two hours go? Did they get spent on something that actually matters to the business — following up warm leads, doing the work that pays, going home on time? Or did the same two hours simply get swallowed by more of the same kind of busywork, so that nothing about the working week actually feels different?
Saved time that doesn’t get redirected on purpose tends to just refill with whatever’s sitting nearby. That’s not a failure of the automation — it’s a failure to decide, in advance, what the time was for. Worth deciding before you build, not after.
A sensible review rhythm
Don’t wait a year to find out if something worked, and don’t judge it after three days either — most new processes look clunky in the first week simply because people are still learning them. A more sensible rhythm:
- Two weeks in — a quick check. Is it running at all? Are people actually using it, or working around it? Any obvious exceptions it can’t handle that weren’t in the original plan?
- Three months in — the real review. Pull the same numbers you took as a baseline and compare them honestly. Better, worse, or no different? If it’s no different, that’s a useful and slightly uncomfortable answer, and it’s far better to know it at three months than to assume it’s working forever.
What to track: a simple before/after structure
You don’t need anything elaborate for this. A short table with the numbers you already gathered as your baseline, revisited at the three-month mark, is enough to have an honest conversation:
| Measure | Before | After three months |
|---|---|---|
| Time from enquiry to first reply | ||
| Quotes sent later than they should be | ||
| Invoices older than thirty days | ||
| Hours a week spent on the task |
Fill in the “before” column now, whatever “now” is for you, even if you’re only just starting to think about automating something. That single act — writing down the boring numbers before anything changes — will do more for your ability to judge whether a project worked than almost anything a supplier can promise you upfront.
The point of all this
Automation that can’t show its working is just a story someone is telling you about efficiency. Measurement is what turns that story into something you can actually trust — or, just as usefully, something you can catch early and fix. Either outcome beats not knowing.