Measuring Automation Gains: What 90% Faster Leaves Out
Vendors measure the generation step alone. Add preparation, verification and rework and the honest gain lands at 30-50% — with a payback you can calculate.
In This Article
The real gain of an automation cannot be expressed as a single percentage. Honest measurement compares four items at once: preparation, generation, verification and rework. The "90% faster" figure vendors publish almost always measures only the second — the generation step. Measured across all four, a typical real gain lands between 30% and 50%; still an excellent result, but expectations should be set there.
Automation tool pages always carry a percentage: 70%, 85%, sometimes 90%. Those numbers are not invented — but what they measure is almost never stated. This post is about how to read them and how to set up your own measurement.
When a vendor says "drawing time dropped 90%," what was usually measured is: the time for a single operation, under ideal conditions, to reach a first output.
What was not measured:
- Time to prepare the input (collecting parameters, cleaning the model)
- Time to verify the output
- Time to fix what verification caught
- The automation's own maintenance time
Add those four and the 90% often falls into the 30–50% band.
An automation's return is bounded by its slowest link. Speed the generation step up tenfold and leave verification untouched, and the total gain is whatever verification allows.
Amdahl's law in the engineering office
Computer architecture has a familiar rule: a system can only speed up as much as the part you sped up allows. The same arithmetic applies in an engineering office.
Say the whole task takes 100 minutes: preparation 30, generation 40, verification 20, rework 10. You make generation ten times faster — 40 minutes becomes 4.
| Step | Before | After |
|---|---|---|
| Preparation | 30 min | 30 min |
| Generation | 40 min | 4 min |
| Verification | 20 min | 20 min |
| Rework | 10 min | 10 min |
| Total | 100 min | 64 min |
That is a 90% gain in generation and 36% overall. Both numbers are true; which one you publish is a choice. The problem is when the purchase decision is made on the first one.
The framework I use in my own projects has four rows:
| Step | Before | After | Why it is measured |
|---|---|---|---|
| Preparation | ? min | ? min | Automation demands input; that input is not free |
| Generation | ? min | ? min | This is usually the only thing a vendor measured |
| Verification | ? min | ? min | Output you do not trust is checked every time |
| Rework | ? min | ? min | If the error rate did not drop, the gain is misleading |
Filling all four rows before and after resolves most inflated claims on its own.
Who keeps the log, and how
The person who wrote the automation should not keep the log. They know the path where the tool works best and will pick it without noticing. The log is kept by the person who actually does the work; the developer only supplies the table format.
The simplest method that works in practice: two weeks before, two weeks after, same person, same task type, a stopwatch on a phone. No better tool is needed — what is needed is that the same task is measured.
Time saved alone does not make a decision. Payback period does — and the calculation is three sentences long.
First find the monthly gain: multiply the minutes saved per job by the monthly repetition count, then convert to hours. Next write down the total cost: development hours plus the maintenance hours you will set aside for the first year. Payback period is that cost divided by the monthly gain.
Take the example above: 36 minutes saved per job, 40 jobs a month. That is 1440 minutes, or 24 hours a month. If development takes 80 hours and first-year maintenance 20, total cost is 100 hours. 100 ÷ 24 ≈ 4.2 months.
Under six months the decision is usually clear. Past twelve months, one of two things has to happen: narrow the scope to cut development time, or drop the idea.
1. Measuring the best case. Try the automation on the most convenient part and you will get a lovely number. Try it on an average job — that is the number you will actually live with.
2. Ignoring the learning curve. A new tool slows work down for the first two weeks. Measure in week one and you will miss the gain; never measure it and you hide the real cost. The right approach: start measuring from week three and record the learning period separately as a one-off cost.
3. Forgetting maintenance. Automation is live software. When a product family changes, the rule layer changes; when the CAD version moves up, API calls get re-checked. A few days a year comes out of the gain.
4. Not converting the gain into anything. Say you saved 24 hours a month; if those hours were not redirected to other work, nothing changes in the ledger. Deciding where the gain goes is part of the measurement.
5. Measuring one person. The number from your best user does not represent the team average. Measure at least two different users.
Honestly, not every automation needs measuring.
Setting up measurement for a ten-minute task done twice a month costs more than the automation itself. In such cases the decision is already clear: either it is an obvious small convenience, or it should not be built at all.
Measurement matters when the decision is genuinely uncertain: deciding whether to widen scope, asking for budget, or arguing about whether a tool actually works.
When the table is filled, I look for three answers:
Which step grew? Automation can shorten one step while lengthening another. The most common: generation shrinks, preparation grows — because the automation now demands tidier input. That is not a bad outcome, but it must be visible.
Did the rework rate drop? Even if total time is unchanged, fewer corrections means the automation is producing value. Consistency is a gain worth measuring on its own.
Did variance drop? If a manual task ranged from 20 to 70 minutes and now sits between 8 and 12, you have gained planning capability. That line never appears in the table, but for teams doing production planning it can be worth more than the average gain.
On the light steel CAD/CAM platform, internal measurements against the previous CAD system reported a clear speedup in the preparation phase. That number is meaningful because what was measured was not a single operation but the whole preparation phase — input collection and output checking included.
I applied the same discipline to the SolidWorks add-in work, where the main gain turned out to be output consistency rather than raw time: drawings for a product family now come out in the same layout no matter who produces them. The same release-package problem is often prototyped first with SolidWorks Python scripting before it becomes an add-in.
Project details are in the light steel CAD/CAM case study.
How do you calculate the gain from automation? Measure preparation, generation, verification and rework separately, before and after. Multiply the total difference by monthly repetitions, then divide development plus maintenance hours by that monthly gain to get the payback period.
Are vendor claims like "90% faster" reliable? The number is usually accurate but narrow: it measures only the generation step with ideal input. Including preparation, verification and rework, the same tool typically delivers a 30–50% total gain.
When should measurement start? From the third week after rollout. The first two weeks are the learning period and understate the gain; record that period separately as a one-off cost.
What payback period is acceptable? Under six months the decision is usually clear. Past twelve months you should either narrow the scope to cut development time or drop the idea.
How is maintenance cost included? Budgeting maintenance hours at roughly a quarter of the annual development time is a realistic starting point for year one. CAD version upgrades, product family changes and rule updates come out of that line.
Besides time, what else should be measured? Rework rate, first-time-right percentage and variance in duration. Reduced variance in particular can matter more to production planning teams than the average time saved.
Look at the four steps, not the percentage. The only question worth asking about an automation tool: which step does it shorten, and which step does it lengthen?
Next in this series: automating the move from 2D drawings to 3D models — where the real question is again which step can genuinely be handed over.