Can we claim a 25% productivity gain?
Name the measured task, participants and review work before turning a pilot result into a wider claim.
“you've got to evaluate and test before you can get to a fine-tuned approach.” (opens the source)
The question
Twelve Meridian employees volunteer for a drafting pilot. Each took 20 minutes on the measured task before the pilot. There is no random assignment or concurrent control group.
| Participants | Minutes after | |
|---|---|---|
| Faster group | 9 | 12 |
| Unchanged group | 2 | 20 |
| Slower participant | 1 | 32 |
All 12 participants are included. These are task times, not whole-job output.
“Calculate the observed change, write the claim you would approve, then account for the new review work.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Keep every participant in the mean
What would you do here? Open the move you’d make.
Use all 12 participants.
The strong move. Removing the slower result would overstate the gain.
Report only the nine faster participants.
Tempting, but it misses something. The pilot includes the unchanged and slower outcomes.
See how a strong candidate says it
“The total is 9 × 12 + 2 × 20 + 1 × 32 = 180 minutes. Divide by 12: the mean falls from 20 to 15 minutes, a 25% task-time reduction.”
Why it works: Removing the slower result would overstate the gain.
Step 2: Name the claim’s boundary
What would you do here? Open the move you’d make.
Say AI made the whole company 25% more productive.
Tempting, but it misses something. The packet does not measure the whole company or isolate cause.
State the task, sample and lack of a control.
The strong move. The sample and design limit the conclusion.
See how a strong candidate says it
“Approve an observed mean task-time reduction in this 12-person volunteer pilot. Do not call it a causal company-wide productivity gain.”
Why it works: The sample and design limit the conclusion.
Step 3: Keep time and output distinct
What would you do here? Open the move you’d make.
Keep the observed task-time claim.
The strong move. A time reduction and an output increase have different denominators.
Relabel the result as 25% more output.
Tempting, but it misses something. Neither the arithmetic nor the evidence supports that wording.
See how a strong candidate says it
“At fixed time, 20/15 − 1 is 33.3% more task capacity, if everything else is equal. That is a mechanical ratio, not measured output or quality.”
Why it works: A time reduction and an output increase have different denominators.
Step 4: Include the work found in review
Quality review finds 2 faster drafts each need 10 extra minutes of rework. This work was outside the original task timer.
What would you do here? Open the move you’d make.
Keep 25% because the first draft was faster.
Tempting, but it misses something. That excludes work needed to finish the task.
Add both rework periods and update the claim.
The strong move. The extra work belongs in the same cost boundary.
See how a strong candidate says it
“Rework adds 20 minutes: 200/12 = 16 minutes 40 seconds per task. The all-in time reduction is now 16.7%. Revise the claim.”
Why it works: The extra work belongs in the same cost boundary.
The answer, in short
- Decision
- Use the narrower observed claim and include review work.
- Because
- The pilot supports a task-time result with clear limits.
- What would change it
- A broader comparison with quality and all-in time could support a stronger claim.
Two drafts each need 10 extra minutes. What is the revised all-in saving?
Common traps
- Dropping the slow participant.
- Calling saved drafting time proven productivity.
Check your answer
- I included all participants.
- I named the sample.
- I separated time from output.
- I counted review work.
Last reviewed Nov 10, 2026