©2007 Dale Emery
“Dale, I have an estimating problem,” Paul said.
“My product manager, Sandra, wants to know when we’ll be done testing,
and I don‘t know how to estimate that.”
“What do you mean by ‘done’?” I asked.
Paul opened his mouth to say something, then closed it.
After a moment he said, “That’s a good question.
I’m not sure.”
Clients often ask me to help with ‘estimating problems.’
Sometimes a few quick questions reveal that
the problem is not an estimating problem, but a clarity problem.
It’s difficult to estimate well when you’re
unclear what you‘re estimating.
If you’re having difficulty estimating,
first define the variable you’re estimating.
Ask yourself, What variable, exactly, am I estimating?
Write your definition down as clearly as you can.
Paul wants to estimate the date when we will be done testing.
Next, test the definition for fuzziness.
One effective test is the ‘Mary had a little lamb’ Heuristic
from Gause and Weinberg’s book Exploring Requirements.
To apply this heuristic, read your definition several times,
each time emphasizing a different word.
What does each emphasis suggest about the clarity of the definition?
Let’s try this with Paul’s definition.
Emphasize the first word: The
date when we will be done testing.
This emphasis suggests that there is a single date when the testing will be
complete.
Does Paul’s product manager need
a single date?
Maybe not.
Perhaps Paul could structure the testing in
phases and estimate each phase separately.
That would reduce the complexity of each estimating task.
Try the technique yourself.
Emphasize each of the next few words in Paul’s definition and notice
what each emphasis suggests about the variable Paul is being asked
to estimate.
I’ll skip to the word done:
The date when we will be done testing.
What does it mean to be done testing?
When I asked Paul this question, he didn’t have an answer.
Other clients answer the question, but are
immediately unsatisfied by what comes out of their mouths.
“We’re done when we’ve found all the bugs,”
or, “We’re done when we’ve tested everything.”
As people utter these answers out loud, they realize that they will
never test everything, and will never know when they’ve found all the
bugs.
If these are our criteria for
being ‘done,’ we’ll never be done.
No wonder it’s hard to estimate.
So how can we solve this ‘estimating problem’?
Before we can answer that, we have to explore a related fuzziness,
the last word of the definition: The date when we will be
done testing.
What does testing mean?
There are lots of possible meanings.
It might mean looking for defects.
It might mean exercising the
system to assess its correctness.
It might mean discovering information about the quality of the system.
It might mean other things.
Let’s see if any of these meanings help us understand what done means.
When will we be done identifying the system’s
defects, or assessing its correctness, or discovering information about its
quality?
I don’t see a clear stopping point for any of those tasks.
We could continue each indefinitely.
The ‘Mary had a little lamb’ Heuristic has helped us to pinpoint the
ambiguity, but not to resolve it.
To resolve the ambiguity,
we can apply another clarifying technique that I call The Value Question:
“If we did that, what would it do for us?”
If we tested well, what would that do for us?
What value do we want testing to produce?
I see testing as an information service.
A primary purpose is to inform product stakeholders, to answer their questions
about the quality of the system so that they can make good decisions.
We inform developers so that they can trace
defects to their origins and fix them.
We inform project managers so that they can steer project
activities.
We inform product managers
so that they can decide whether to put the system into production.
If testing is about answering stakeholders’ questions about the quality of the
system, we will want to know what their key questions are, and what decisions
they will make based on the answers.
Paul met with Sandra to identify her key decisions and questions.
Her primary decision was whether to put the system into production.
She was confident, from earlier testing, that the system’s functionality
was okay, but less confident about its performance.
Her most important question was, “Will the system support 1000
concurrent users?”
And right now she
wants Paul to estimate when he will be able to answer that question.
Paul now has a more specific variable to estimate:
The date when we will determine whether the system will support 1000
concurrent users.
Though there is plenty of ambiguity here (as you will see if you apply the
‘Mary had a little lamb’ Heuristic), this is far more specific than
the date when we will be done testing.
Paul was able to estimate, with reasonable confidence,
when he would be able to determine Sandra’s primary question.
Sandra had several other key questions about the quality of the system.
“Does the new version start up at least as
fast as the previous version?
Does the daily billing report run in less than five minutes?
As data volume approaches one billion
records, will performance be fast enough with our current computers?”
Paul was able to translate each of these
questions into a variable that he could estimate well.
Notice that, as often happens, in clarifying Paul’s ‘estimating problem’
we have shifted the definition of the problem.
Even if Paul answers all of Sandra’s questions, he still might not be
done testing.
For example, if he discovers that the system will support only 300
concurrent users, he has answered Sandra’s question, but there will
be more testing to do after the developers increase the system’s capacity.
As it turns out, what Sandra wanted to know was not when Paul would be done
testing, but when he would be able to answer her questions about
the system’s performance.
We have redefined the problem so that it is not only clearer and easier
for Paul to solve, but also more valuable to Sandra.
The next time you have trouble estimating, test for clarity.
Use the ‘Mary had a little lamb’ Heuristic,
The Value Question, and any other techniques you know to clarify the variable
you are estimating.
Clarifying the problem can take you a long way toward solving it.