How to measure anything – book review and summary

I recently finished How to Measure Anything: finding the value of intangible in business. It’s a great book, I highly recommend it. Especially if you have a engineering/science background and find yourself in management.

Hubbard claims that there are no intangibles that you can’t measure. Central to this claim is his definition of measurement as “uncertainty reduction.” This means that the point of measuring is not to find THE answer, but to reduce our uncertainty about what the answer is. He suggests starting with the estimate of” calibrated” experts who can consistently estimate within a 90% confidence interval. From this initial range you calculate the value of more information. I.e. how much is it worth to further reduce your uncertainty. If the benefit of more measurement is worth the cost you should perform that measurement. He then provides some helpful measurement techniques.

The most helpful part of the book, at least for me, was his definition of measurement and his value of information calculations. Defining measurement as uncertainty reduction helped to break me out of an engineering mode of measurement. In engineering you can usually measure very accurately. In either case you then use “the answer” in the rest of your calculations. The problem with this is assuming that a measurement can or needs to give you “the answer” to be helpful. It doesn’t keep in mind what the measurement is for in most cases.

His calculations of information is a great way to examine current or potential measurements and ask if it’s worthwhile. I’m certain that many businesses are wasting lots  of money measuring things that are easy to measure with high certainty, but are mostly worthless in terms of information value. It would be much more valuable to gather a few important metrics, even if there is a big range at a 90% confidence interval.

The basic approach is:

1. Build a model of the problem. If you don’t understand how a variable affects the outcome, you won’t know what’s important and what you need to measure to what precision.
2. Gather what you already know about the problem. Use current estimates and “calibrated experts”
3. Calculate the value of gathering more information. You may already have enough information to come to your conclusion or make your decision. If not, you need to figure out if more information is worth the cost.
4. Take measurements of the high value variables.
5. Make your decision

My notes from the book:

Section 1 – No “intangible” is unmeasurable.

Chapter 1 – Hubbard suggests going through the book with some tough measurement problem in mind, you’ll find a way to measure it.

Also, he states there are three important propositions that define an approach to measurement in business:

1. Measurements inform uncertain decisions
2. There are many things to measure and many ways. Perfect certainty is rarely a realistic option.
3. Therefore management needs ways to reduce uncertainly about decisions.

Chapter 2 – Hubbard provides examples and strategies for  measurement: Fermi decompositions (like what MBAs use for case questions). He states, “The concept of measurement as ‘uncertainty reduction’ and not necessarily the elimination of uncertainty is a central theme of this book.”

Chapter 3 – People don’t think some things are measurable because they have misconceptions about three aspects of measurement:

• Concept of measurement (they      have an incorrect definition of measurement)
• Object of measurement (they      are not clear on what they need to measure)
• Methods of measurement (they      don’t have enough ways to measure)

Def of measurement: a quantitatively expressed reduction if uncertainty based on one or more observations.

Even a small reduction in uncertainty can be worth millions.

Clarification Chain:

1. If it matters at all, it is detectable/observable
2. If it is detectible, it can be detected as an amount (or range of possible amounts)
3. If it can be detected as a range of possible amounts, it can be measured.

Helpful way to think of what you need to measure: thought experiments. Ask “What if?” what you wanted to measure was happening in one group and not in another, what would be different?

It’s very important to know why you want to measure something and what decision will it effect.

“If you don’t know what to measure, measure anyway. You’ll learn what to measure.” – David Moore

Four useful measurement assumptions:

1. Your problem is not as unique as you think

1. You have more data than you think

1. You need less data than you think

1. An adequate amount of new data is more accessible than you think.

Rule of five: there is a 93.75% chance that the median of a population is between the smallest and largest  values in a random sample of five from that population.

His five step process for measuring

1. Define a decision problem and the relevant uncertainties
2. Determine what you know now
3. Compute the value of additional information
4. Apply the relevant measurement instrument(s) to high-value measurements
5. Make a decision and act on it

Section 2 – Before you measure

Chapter 4 – Clarify the measurement problem. What decision are you supporting? What will more information do for you? how will it change your decisions?

You need to understand the difference between uncertainty and risk. Uncertainty is simply the lack of certainty. You do not know the “true” outcome/state. The measurement of uncertainty is a set of probabilities assigned to a set of possibilities, e.g. there is a %50 chance of X, %30 chance of Y, and 20% chance of Z occurring. Risk is just a state of uncertainty where one or more of the possible outcome is negative.

Chapter 5 – Calibrating experts

You need to have a good grasp of probabilities and calibrate your intuitive understanding if a 90% confidence interval (add Wikipedia link).

It is possible to “calibrate” people so they are capable of repeatedly making guesses in a 90% CI (confidence interval). Calibration requires several exercises the most important if which is the equivalent bet exercise where you pretend to your 90% CI estimate is a bet versus an actual true 90% probability. If you are indifferent, then you are at a 90% confidence interval.

Chapter 6 – measuring risk through modeling

Monte Carlo simulations.

Chapter 7 – Measuring the value of information

Expected Value of information (EVI) = Reduction in expected opportunity loss (EOL)
EVI = EOLbefore info – EOLafter info

EOL = Chance of making a sub-optimal decision x Cost of making a sub-optimal decision.

Expected Value of Perfect Information (EVPI) = EOLbefore info

(EOL after is zero if information is perfect, you’d make the perfectly optimal decision)

A common measurement myth: When you have a lot of uncertainty, you need a lot of data to tell you something useful. Actually, if you have a lot of uncertainty now, you don’t need much data to reduce uncertainty significantly. When you have a lot of certainty already, then you need a lot of data to reduce uncertainty significantly.

Section 3 – Measurement methods

Decompose the problem into its parts.

Perform secondary research

Use basic methods of observation:

1. Does it leave a trail of any      kind?

1. If the trail doesn’t already      exist, can you observe it directly or at least a sample of it?
2. If it doesn’t appear to leave      behind a detectible trail can you devise a way to begin tracking it?
3. If tracking the existing      conditions doesn’t suffice, can the phenomenon be “forced” to      occur under conditions that allow for easier observation? (i.e. an      experiment)

Measure Just enough

Consider the Error

He then goes into more detailed statistical techniques