Valid or reliable – trying to break the tradeoff
Yesterday I wrote about how measurements are often in conflict between being valid indicators and reliable indicators in Valid or reliable – take your pick. How can we possibly break this tension so there is no tradeoff?
One method, staying within the domain of measurement, is a stronger use of ratios. Many of the reliability failures of measures are because other factors affect the outcome of the metric. If I’m measuring costs for example, sales going up or down may impact what the costs have to be. If I measure defects, the volume of work can increase the number even if my process is improved.
It sounds simple, yet too uncommon, to measure our goals as ratios. Don’t measure costs in dollars; instead measure it as a ratio but whatever other factor most directly impacts the outcome. Overall costs should be measured as a ratio against sales.
Avoid absolute measures when setting metrics as goals and experiments. Ratios are much more reliable. It doesn’t mean that this will solve everything that’s wrong with the reliability of measures. Remember, as Einstein said, “not everything that can be measured matters, and not everything that matters can be measured.”