Valid or reliable – trying to break the tradeoff
Yesterday I wrote about how measurements are often in conflict between being valid indicators and reliable indicators in Valid or reliable – take your pick. How can we possibly break this tension so there is no tradeoff?
One method, staying within the domain of measurement, is a stronger use of ratios. Many of the reliability failures of measures are because other factors affect the outcome of the metric. If I’m measuring costs for example, sales going up or down may impact what the costs have to be. If I measure defects, the volume of work can increase the number even if my process is improved.
It sounds simple, yet too uncommon, to measure our goals as ratios. Don’t measure costs in dollars; instead measure it as a ratio but whatever other factor most directly impacts the outcome. Overall costs should be measured as a ratio against sales.
Avoid absolute measures when setting metrics as goals and experiments. Ratios are much more reliable. It doesn’t mean that this will solve everything that’s wrong with the reliability of measures. Remember, as Einstein said, “not everything that can be measured matters, and not everything that matters can be measured.”
Too often I have seen management look at a particular increase in defects and jump to conclusions. However, when you look more closely at the rate, it falls within statistical variance. I guess it must be psychological. We feel that more is bad, and something needs to be done, even though the ratio is the same!
Too often I have seen management look at a particular increase in defects and jump to conclusions. However, when you look more closely at the rate, it falls within statistical variance. I guess it must be psychological. We feel that more is bad, and something needs to be done, even though the ratio is the same!
Too often I have seen management look at a particular increase in defects and jump to conclusions. However, when you look more closely at the rate, it falls within statistical variance. I guess it must be psychological. We feel that more is bad, and something needs to be done, even though the ratio is the same!
Good post – this is why ROS (return on sales) has become my most important financial metric. What has been surprising is how hard it has been to convince others – including senior finance people – that it is valid, even though it is nearly synonymous with basic profitability. It’s almost as if it is TOO simple and therefore cannot be valid, even though those guys work in a world of ratios – just fancier sounding ones!
Good post – this is why ROS (return on sales) has become my most important financial metric. What has been surprising is how hard it has been to convince others – including senior finance people – that it is valid, even though it is nearly synonymous with basic profitability. It’s almost as if it is TOO simple and therefore cannot be valid, even though those guys work in a world of ratios – just fancier sounding ones!
Good post – this is why ROS (return on sales) has become my most important financial metric. What has been surprising is how hard it has been to convince others – including senior finance people – that it is valid, even though it is nearly synonymous with basic profitability. It’s almost as if it is TOO simple and therefore cannot be valid, even though those guys work in a world of ratios – just fancier sounding ones!