Canary in a Coal Mine Metrics
Last week my IndustryWeek column titled Don’t Waste Your Metrics was released. It has generated a significant amount of discussion, as many companies find themselves in the situation of measuring far too many things, yet still not feeling like they know what is going on. I plan to build on this column, which is always quite space limited, with a series of blog posts on topics about metrics. This post is titled “The Canary in the Coal Mine.”
I’m not talking about the song by The Police; I’m talking about the analogy of how canaries were used in coal mines. They were used as sentinels to be early warning systems of toxic gases in the mine. The canaries would become sick from toxic fumes sooner than miners would notice them, and that would give them time to either escape or put on respirators.
When designed properly, metrics can accomplish the same thing.
Instead of trying to measure everything, choose something to measure that gives you an earlier indicator that something requires attention. That’s what metrics are for, after all. Here’s a couple of examples.
Many companies, particularly manufacturing companies, are employing the techniques of 5S, also known as workplace organization. I won’t go into all the reasons why; for that you can read one of my other posts or columns on the subject. But one critical aspect of 5S is the challenge of sustaining it. Audits because perhaps the most effective, albeit necessarily wasteful, methods to sustain 5S. For those in the trenches, having detailed checklists of what to check is very useful, because one thing out of place could create a significant problem down the path. But what if you’re the factory manager. Can you go through every shelf and toolbox looking for problems? Certainly not, at least not in a way that you can personally sustain. So what would be your canary in a coal mine?
In many areas, there is usually one spot that fills up with junk first. It’s usually a spot a little out of the way, and one that doesn’t have a purpose so clear that it keeps the junk out, much like the hall coat closet in many people’s homes. If our sustaining of 5S is going to fall apart, this is likely the first place it would show up. So instead of checking everything, just check this one spot. If you see it falling apart, you know some of the sustaining efforts are slipping and you can take corrective action before the rest of the area deteriorates. This one area is your canary in a coal mine.
What about customer satisfaction? Especially for companies with a broad customer base, deterioration of customer satisfaction doesn’t show up statistically very quickly. Further exacerbating the problem is that customers often have to experience the problem multiple times or at severe levels before they can be bothered with sharing the negative feedback. Therefore, by the time it shows up on your metrics dashboard, it may be very late.
Many companies have customers who are perhaps more sensitive. Perhaps they complain faster, or about more things, or about seemingly trivial things. These customers are not often talked about in favorable ways. But they can be used to serve you. If instead of looking at broad customer satisfaction data, use this one client, or group of customers, more closely. They will complain before other customers will. Yes, you will have to separate legitimate claims from the noise, but they are still an early warning system. If you insert a closer and more frequent look at that group of customers, it is your canary in a coal mine.
Most “canary in a coal mine” metrics are not comprehensive. They can’t be, by design. They are trying to predict a bigger problem, and they will sometimes get it wrong. Designing them will require both some creativity as well as deep understanding of cause-and-effect (a useful investment of time anyway). It will also require learning how to react, since there can be false signals.
But, wouldn’t you rather put on the respirator unnecessarily when the canary falls ill than end up experiencing the same condition yourself.
Reflection question: how do you design proactive canary-in-a-coal-mine metrics, and how do you manage the reaction to those metrics?
This is also exactly the idea with in-process measures (in addition to others). You catch the problem early, hopefully before you create a bunch of waste. The canary alerts you to go look. YOu do and fix things quickly and then get going again
It is good to note why these measures are useful as canary measure (because these may well be seen as wasteful since you are not acting on them for a long time). You do want to evaluate if they do become useless, but it can often be extremely valuable to have a measure that doesn’t give a signal for year if when it does it saves you a ton. You have a design a process that works for this – low cost, but that will be noticed if it goes bad (if people stop paying attention it might not do any good). If the canary drops dead but they were no longer monitored it doesn’t help.
The most useful tip is probably to have the measures monitored by the people working on the relevant process. This is obvious to decent lean operations but not obvious to many places canary measures would be used (lets have some new MBA is St. Louis get sent these reports each week and then send summaries up to the regional directors each month… ARGH).
Jamie. I love the pragmatic style and focus of this blog.
The shortest course in metrics should include this blog and the simple idea that metrics must be useful. Metrics must be used to DRIVE the business. Too many management teams have people simply collecting data. Pick a canary and when it looks sick; fix it. Glen
Jamie, I love the analogy, though I am sure the some people won’t. It is not how much data you can collect and analyze that will ever matter, it is how fast you can obtain and analyze and react to a few key measures that will determine how successful you will be.
In the seventies ther was a lot of talk about Activity Based Costing, but most of its efforts failed, not because it did not work, but because people especially at the top did not like what it revealed. Much of the real reason behind tracking excess useless data has nothing to do with improving anything, rather it is an effort to hide the truth. It feel good when you use the wider customer satisfaction data, instead of analyzing the narrower early compliants, but it is those early complaints that coulkd prevent huge problems.
Most executives spend more time talking about last years financial results, because they can spin them, it is very easy to spin poor early production data that says your quality, and production are slipping, though reacting to them in the right way will help build long-term success. The golden staff do not like to admit that there equipment may have been pushed to hard in the past, and now needs to be overhauled if it is to perform in the longrun, instead they blame production supervisors and staff for failing to meet their financial needs. Using the right types of data would prevent and detect problems earlier, and help keep production capacity at stable productive rates, but that doesn’t look or sound as good, nor does it earn you those fat quarterly bonuses most companies pay for financial performance.
Usually before a customer encounters a problem, an employee is irritated by a task he is performing. Something is incorrect with the process. Many subscribe to don’t bring me a problem, bring me a solution. Hence, problems persist.
Sometimes I wonder why instead of just focusing on metrics, management does not just ask employees what irritates them.