Measurement Misnomers, and Toyota Dealership Problems
On our LinkedIn Group we get many good discussions and questions. We had one from Roger Cook that I thought was worth repeating and expanding upon. Here is the question:
I’m curious if any of you have foolproof ways of insuring your metrics (which lean folks are famous for measuring everything) are actually measuring what you think they are.
The genesis of this question is my experience with Toyota, itself. I own 2 Toyotas and have them serviced at a Toyota dealership every 5000 miles. Each time, as I’m leaving, the service attendant tells me I am likely to get a questionaire from Toyota rating the service I just received, and would I please rate them excellent because this is the only acceptable answer. They obviously get graded on the results of these questionaires. Its a little alarming to me that a company as famous as Toyota is for metrics and lean, wouldn’t see that their survey results are not more than a little biased if the only acceptabale answer is excellent. Or am I the only one to see a bit of irony in this situation?
There are actually a couple of different levels at which I would like to answer this question. First, on the challenge of the dealership. And second, on measurements.
On Dealership, and Why Your Customer Doesn’t Care Whose Problem It Is
Roger’s example makes a great point. Your customer doesn’t care if the problem they experienced is with you, the supplier, the retail outlet. If Amazon puts wrong information on the product page, it reflects badly on you? If your supplier ships you defective components, your customer doesn’t care that it’s their fault.
If Wal-Mart sells a faulty product by Dell, it hurts both their brands. If Wal-Mart delivers faulty service when selling the Dell, it hurts both their brands. How does this apply to Roger’s problem, or Toyota’s?
Dealers are owned by the same people at Toyota as they are at Chevy. You get a wide range of service levels, behaviors, and attitudes, many of which Toyota would be and are ashamed of. With Roger’s specific situation, it gets worse than that. There have been cases in the past where dealerships hold banquets and other nice events for customers. The price of admission: an incomplete survey. Then the dealer would complete the surveys themselves. I’m not sure if this is quite illegal, but we don’t need to debate how far it crosses the ethics lines. Fraud of course destroys the integrity of any measurements, to Roger’s point.
The Fallacy of Measurement as King
Measurements are important. They serve a useful purpose. When I visit an organization to perform an assessment and find they have almost no measurements, I can bet the organization won’t be very strong. Facts and data are not the same thing, as Mark Graban commented on.
Any time a measurement is tied to an incentive, then it is likely to be manipulated at some level. If my pay is tied to the rating at the dealership, then the small step of asking you to rate them excellent seems worth it, because most people don’t knowingly stick it to the person that just helped them. When this happens to me, I usually lower my score and comment that I was asked to rate them a perfect score.
To answer Roger’s underlying real question about ensuring metric accuracy, the only real way that I’ve found is direct observation. Metrics are abstractions, by definition. That means they never truly represent reality. That means they are subject to manipulation, intentional or otherwise. To truly understand what is going on, you must get past the abstract and see what is truly happening.
We talk about direction observation in The Hitchhiker’s Guide to Lean . Many “lean” people talk about going to the gemba, which means “actual place.” But it takes more than going to the actual place. It required observation. This means study. It means patience. And it requires that you have the ability to digest, understand, and add meaning to what you see.
Don’t throw out your measurements. Just know that they don’t give you the whole story. Take the time to understand the current state as it actually happens, through direct observation.