Measurement Misnomers, and Toyota Dealership Problems
On our LinkedIn Group we get many good discussions and questions. We had one from Roger Cook that I thought was worth repeating and expanding upon. Here is the question:
I’m curious if any of you have foolproof ways of insuring your metrics (which lean folks are famous for measuring everything) are actually measuring what you think they are.
The genesis of this question is my experience with Toyota, itself. I own 2 Toyotas and have them serviced at a Toyota dealership every 5000 miles. Each time, as I’m leaving, the service attendant tells me I am likely to get a questionaire from Toyota rating the service I just received, and would I please rate them excellent because this is the only acceptable answer. They obviously get graded on the results of these questionaires. Its a little alarming to me that a company as famous as Toyota is for metrics and lean, wouldn’t see that their survey results are not more than a little biased if the only acceptabale answer is excellent. Or am I the only one to see a bit of irony in this situation?
There are actually a couple of different levels at which I would like to answer this question. First, on the challenge of the dealership. And second, on measurements.
On Dealership, and Why Your Customer Doesn’t Care Whose Problem It Is
Roger’s example makes a great point. Your customer doesn’t care if the problem they experienced is with you, the supplier, the retail outlet. If Amazon puts wrong information on the product page, it reflects badly on you? If your supplier ships you defective components, your customer doesn’t care that it’s their fault.
If Wal-Mart sells a faulty product by Dell, it hurts both their brands. If Wal-Mart delivers faulty service when selling the Dell, it hurts both their brands. How does this apply to Roger’s problem, or Toyota’s?
Dealers are owned by the same people at Toyota as they are at Chevy. You get a wide range of service levels, behaviors, and attitudes, many of which Toyota would be and are ashamed of. With Roger’s specific situation, it gets worse than that. There have been cases in the past where dealerships hold banquets and other nice events for customers. The price of admission: an incomplete survey. Then the dealer would complete the surveys themselves. I’m not sure if this is quite illegal, but we don’t need to debate how far it crosses the ethics lines. Fraud of course destroys the integrity of any measurements, to Roger’s point.
The Fallacy of Measurement as King
Measurements are important. They serve a useful purpose. When I visit an organization to perform an assessment and find they have almost no measurements, I can bet the organization won’t be very strong. Facts and data are not the same thing, as Mark Graban commented on.
Any time a measurement is tied to an incentive, then it is likely to be manipulated at some level. If my pay is tied to the rating at the dealership, then the small step of asking you to rate them excellent seems worth it, because most people don’t knowingly stick it to the person that just helped them. When this happens to me, I usually lower my score and comment that I was asked to rate them a perfect score.
To answer Roger’s underlying real question about ensuring metric accuracy, the only real way that I’ve found is direct observation. Metrics are abstractions, by definition. That means they never truly represent reality. That means they are subject to manipulation, intentional or otherwise. To truly understand what is going on, you must get past the abstract and see what is truly happening.
We talk about direction observation in The Hitchhiker’s Guide to Lean . Many “lean” people talk about going to the gemba, which means “actual place.” But it takes more than going to the actual place. It required observation. This means study. It means patience. And it requires that you have the ability to digest, understand, and add meaning to what you see.
Don’t throw out your measurements. Just know that they don’t give you the whole story. Take the time to understand the current state as it actually happens, through direct observation.
For the record, this is not about Toyota’s current quality problem. I am choosing to postpone talking about it until we learn more facts, since speculation and manipulation seems to be much greater than facts at this point.
For the record, this is not about Toyota’s current quality problem. I am choosing to postpone talking about it until we learn more facts, since speculation and manipulation seems to be much greater than facts at this point.
For the record, this is not about Toyota’s current quality problem. I am choosing to postpone talking about it until we learn more facts, since speculation and manipulation seems to be much greater than facts at this point.
Great thinking Jamie!
I would add that probably some dealers are acting upon what other suppliers have teached them: you must have a perfect score on the query or you’re out of business. I don’t think Toyota has teached that behaviour to them and also think that Toyota would have been happier to know their problems and customers concerns in advance…
It could be that today’s Toyota problems depend also on this factor, of not having teached their dealers how to comunicate with them and with the customers effectively in order to not have the need to take all the vehicles off the market…
Great thinking Jamie!
I would add that probably some dealers are acting upon what other suppliers have teached them: you must have a perfect score on the query or you’re out of business. I don’t think Toyota has teached that behaviour to them and also think that Toyota would have been happier to know their problems and customers concerns in advance…
It could be that today’s Toyota problems depend also on this factor, of not having teached their dealers how to comunicate with them and with the customers effectively in order to not have the need to take all the vehicles off the market…
Great thinking Jamie!
I would add that probably some dealers are acting upon what other suppliers have teached them: you must have a perfect score on the query or you’re out of business. I don’t think Toyota has teached that behaviour to them and also think that Toyota would have been happier to know their problems and customers concerns in advance…
It could be that today’s Toyota problems depend also on this factor, of not having teached their dealers how to comunicate with them and with the customers effectively in order to not have the need to take all the vehicles off the market…
Jamie,
Well, my Acura dealer pulls the same perfect survey stunt with me. Can’t say that I have figured out a proper and pithy response yet – although the service is typically excellent.
As for measurement not quite being king, I agree. There must be balance, a balance that I believe is provided in a mature lean management system. Within that system, leader standard work and the related visual controls facilitate direct observation. The system’s daily accountability process encompasses, among other things, the tracking and review of balanced performance metrics (safety, quality, delivery, cost, etc.) within the context of brief, focused meetings. These same meetings take into account the “findings” from direct observations – those prompted by leader standard work and formal and informal gemba walks. This three legged stool of leader standard work, visual controls and a daily accountability drives both process performance and process adherence.
While metrics may be an abstraction, they do have their place. They do provide stakeholders with a perspective on performance trends and gaps and, if properly done, tell a story and provide some (initial) direction relative to countermeasures. The danger is when metrics are not informed by direct observation. Unfortunately, often the higher one goes in an organization the more the balance is tilted to metrics…until there is no balance, just “really smart” people in a conference room with charts and graphs, unencumbered by reality.
Jamie,
Well, my Acura dealer pulls the same perfect survey stunt with me. Can’t say that I have figured out a proper and pithy response yet – although the service is typically excellent.
As for measurement not quite being king, I agree. There must be balance, a balance that I believe is provided in a mature lean management system. Within that system, leader standard work and the related visual controls facilitate direct observation. The system’s daily accountability process encompasses, among other things, the tracking and review of balanced performance metrics (safety, quality, delivery, cost, etc.) within the context of brief, focused meetings. These same meetings take into account the “findings” from direct observations – those prompted by leader standard work and formal and informal gemba walks. This three legged stool of leader standard work, visual controls and a daily accountability drives both process performance and process adherence.
While metrics may be an abstraction, they do have their place. They do provide stakeholders with a perspective on performance trends and gaps and, if properly done, tell a story and provide some (initial) direction relative to countermeasures. The danger is when metrics are not informed by direct observation. Unfortunately, often the higher one goes in an organization the more the balance is tilted to metrics…until there is no balance, just “really smart” people in a conference room with charts and graphs, unencumbered by reality.
Jamie,
Well, my Acura dealer pulls the same perfect survey stunt with me. Can’t say that I have figured out a proper and pithy response yet – although the service is typically excellent.
As for measurement not quite being king, I agree. There must be balance, a balance that I believe is provided in a mature lean management system. Within that system, leader standard work and the related visual controls facilitate direct observation. The system’s daily accountability process encompasses, among other things, the tracking and review of balanced performance metrics (safety, quality, delivery, cost, etc.) within the context of brief, focused meetings. These same meetings take into account the “findings” from direct observations – those prompted by leader standard work and formal and informal gemba walks. This three legged stool of leader standard work, visual controls and a daily accountability drives both process performance and process adherence.
While metrics may be an abstraction, they do have their place. They do provide stakeholders with a perspective on performance trends and gaps and, if properly done, tell a story and provide some (initial) direction relative to countermeasures. The danger is when metrics are not informed by direct observation. Unfortunately, often the higher one goes in an organization the more the balance is tilted to metrics…until there is no balance, just “really smart” people in a conference room with charts and graphs, unencumbered by reality.
Thanks for the comments Mark. Metrics absolutely have their place. In fact, you might even call me a metric nut. But you must be careful in (a) how you design the metric and (b) how you use it to make decisions. Any powerful tool must be used with care.
Thanks for the comments Mark. Metrics absolutely have their place. In fact, you might even call me a metric nut. But you must be careful in (a) how you design the metric and (b) how you use it to make decisions. Any powerful tool must be used with care.
Thanks for the comments Mark. Metrics absolutely have their place. In fact, you might even call me a metric nut. But you must be careful in (a) how you design the metric and (b) how you use it to make decisions. Any powerful tool must be used with care.
We follow strict metrics as well to ensure we ‘re delivering a plesaurable home buying, building and ownership experience. And our production, construction and sales teams review those metrics often.
It’s funny that you mentioned manipulating the metrics in your article. At one point, many years ago, we had incentives tied to the performance depicted on those metrics. However, we found people circumventing the system to improve their numbers, and came to the realization that metrics and incentives must be mutually exclusive.
Today, our metrics play a large role in our data gathering and follow-up on PDCAs, and are warning lights that there may be problem occurring and we need to “go see.”
We follow strict metrics as well to ensure we ‘re delivering a plesaurable home buying, building and ownership experience. And our production, construction and sales teams review those metrics often.
It’s funny that you mentioned manipulating the metrics in your article. At one point, many years ago, we had incentives tied to the performance depicted on those metrics. However, we found people circumventing the system to improve their numbers, and came to the realization that metrics and incentives must be mutually exclusive.
Today, our metrics play a large role in our data gathering and follow-up on PDCAs, and are warning lights that there may be problem occurring and we need to “go see.”
We follow strict metrics as well to ensure we ‘re delivering a plesaurable home buying, building and ownership experience. And our production, construction and sales teams review those metrics often.
It’s funny that you mentioned manipulating the metrics in your article. At one point, many years ago, we had incentives tied to the performance depicted on those metrics. However, we found people circumventing the system to improve their numbers, and came to the realization that metrics and incentives must be mutually exclusive.
Today, our metrics play a large role in our data gathering and follow-up on PDCAs, and are warning lights that there may be problem occurring and we need to “go see.”
I am glad this topic was brought up again. I do function from observation. The automotive supplier I last worked at had company, plant, department and personal KPI’s to meet. Everyone wanted a plan for improvement, which I provided at my manager level. However, I never followed let alone look at what I wrote down as a plan. There was an intent for what was written down, which was continual improvement. So I performed just that. Observations, working on the current “seen” bottleneck, improving processes with the people in mind who had to work within the processes, and so on. Every quarterly review of the KPI’s I had improvement as an outcome of “do what needs to be done now”. So at least the measurements had shown I was doing something positive. Though none of the improvements were done for the sake of the measurement.
I am glad this topic was brought up again. I do function from observation. The automotive supplier I last worked at had company, plant, department and personal KPI’s to meet. Everyone wanted a plan for improvement, which I provided at my manager level. However, I never followed let alone look at what I wrote down as a plan. There was an intent for what was written down, which was continual improvement. So I performed just that. Observations, working on the current “seen” bottleneck, improving processes with the people in mind who had to work within the processes, and so on. Every quarterly review of the KPI’s I had improvement as an outcome of “do what needs to be done now”. So at least the measurements had shown I was doing something positive. Though none of the improvements were done for the sake of the measurement.
I am glad this topic was brought up again. I do function from observation. The automotive supplier I last worked at had company, plant, department and personal KPI’s to meet. Everyone wanted a plan for improvement, which I provided at my manager level. However, I never followed let alone look at what I wrote down as a plan. There was an intent for what was written down, which was continual improvement. So I performed just that. Observations, working on the current “seen” bottleneck, improving processes with the people in mind who had to work within the processes, and so on. Every quarterly review of the KPI’s I had improvement as an outcome of “do what needs to be done now”. So at least the measurements had shown I was doing something positive. Though none of the improvements were done for the sake of the measurement.