Test for Actual Use, not Intended Use
When you test, what attributes are you testing for? Most testing begins with design criteria. This is reasonable to include but not the right starting point. You must develop with the user in mind. You must test with the user in mind. You must test for actual use, not just the use you intended.
This past week included a recall of over 50 million window blinds. 50 million! That’s a lot of production before finding a problem. Now this is not a production flaw, it is as design flaw. This is of course a tragedy for those who lost a young child from the product. But it begs a question of how we test a product in the design phase.
Most every organization tests products, whether software all the way to kitchen utensils, for the intended use. That means for a blind, you roll it up and a down thousands of times to make sure it will still work. If you are testing car doors, you open and close them thousands of times. If you are testing kitchen tongs, you make sure that they can withstand head, can be cleaned, and don’t flake or brake. You start with the key design criteria, figure out the limits of the design needs, then figure out how to test them. But this by no means ensures the product won’t fail. The reason for this?
Customers don’t always follow the intended use.
And just for the record, “customers are stupid” is not a good defense. Products get used however they get used. We need to learn to understand and even anticipate how customers will use a product, and test for that.
As an example, from a much earlier life of mine, there were problems with minivans. The rear window wipers would always break. The motors and mechanisms were tested under wind, rain, snow, and ice and there was never a problem. All the testing proved the product was fine. All the field results suggested things were all wrong.
Until a team really spend time observing the product in use, by its real user, a family. What do kids do? They climb. What do they climb on? Anything that’s around. And what do they use to pull themselves up on the bumper of the car? The window wiper. The intended use was to clear the window. The actual use was a climbing handle. Once the actual use was understood, design changes and testing changes could be made appropriate.
Here are your action steps:
- Don’t say you understand your customer’s use until you really observe it in action.
- Develop a list of criteria that represents testing their real and extreme uses.
- Develop the methods that will test for the real use of the product.
Design and test for actual use, not intended use.
Jamie, nice thoughts. Now not to be preaching tools this morning but wouldn’t a DFMEA (Design Failure Mode Effects Analysis) be an effective method to understand the customers requirements and also their expecations. I konw this can’t catch everything but it certaintly is a good process to make you think about your product form, fit, and function in a means where you can understand the risk of failure. And of course as you suggest this toola can be used with the end customers to better understand how they may use or misuse the product.
Tim McMahon
http://leanjourneytruenorth.blogspot.com
Jamie, nice thoughts. Now not to be preaching tools this morning but wouldn’t a DFMEA (Design Failure Mode Effects Analysis) be an effective method to understand the customers requirements and also their expecations. I konw this can’t catch everything but it certaintly is a good process to make you think about your product form, fit, and function in a means where you can understand the risk of failure. And of course as you suggest this toola can be used with the end customers to better understand how they may use or misuse the product.
Tim McMahon
http://leanjourneytruenorth.blogspot.com
Jamie, nice thoughts. Now not to be preaching tools this morning but wouldn’t a DFMEA (Design Failure Mode Effects Analysis) be an effective method to understand the customers requirements and also their expecations. I konw this can’t catch everything but it certaintly is a good process to make you think about your product form, fit, and function in a means where you can understand the risk of failure. And of course as you suggest this toola can be used with the end customers to better understand how they may use or misuse the product.
Tim McMahon
http://leanjourneytruenorth.blogspot.com
I had opportunity to work for VW/Audi in the ‘80’s during Audi’s unintended acceleration predicament.
Audi’s “driver error†defense basically translated to “customers are stupidâ€. Audi test/product validation engineers in lab coats went on 60 minutes and declared this to be true. Turns out, the producers of 60 minutes were not stupid. They interviewed several intelligent folks who drove their cars through walls, into swimming pools, and off bridges.
What was most interesting is that many VW/employees at the time also referred to their customers as “stupidâ€. You were not popular at work if you disagreed with the driver error defense. I heard our CFO claim he drove 40 miles and not once did his Audi 5000 “surge forward unintentionallyâ€. What? More damaging, your future career status depended on you supporting “stupid is as stupid does†theory.
The result was that a group of stupid people hired stupid legal counsel and hit Audi on the head square and smartly.
Stupid words hurt more than customer relationships. They hurt cash flow. They damage careers.
Jim Baran
Owner, Value Stream Leadership.
I had opportunity to work for VW/Audi in the ‘80’s during Audi’s unintended acceleration predicament.
Audi’s “driver error†defense basically translated to “customers are stupidâ€. Audi test/product validation engineers in lab coats went on 60 minutes and declared this to be true. Turns out, the producers of 60 minutes were not stupid. They interviewed several intelligent folks who drove their cars through walls, into swimming pools, and off bridges.
What was most interesting is that many VW/employees at the time also referred to their customers as “stupidâ€. You were not popular at work if you disagreed with the driver error defense. I heard our CFO claim he drove 40 miles and not once did his Audi 5000 “surge forward unintentionallyâ€. What? More damaging, your future career status depended on you supporting “stupid is as stupid does†theory.
The result was that a group of stupid people hired stupid legal counsel and hit Audi on the head square and smartly.
Stupid words hurt more than customer relationships. They hurt cash flow. They damage careers.
Jim Baran
Owner, Value Stream Leadership.
I had opportunity to work for VW/Audi in the ‘80’s during Audi’s unintended acceleration predicament.
Audi’s “driver error†defense basically translated to “customers are stupidâ€. Audi test/product validation engineers in lab coats went on 60 minutes and declared this to be true. Turns out, the producers of 60 minutes were not stupid. They interviewed several intelligent folks who drove their cars through walls, into swimming pools, and off bridges.
What was most interesting is that many VW/employees at the time also referred to their customers as “stupidâ€. You were not popular at work if you disagreed with the driver error defense. I heard our CFO claim he drove 40 miles and not once did his Audi 5000 “surge forward unintentionallyâ€. What? More damaging, your future career status depended on you supporting “stupid is as stupid does†theory.
The result was that a group of stupid people hired stupid legal counsel and hit Audi on the head square and smartly.
Stupid words hurt more than customer relationships. They hurt cash flow. They damage careers.
Jim Baran
Owner, Value Stream Leadership.
Great post.
Great post.
Great post.
I love this post! I was working with a friend of mine who owns a veterinary clinic and came across this same challenge. He was having trouble with his staff running a particular blood test. The steps were to draw the blood, place it in the centrifuge, extract the correct material and place it on a test strip. insert test strip into machine and then press a sequence of about 7 buttons that were not intuitive. If anything was done incorrectly then you’d have to start the test over. He thought the employees were incompetent. I told him your process is impossible and not every intuitive.
We standardized the processes and gave a check list so every step was followed consistently and what do you know no more errors. When it came time to buy a new machine we took into consideration the ease of use and the new process only takes 1 step of loading the blood into the machine and setting the animal type.
It surprises me how many companies won’t ask for customer input.
Ankit
I love this post! I was working with a friend of mine who owns a veterinary clinic and came across this same challenge. He was having trouble with his staff running a particular blood test. The steps were to draw the blood, place it in the centrifuge, extract the correct material and place it on a test strip. insert test strip into machine and then press a sequence of about 7 buttons that were not intuitive. If anything was done incorrectly then you’d have to start the test over. He thought the employees were incompetent. I told him your process is impossible and not every intuitive.
We standardized the processes and gave a check list so every step was followed consistently and what do you know no more errors. When it came time to buy a new machine we took into consideration the ease of use and the new process only takes 1 step of loading the blood into the machine and setting the animal type.
It surprises me how many companies won’t ask for customer input.
Ankit
I love this post! I was working with a friend of mine who owns a veterinary clinic and came across this same challenge. He was having trouble with his staff running a particular blood test. The steps were to draw the blood, place it in the centrifuge, extract the correct material and place it on a test strip. insert test strip into machine and then press a sequence of about 7 buttons that were not intuitive. If anything was done incorrectly then you’d have to start the test over. He thought the employees were incompetent. I told him your process is impossible and not every intuitive.
We standardized the processes and gave a check list so every step was followed consistently and what do you know no more errors. When it came time to buy a new machine we took into consideration the ease of use and the new process only takes 1 step of loading the blood into the machine and setting the animal type.
It surprises me how many companies won’t ask for customer input.
Ankit
Tim, DFMEA is a tool, not a process. It only helps you capture and plan around known failures. If you are evaluating a product from how it would actually be used, those failures wouldn’t show up on your DFMEA.
I have forgotten about the VW story Jim, that’s a good example. And thanks for sharing your example Ankit.
Tim, DFMEA is a tool, not a process. It only helps you capture and plan around known failures. If you are evaluating a product from how it would actually be used, those failures wouldn’t show up on your DFMEA.
I have forgotten about the VW story Jim, that’s a good example. And thanks for sharing your example Ankit.
Tim, DFMEA is a tool, not a process. It only helps you capture and plan around known failures. If you are evaluating a product from how it would actually be used, those failures wouldn’t show up on your DFMEA.
I have forgotten about the VW story Jim, that’s a good example. And thanks for sharing your example Ankit.
Mmm. This is a great post. If only we could bring more cultural anthropologists into the modern corporation.
Mmm. This is a great post. If only we could bring more cultural anthropologists into the modern corporation.
Mmm. This is a great post. If only we could bring more cultural anthropologists into the modern corporation.
Jon, that’s a very interesting concept. It is rare that I get to meet a true anthropologist that is studying business, but every time I do I am impressed both by their observations and also their process. I think large organizations that can afford it would be wise to put someone from this background on their organizational development staffs. And perhaps we should train OD professionals in the methods of anthropology as well.
Jon, that’s a very interesting concept. It is rare that I get to meet a true anthropologist that is studying business, but every time I do I am impressed both by their observations and also their process. I think large organizations that can afford it would be wise to put someone from this background on their organizational development staffs. And perhaps we should train OD professionals in the methods of anthropology as well.
Jon, that’s a very interesting concept. It is rare that I get to meet a true anthropologist that is studying business, but every time I do I am impressed both by their observations and also their process. I think large organizations that can afford it would be wise to put someone from this background on their organizational development staffs. And perhaps we should train OD professionals in the methods of anthropology as well.
It’s good that you emphasize “going to the gemba” as we’d say in the lean world. That doesn’t apply just to going to the shopfloor (or the nursing unit) but also applies to going to see your real customers using the product in every day life.
It’s good that you emphasize “going to the gemba” as we’d say in the lean world. That doesn’t apply just to going to the shopfloor (or the nursing unit) but also applies to going to see your real customers using the product in every day life.
It’s good that you emphasize “going to the gemba” as we’d say in the lean world. That doesn’t apply just to going to the shopfloor (or the nursing unit) but also applies to going to see your real customers using the product in every day life.
Is this not why you should routinely conduct D and P FMEAs? And also conduct formal VOC events; whether or not it’s QFD, focus groups, individual interviews, contextual inquiry, ethnographic techniques, etc
Is this not why you should routinely conduct D and P FMEAs? And also conduct formal VOC events; whether or not it’s QFD, focus groups, individual interviews, contextual inquiry, ethnographic techniques, etc
Is this not why you should routinely conduct D and P FMEAs? And also conduct formal VOC events; whether or not it’s QFD, focus groups, individual interviews, contextual inquiry, ethnographic techniques, etc
Rob, it is, but that’s doesn’t solve the problem. If you do a focus group, and ask them about features or use, they will often tell you about how they plan to use it. That doesn’t necessarily match how they ACTUALLY use it. And D and P FMEAs only work if you KNOW what the failure modes are and ask the right questions. FMEA doesn’t solve that problem for you, it only gives you the template on which to capture it.
Rob, it is, but that’s doesn’t solve the problem. If you do a focus group, and ask them about features or use, they will often tell you about how they plan to use it. That doesn’t necessarily match how they ACTUALLY use it. And D and P FMEAs only work if you KNOW what the failure modes are and ask the right questions. FMEA doesn’t solve that problem for you, it only gives you the template on which to capture it.
Rob, it is, but that’s doesn’t solve the problem. If you do a focus group, and ask them about features or use, they will often tell you about how they plan to use it. That doesn’t necessarily match how they ACTUALLY use it. And D and P FMEAs only work if you KNOW what the failure modes are and ask the right questions. FMEA doesn’t solve that problem for you, it only gives you the template on which to capture it.
Two stories:
When Alan Mulally first came to Ford, he arranged to meet a panel of testers at the Consumer Reports facility and took along a group of engineers. A woman had been testing the new Ford Edge and said that she had been grocery shopping on a rainy day. When she got to the vehicle with her bags, she found it difficult to operate the cargo door and juggle her groceries. We’ve all been there. The engineers proceeded to tell her why the vehicle was the way it was, actually beginning to argue with her, and Mullally basically told them to shut up and listen.
The other is about Mike’s first IT supervisor at Ford in the mid-80s when they were writing a new version of their material supply system for the assembly plants. Remember that if there’s a problem with this system, the plant goes down and all hell breaks loose at the data center. Jim sat up nights trying to come up with ways to break the system, and the programmers would come in the next day and have to find a way to prevent that failure. A team-based FMEA would probably have been better than relying on one obsessive person, but he taught Mike to have a passion for anticipating problems and respect for the user who might come up with a condition that never occurred to the programmers in their Dearborn offices.
Of course, in those days, each programmer took a turn on the night-call list. No remote access–if a call came at 2 a.m., he or she had to get out of bed and drive to the office and sit there alone trying to fix the problem while the plant guys kept reminding him how many millions it was costing. That’ll make you build in reliability!
Two stories:
When Alan Mulally first came to Ford, he arranged to meet a panel of testers at the Consumer Reports facility and took along a group of engineers. A woman had been testing the new Ford Edge and said that she had been grocery shopping on a rainy day. When she got to the vehicle with her bags, she found it difficult to operate the cargo door and juggle her groceries. We’ve all been there. The engineers proceeded to tell her why the vehicle was the way it was, actually beginning to argue with her, and Mullally basically told them to shut up and listen.
The other is about Mike’s first IT supervisor at Ford in the mid-80s when they were writing a new version of their material supply system for the assembly plants. Remember that if there’s a problem with this system, the plant goes down and all hell breaks loose at the data center. Jim sat up nights trying to come up with ways to break the system, and the programmers would come in the next day and have to find a way to prevent that failure. A team-based FMEA would probably have been better than relying on one obsessive person, but he taught Mike to have a passion for anticipating problems and respect for the user who might come up with a condition that never occurred to the programmers in their Dearborn offices.
Of course, in those days, each programmer took a turn on the night-call list. No remote access–if a call came at 2 a.m., he or she had to get out of bed and drive to the office and sit there alone trying to fix the problem while the plant guys kept reminding him how many millions it was costing. That’ll make you build in reliability!
Two stories:
When Alan Mulally first came to Ford, he arranged to meet a panel of testers at the Consumer Reports facility and took along a group of engineers. A woman had been testing the new Ford Edge and said that she had been grocery shopping on a rainy day. When she got to the vehicle with her bags, she found it difficult to operate the cargo door and juggle her groceries. We’ve all been there. The engineers proceeded to tell her why the vehicle was the way it was, actually beginning to argue with her, and Mullally basically told them to shut up and listen.
The other is about Mike’s first IT supervisor at Ford in the mid-80s when they were writing a new version of their material supply system for the assembly plants. Remember that if there’s a problem with this system, the plant goes down and all hell breaks loose at the data center. Jim sat up nights trying to come up with ways to break the system, and the programmers would come in the next day and have to find a way to prevent that failure. A team-based FMEA would probably have been better than relying on one obsessive person, but he taught Mike to have a passion for anticipating problems and respect for the user who might come up with a condition that never occurred to the programmers in their Dearborn offices.
Of course, in those days, each programmer took a turn on the night-call list. No remote access–if a call came at 2 a.m., he or she had to get out of bed and drive to the office and sit there alone trying to fix the problem while the plant guys kept reminding him how many millions it was costing. That’ll make you build in reliability!
Great stories Karen. Good reminders that this is a mindset or behavior problem, not a process problem. If we don’t welcome insight into how things might fail when used, no tool or process will help you.
Great stories Karen. Good reminders that this is a mindset or behavior problem, not a process problem. If we don’t welcome insight into how things might fail when used, no tool or process will help you.
Great stories Karen. Good reminders that this is a mindset or behavior problem, not a process problem. If we don’t welcome insight into how things might fail when used, no tool or process will help you.
Not an apples-to-apples comparison but your story reminded me of one a previous Ops VP mentor shared with me to inform a challenge I was facing at the time.
“A prominent brand of washers and dryers was having costly difficulty with returned damage product. With a little scrutiny, it was determined to be coming almost totally from one location of a big box home improvement store.
Memos and admonishments began. The problem was not corrected. The feedback was escalated. Bigger and bigger players were involved. Edicts were issued; the threat of stopping business with this location was imminent.
Then, someone from the brand said – let’s go see {What seems automatic for some, comes harder to many. We would say they went to gemba}. Waiting and watching at the delivery docks when the washers and dryers arrived, the simple discovery was that the washers and dryers were unloaded by the same practices and same tools as many other products.
In fact, the use of a dolly designed for lifting cylinders (water heaters) was causing immediate damage to almost every cubical (washers and dryers) product. No one previously had bothered to observe.
The washer and dryer company purchased the correct dolly, made them available {we assume with visual controls and standardized work to sustain}, and for minimum investment avoided a catastrophic situation.”
The moral of the story –
Go to the process; talk to the people; embrace problems.
Not an apples-to-apples comparison but your story reminded me of one a previous Ops VP mentor shared with me to inform a challenge I was facing at the time.
“A prominent brand of washers and dryers was having costly difficulty with returned damage product. With a little scrutiny, it was determined to be coming almost totally from one location of a big box home improvement store.
Memos and admonishments began. The problem was not corrected. The feedback was escalated. Bigger and bigger players were involved. Edicts were issued; the threat of stopping business with this location was imminent.
Then, someone from the brand said – let’s go see {What seems automatic for some, comes harder to many. We would say they went to gemba}. Waiting and watching at the delivery docks when the washers and dryers arrived, the simple discovery was that the washers and dryers were unloaded by the same practices and same tools as many other products.
In fact, the use of a dolly designed for lifting cylinders (water heaters) was causing immediate damage to almost every cubical (washers and dryers) product. No one previously had bothered to observe.
The washer and dryer company purchased the correct dolly, made them available {we assume with visual controls and standardized work to sustain}, and for minimum investment avoided a catastrophic situation.”
The moral of the story –
Go to the process; talk to the people; embrace problems.
Not an apples-to-apples comparison but your story reminded me of one a previous Ops VP mentor shared with me to inform a challenge I was facing at the time.
“A prominent brand of washers and dryers was having costly difficulty with returned damage product. With a little scrutiny, it was determined to be coming almost totally from one location of a big box home improvement store.
Memos and admonishments began. The problem was not corrected. The feedback was escalated. Bigger and bigger players were involved. Edicts were issued; the threat of stopping business with this location was imminent.
Then, someone from the brand said – let’s go see {What seems automatic for some, comes harder to many. We would say they went to gemba}. Waiting and watching at the delivery docks when the washers and dryers arrived, the simple discovery was that the washers and dryers were unloaded by the same practices and same tools as many other products.
In fact, the use of a dolly designed for lifting cylinders (water heaters) was causing immediate damage to almost every cubical (washers and dryers) product. No one previously had bothered to observe.
The washer and dryer company purchased the correct dolly, made them available {we assume with visual controls and standardized work to sustain}, and for minimum investment avoided a catastrophic situation.”
The moral of the story –
Go to the process; talk to the people; embrace problems.
Great story with a great moral Scott. Thanks for sharing.
Great story with a great moral Scott. Thanks for sharing.
Great story with a great moral Scott. Thanks for sharing.