A successful Lean Six Sigma (LSS) project will produce either breakthrough performance – a shift in the mean or median – or variation reduction – a shift in the standard deviation or variation. How do you know your project has undergone a shift in the process that benefits the business? If there has been a shift, how can you be sure it did not happen by chance?

LSS project teams should know what the primary metric is for the business problem being addressed. The definition of the primary metric is key to focusing the team on the voice of the business and providing accurate statistical and financial results.

Example: The Search for a Primary Metric by XYZ Company

Consider the following example of the XYZ Company’s efforts to select a proper primary metric.

The XYZ Company is losing, at an alarming rate, sales of a chemical product as measured by percentage of market share. The company does not have a diversified product offering since it is a younger company – thus, loss of market share of a key product is essential to address. The primary complaint from customers about the product is the ability of the customer to empty the railcar that delivers the product.

Project Metric Options

Three proposals for a project metric were considered. The team knew from basic process knowledge that variation of the product’s viscosity, or ability to flow, affected the customer and was probably related to the customer’s ability to fully empty the railcars that contain XYZ’s product. One suggestion for the project metric was the market share of XYZ’s product in the market. Another suggestion was to consider the quality variation of the viscosity since the internal specifications showed a defect rate of 5 percent or 50,000 ppm. The third idea was to track the number of customer complaints received per month about railcars not fully unloading (a metric that was already being measured).

Evaluating the Options for Primary Project Metric

The least appropriate measure for the primary metric was the percentage of market share. Most market share analysis is infrequently determined – long delays can occur between changes in product delivery, impeding the measurement system’s ability to detect change.

Selecting the improvement of the viscosity’s quality defective rate as primary metric holds some risk. XYZ would be assuming reduced viscosity variation or a shift in average viscosity would alleviate the customers’ problem of unloading the railcars – the company’s “burning platform,” the crisis that threatens the viability of the whole business or local operation. By going with this metric, the team may not end up fighting the source of the fire on its burning platform. They may improve viscosity, but that may not satisfy the customer if the product still cannot be easily emptied from the railcars.

To address their customers’ concerns, the XYZ team should address the unloading of the product by using a weekly or monthly number of customer complaints from poor unloading. If the viscosity is the root cause for poor railcar unloading, then the customer will still be delighted once the problem is addressed. In addition, with the number of customer complaints about poorly unloaded railcars as the primary metric, then the team will most likely go to the customers’ unloading facilities to find the root cause for poor unloading. When the team visits customer sites, XYZ will learn where the problem is (or problems are) occurring and with what equipment. The team may find an easier, quicker solution than improving the variability or mean of the product viscosity – that is to say, other inputs exist that impact loading, and processes or equipment can use other inputs to solve a problem by improving robustness.

The lesson of the XYZ Company example is that it is imperative for a team to accurately select its primary metric. A team may focus on an assumed solution to a problem if a problem is scoped too tightly – and may not solve the actual problem. In XYZ’s case, the company also needs quick feedback on whether the problem has been solved.

Is Your Primary Metric the Right Metric?

Assuming you have chosen the metric that best describes the business problem, how do you determine that you have made a difference to the business? By following LSS steps, a team leader will have gathered historical data with the process owner to calculate a baseline, often an average of prior weeks’ or months’ data. A successful project should track the results of a project for a few months after improvement were implemented, and the mean or variation should alter during that time.

Some would argue that it is necessary to have three, six or 12 months of historical data and 12 months of post-project closure to verify that a lasting impact has been made to the company. That length of historical performance, however, cannot always be captured. Don’t let that be an impediment to moving forward with improvements. Consider a man who visits the emergency room with a major artery in his leg bleeding. He is not going to deny that he was “improved” by the doctors based on how long he was bleeding before he arrived at the hospital.

Understanding the value provided by a Lean Six Sigma deployment, culture or methodology is important (although few organizations use the same valuation techniques). But worrying about launching a project until the benefit can be quantified seems counterproductive – unless prioritization is the focus for estimating potential benefit. The estimation should not delay projects getting started. Analysis for sustainability is valid, however, to make sure that LSS practitioners are successful in sustaining change and that a system is not created for rewarding firefighters repeatedly fighting the same outbreaks.

The Hawthorne Effect

Some people may question if the changes made by a project were truly significant; others might argue the change was a fluke. Another explanation could be the result of the Hawthorne effect.

The Hawthorne effect has its origin in studies on employee productivity conducted by sociologist Elton Mayo in the late 1920s to the early 1930s. He studied how physical conditions affect productivity at a Western Electric Hawthorne Works facility near Chicago. Mayo changed various conditions in the facility – such as increasing lighting, changing the working hours and setting different break times – all with an effect of increased employee productivity. A surprising result occurred when lighting was decreased: productivity still increased. The conclusion of the series of experiments was that working conditions were not as impactful as the perception of employees that someone was interested in what they were doing and that someone cared about them.

In other words, observations, or measurements, can lead to a process change due simply to the process’s being monitored or managed differently. The term “Hawthorne effect” was coined by researcher Henry A. Landsberger in 1955. The Hawthorne effect is comparable to the Heisenberg uncertainty principle in particle physics, which states that one cannot know the precise position and momentum of a particle simultaneously because the act of measuring affects the particle.

If a project’s primary metric is tracked for one year following a project’s closure, sustainability should be unquestioned because most sources of unacceptable variation, whether common cause or special cause, would have surfaced during that time. The Hawthorne effect would not bear a sustaining effort because the process improvement team and its leadership are not often focused on monitoring the same area for one year after project closure.

It is said that simple habits are formed after 14 to 21 days of repetition. If a system has changed, then sustainability seems to be proven after a year of a process output change for a successful project. A simple two-sample t-test is typically used to show the statistical confidence of a process shift in mean.

Conclusion

LSS projects can be hampered by poor scoping and, consequently, the poor selection of metrics. Changes to process outputs must be impacted by addressing the root causes. Monitoring and measuring process results can support maintained shifts in processes. Sustainability over a year’s time should discredit those who say the process has not changed or argue that it has undergone the Hawthorne effect. It is also important to remember that sustainability cannot be discredited when, for example, a future process manager or owner dismisses the controls put in place for past successes.

Many projects have a primary metric along with secondary metrics, which are either metrics that keep a project honest to a business’ needs or another additional benefit. If these metrics were important enough to be measured during a project, they should also then be tracked for a period of time post-project.