A frequently asked question by Six Sigma practitioners is how to select appropriate metrics for a particular organization or process. Considerable material has been written on this subject. This paper focuses on three aspects that should be of interest to Six Sigma practitioners:
- Why are metrics – and more importantly appropriate metrics – needed?
- A five-step procedure for selecting appropriate metrics for an application.
- Two cases – one simple and constructed, and the other real-life – to explain the selection of appropriate metrics.
This paper should be of use to Six Sigma practitioners beginning work in new organizations and/or on new projects. It should also be useful to those who have begun their Six Sigma work experience in a developed environment where appropriate metrics are already defined, and so they have not grasped the importance and nuances of the subject.
What Is a Metric?
Suppose today is Monday and person A has to meet person B at 10 a.m. the next day. Now suppose there are no watches, no measure of time (i.e., there is no concept of hours, minutes, seconds). How will A communicate to B?
A: “I want to meet you after the sun sets and rises once.”
B: “But how long after the sun rises?”
A: “Hmmmm, when the sun reaches about there…” pointing to an elevation in the sky.
As we would expect, they reach the meeting spot at widely differing times and one of them has to waste time waiting. So what is the problem causing this waste? A unit for measurement of time – day and night – exists between A and B. And yet the waste occurs.
Such a unit of measurement is called a metric (e.g., day for time, kilometer for distance and kilogram for weight are all metrics). But is the metric of day and night serving the needs of A and B? The obvious answer is no. So A and B need to define and agree a more appropriate metric.
Developing Appropriate Metrics
Think about having to solve A and B’s problem without the metrics of hour, minutes and seconds existing and the problem of developing appropriate metrics will become apparent. And this is a problem that most Six Sigma practitioners encounter as they begin work in a new company or a new problem within the same company. The five easy steps listed below will enable them to systematically arrive at the appropriate metrics.
Step 1 Why is the measurement required?
Step 2 What needs to be measured?
Step 3 What is the precision of measurement required?
Step 4 How will it be measured?
Step 5 What use will the measurement be put to? By whom?
These steps are explained using two cases.
Case 1: The problem of A and B outlined above expresses the problem using a simple example that will explain to all – Six Sigma practitioners and operating personnel alike – the issues involved and the methods used for their solution.
Case 2: A real-life case of selecting appropriate metrics encountered while introducing Six Sigma for a large company in the business of digitizing data is more complex. In this case the company had selected the theme of “Improving Quality Consistency” in its operations. When the work was begun the metric being used to measure quality was number of character errors expressed as a percentage of the total number of characters output in a day.
Step 1: Why is the measurement required?
Or to rephrase the question more precisely – why is a new metric required?
Case 1: To reduce the time wasted due to parties A and B not arriving at meetings at the same time.
Case 2: When asked to describe the current state of quality consistency, it was found that the organization was measuring the percentage of correct characters in sampled batches and quality was expressed as the average percentage of characters that were correct. This average was varying at different times. Everyone knew it was varying but no one knew how to measure variability. Thus, one was given figures like 99.95 percent, 99.952 percent, 99.992 percent, etc., for the daily averages of error percentages. The metric was not measuring the problem – inconsistency. Nor was it measuring process outputs. Instead, it was measuring snapshot average quality which led to sporadic knee-jerk problem solving whenever the readings were out of range.
Step 2: What needs to be measured?
Case 1: The obvious answer is that time needs to be measured more precisely so that waste can be avoided.
Case 2: Firstly, the problem relates to errors and therefore rather than measuring the correct characters the percentage of error characters should be measured. Thus, instead of stating quality as 99.92 percent correct it would be expressed much better as 0.05 percent, .048 percent and .008 percent defectives for the purpose of error control. These figures are a bit harder to handle than the figures given above but this aspect will be dealt with in Step 3.
Secondly and more importantly, the standard deviation of errors needs to be introduced as a measure of consistency of quality. This will directly measure the customer problem of inconsistent quality and therefore focus efforts to improve it.
Step 3: What is the precision of measurement required?
This may seem a trivial step but is actually very important. Engineers will recognize this as specifying the least count of a metric.
Case 1: To improve the accuracy of the measurement, a metric measuring smaller intervals of the variable time is required. How small? Suppose A and B decide to divide the day into 24 smaller metrics (i.e., select one hour as the unit). A and B will then agree to come at an appointed hour the next day. However, one may come at say 11 a.m. while the other might come any time between 11 a.m. and 12 p.m. and still claim the time is 11 a.m. The wait (of up to 59 minutes) would in many cases be still intolerably long.
So A&B would conclude they need a more precise metric. They could then decide to select a metric, which was 1/3600th of an hour (i.e., a second). In this case, the specification for a designated time might be 2100 seconds after 11 a.m. They would quickly realize that this unit was too small and waiting for say 300 seconds was not such a big deal. So perhaps they would come to 60 parts of the hour – a minute – as the appropriate metric.
It would be easy to measure, convenient to deal with, and easy to interpret. It should be appreciated that a year, century, month, week and millisecond are all time metrics and can be used to specify time – but try to specify the time of a meeting using any of these and you will quickly conclude that they are not appropriate for A and B’s requirement.
Case 2: When queried about the customer specifications, the reply was that the customer accepts a minimum accuracy in data of 99.995 percent. So a problem was felt whenever sample quality checks indicated batches/lots exceeding this standard. Typical figures in quality reports for successive monthly averages would read as follows:
99.998% OK
99.992% Failed
99.997% OK
99.991% Failed
If a staff member reported that the average quality had improved from 99.993 percent in the previous month to 99.996 percent in the next month, it was not possible to estimate the percentage improvement. Following step 2, when it was realized that measurement of average percentage of errors was more suitable for this application, the readings changed to:
.002% character errors
.008% character errors
.003% character errors
.009% character errors
The customer standard now became less than .005 percent errors permissible. Suddenly one could see that the average errors were 400% more in month 2 compared to month 1.
These decimals were inconvenient to handle, however. For example, a 50 percent improvement on .003 percent meant going to another decimal place – .0015 percent – and so on. It was then realized that if the errors were calculated in per million (ppm defectives), instead of percent, the readings would become much easier to handle as shown below:
20 ppm
80 ppm
30 ppm
90 ppm
And the customer standard on this scale became less than 50 ppm errors permissible. This became easy to handle, express and grasp. A far greater precision of measurement – from percent to per million – was warranted. The final metric selected was ppm errors instead of percent correct.
Step 4: How will it be measured?
Case 1: It is apparent that as A and B went from days to hours and minutes they had to develop measuring instruments that progressed from sundials to mechanical and then quartz watches. As atomic physicists worked with problems involving the need to measure precisely – they developed seconds, milliseconds and nanoseconds, and atomic clocks were required to measure them.
Case 2: The customer problem was “inconsistent quality” and the average of errors measured was not measuring this variable. So how was variability to be measured? The obvious answer is through standard deviation, but as is so often the case it was not obvious to the operating personnel. After training them in the concept and the calculation of sigma, this metric was introduced. As the first sigma readings came out and were interpreted a number of very important changes begun to take place.
The concepts of variability and controlling variability to satisfy the customer became clear. So the whole understanding of quality changed as manifested in the following statements before and after the introduction of the sigma metric:
Before: “The customer quality limit is < 50 ppm errors”
After: “We have to produce so that the (average + 3*sigma) of errors is < 50 ppm.”
Suddenly it dawned on everyone that the average would have to be much less than 50 ppm to ensure that the customer was satisfied despite the variability. And that even with this standard they would be out of specifications .3 percent of the time. Gradually the whole effort shifted to reducing sigma (i.e., variability).
In fact, the introduction of the sigma metric created a mindset change and all the customer lines were shifted to average and sigma control charts for both error and turnaround measurements.
Step 5: What use will the measurement be put to? By whom?
“What is not measured cannot be improved,” describes the tremendous emphasis that Six Sigma places on measurement. Yet it also cautions that measurements that are not used should be avoided. One of the reasons that a measurement might go unused is if the metric is inappropriate for the user. Among other reasons, this may be because it is not understood, does not serve a purpose, or is too cumbersome or complex to use.
Case 1: If the metric selected to measure time was seconds or years then A and B would not have adopted the metric for all the reasons listed above. The metric of minutes helped them and so they would adopt it readily.
Case 2: Both ppm and sigma metrics were readily adopted once the concept, calculation and utility were clearly understood by the personnel as it helped them in improving and controlling their key problem. In the case of ppm this was easy, but in the case of sigma a very simple explanation of the concept had to be developed and explained repeatedly to a number of personnel at all levels until it was internalized and its interpretation as familiar as that of “average.” Today it has become a way of life and most personnel wonder how they were working without it.
Conclusion
These examples are intended to help the inexperienced Six Sigma practitioner recognize the issues involved and the criticality of ensuring that the metrics selected and used at all levels of the organization are the appropriate ones. Fashioning appropriate metrics requires knowledge, experience and a lot of common sense. These examples can be used to explain to key operations personnel the concepts of metrics and why they might need to create new metrics or adjust current metrics.
