A number of improvement projects are hampered because of lack of quality data. Engineers tend to overlook the importance of validating the measurement system, and even more importantly often times don’t select the right measurement instrument. During one such project, a company team was measuring the tube width with a vernier caliper. The specification width was 0.1 mm where as the least count of the caliper was 0.01 mm (based on the AIAG guidelines). After collecting the data for a week and analyzing the observations, the team was surprised by the results.

The normality test showed the results shown in the figure above. Even though the points were sticking close to the normal probability line, p value (0.025) indicated non-normality. Further probing through the histogram revealed that data was in fact following a normal distribution. The reason behind this observation turned out to be that the instrument was not sensitive enough to capture the small variation. Data points were clustered in groups and hence the p value came out to be low.

This shows that if your process has very low variation, it is preferable to consider the total variation for selecting the least count for the measuring instrument. This brings us to the contention that merely comparing the least count with the tolerance can be misleading as it is in the case of a repeatability and reproducibility study. For those working on improvement, if the team is looking for ‘x’ amount of change in the measure, the measurement scale should then at least have the capability to identify ‘x/5’ amount of change in measure irrespective of the specification width.