Key Points
- The margin of error allows you to use smaller sample sizes when conducting customer surveys.
- The measurement directly relates to the size of your sample, with larger sample sizes yielding more accurate results.
- You have to exercise some caution, while this tool is useful, it can lead to mistakes when used incorrectly.
Do you know how to account for the margin of error in your customer surveys?
A survey is a valuable assessment tool in which a sample and information relate to a larger population. Surveying is like tasting a soup, a few spoonfuls tell what the whole pot tastes like.
An Example of Margin of Error and Confidence Levels
The key to the validity of any survey is randomness. Just as the soup needs stirring for the few spoonfuls to represent the whole pot, when sampling a population, the group needs priming before you select a respondent. Respondents are chosen randomly so that the survey results apply to the whole population.
How well the sample represents the population relies on two important statistics – the survey’s margin of error and confidence level. They tell us how well the spoonfuls represent the entire pot. For example, a survey may have a margin of error of plus or minus 3% at a 95% level of confidence.
These terms simply mean that if the survey were conducted 100 times, the data would be within a certain number of percentage points above or below the percentage reported in 95 of the 100 surveys.
Company X surveys customers and finds that 50% of the respondents say its customer service is “very good.” The confidence level is 95% plus or minus 3%. This information means that if the survey were conducted 100 times, the percentage who say service is “very good” would range between 47 and 53% most (95%) of the time.
| Survey Sample Size | Margin of Error |
|---|---|
| 2,000 | 2% |
| 1,500 | 3% |
| 1,000 | 3% |
| 900 | 3% |
| 800 | 3% |
| 700 | 4% |
| 600 | 4% |
| 500 | 4% |
| 400 | 5% |
| 300 | 6% |
| 200 | 7% |
| 100 | 10% |
| 50 | 14% |
Sample Size and the Margin of Error
The margin of error, the plus or minus 3% in the above example, decreases as the sample size increases. This is only to a point. A very small sample, such as 50 respondents, has about a 14 % margin of error.
A sample of 1,000 has a margin of error of 3%. The size of the population does not matter. This statement assumes that the population is larger than the sample.
There are, however, diminishing returns. By doubling the sample to 2,000, the margin of error only decreases from plus or minus 3% to plus or minus 2%. Although a 95% level of confidence is an industry standard, a 90% level may suffice in some instances.
A 90% level is easily attained with a smaller sample, which usually translates into a less expensive survey. A 3% margin of error at a 90% level of confidence requires a sample size of about 750. For a 95% level of confidence, the sample size would be about 1,000.
Determining the margin of error at various levels of confidence is easy. Although the statistical calculation is relatively simple, the margin of error is readily found in the above figure.
A few websites also calculate the sample size needed to obtain a specific margin of error. Thus, if the researcher can only tolerate a margin of error of 3%, the calculator will say what the sample size should be.
Calculating Margin of Error for Individual Questions
Margins of error typically are calculated for surveys overall but also should be calculated again when a subgroup of the sample is considered. Some surveys do not require every respondent to receive every question. Sometimes only certain demographic groups need analyzing.
If only those who say customer service is “bad” or “very bad” are asked a follow-up question as to why, the margin of error for that follow-up question increases due to the respondents being smaller than the overall survey sample. Similarly, if results from only female respondents are analyzed, the margin of error will be higher. That is assuming females are a subgroup of the population.
Survey Data Is Imprecise
The margin of error reveals the imprecision inherent in survey data. Survey data provide a range, not a specific number.
A researcher surveying customers every six months to understand whether customer service is improving may see the percentage of respondents who say it is “very good” go from 50% in one period to 47% in the next six-month period.
Both are accurate because they fall within the margin of error. The decrease is not statistically significant. On the other hand, if those percentages go from 50% to 54%, the conclusion is that there is an increase in those who say service is “very good” albeit a small one.
Why It Matters
Understanding customer feedback is a valuable tool in any industry. However, you likely don’t have the resources or time to survey all of your customers. By having a margin of error and confidence level given for a sample size, you have a way to effectively gauge how you are meeting customer needs. This is invaluable, especially if you have a customer base in the millions.
The Dark Side of Confidence Levels
A 95% level of confidence means that 5% of the surveys will be off the wall with numbers that do not make much sense. Therefore, if 100 surveys are conducted using the same customer service question, five of them will provide somewhat wacky results.
Normally researchers do not worry about this 5%. They are not repeating the same question over and over. Odds are that they will obtain results among the 95%.
However, if the same question is asked repeatedly in a tracking study, researchers should beware of unexpected numbers that seem way out of line.
For example, customers are asked the same question about customer service every week for months.
“Very good” is selected each time by 50%, 54%, 52%, 49%, 50%, and so on. If 20% surfaces in another period. 48% follows in the next period, it is probably safe to assume the 20% is part of the “wacky” 5%, assuming proper methodology is followed.
Other Useful Tools and Concepts
Data has a ton of tools associated with it. Since Six Sigma is data-driven by design, you’re going to be seeing a lot of it while you navigate the analysis of any process. As such, I heavily recommend reading up on why it’s important to know your sample size.
Additionally, learning about the benefits of random sampling is a net benefit. If you’re surveying customers and only selecting those who are avid appreciators of your work, something is going wrong. We detail how to avoid this pitfall in our comprehensive guide.
Conclusion
Understanding confidence levels and the margin of error in your surveys can lead to a better understanding of your customer needs. However, as with all data you collect in the analysis stage, you have to consider everything. Don’t fall into a trap around confidence levels, account for everything.
