Hypothesis testing is a branch of statistics in which, using data from a sample, an inference is made about a population parameter or a population probability distribution. 

Let’s look at the purpose of hypothesis testing, the underlying statistical concepts, the importance of using hypothesis testing, and some best practices for optimizing the value of the technique.

Overview: What is hypothesis testing? 

Hypothesis testing in statistics is a way for you to test the results of an experiment to see if you have meaningful results. The outcome of your hypothesis test will tell you whether your results are caused by pure chance or not. If your results are due to pure chance, then you will have a difficult time replicating or repeating them. But, if your results were due to something significant happening, you can use that knowledge to repeat the results.

Hypothesis testing applies to all forms of statistical inquiry. For example, it can be used to determine whether there are differences between population parameters or to gain an understanding about slopes of regression lines or equality of probability distributions.

The process for completing a hypothesis test

First, a hypothesis statement and assumption is made about the population parameter or probability distribution. This initial statement is called the null hypothesis and is denoted by Ho. An alternative or alternate hypothesis (denoted with Ha), is then stated, which will be the opposite of the null hypothesis. 

The hypothesis testing process and analysis involves using sample data to determine whether or not you can be statistically confident that you can reject the Ho. If the Ho is rejected, the statistical conclusion is that the alternative or alternate hypothesis Ha is true.

Since you’re using sample data to make your inferences about the population, there’s a risk that you’ll make an error. You can choose the level of risk you’re willing to accept for this error. In the case of the null hypothesis, we can make one of two errors.

1. Type 1, or alpha error: An alpha error is when you mistakenly reject the null and believe that something significant happened when it did not. In other words, you might believe that the means of the two populations are different when they aren’t. You can pick your risk of being wrong to be 1%, 5%, 10%, or any other reasonable value.
2. Type 2, or beta error: A beta error is when you fail to reject the null when you should have. In this case, you missed something significant and failed to take action. This risk typically runs in the range of 80% to 90%, or again, any other reasonable value.

The next step would be to determine the appropriate sample size you would need to be able to rely on the statistical validity of your results. Then, go out and collect your data.

You will then need to decide what is the appropriate statistical test for the type of data and hypothesis you’re testing. Using a statistical software package, you’ll get a multitude of numerical results. One you can focus on is something called the p-value. The p-value is your calculated risk of making an alpha error, or claiming success when there really wasn’t any. You will likely require the advice and guidance from a Lean Six Sigma Black Belt or Master Black Belt to sort all of this out.

Finally, you’ll need to make a decision to reject or not reject the null hypothesis. Your decision is made by comparing the alpha value (your assumed risk) with the p-value (actual risk). If the actual risk is lower than your assumed risk, you can feel comfortable in rejecting the null and claiming something has happened. But if the actual risk is higher than your assumed risk, you will be taking a bigger risk than you want by rejecting the null.

3 benefits of hypothesis testing

Hypothesis testing is a statistical technique for making decisions about whether the changes in your process are a result of chance or a result of some improvements you made. Here’s why you want to use hypothesis testing.

1. Statistical assurance of determining differences between population parameters

Just looking at the mathematical difference between the means of two samples and making a decision is woefully inadequate. By statistically testing the null hypothesis, you will have more confidence in any inferences you want to make about populations based on your samples.

2. Statistically-based estimation of the probability of a population distribution

Many statistical tests require assumptions of specific distributions. Many of these tests assume that the population follows the normal distribution. If it doesn’t, the test may be invalid.  

3. Assess the strength of your conclusions as to what to do with the null hypothesis

Hypothesis testing calculations will provide some relative strength to your decisions as to whether you reject or fail to reject the null hypothesis.

Why is hypothesis testing important to understand?

The interpretation of the statistics relative to the null hypothesis is what’s important.

Properly write the null hypothesis to properly capture what you are seeking to prove

The null is always written in the same format. That is, the lack of difference or some other condition. The alternative hypothesis can be written in three formats depending on what you want to prove. 

Frame your statement and select an appropriate alpha risk

You don’t want to place too big of a hurdle or burden on your decision-making relative to action on the null hypothesis by selecting an alpha value that is too high or too low.

There are decision errors when deciding on how to respond to the null hypothesis

Since your decision relative to rejecting or not rejecting the null is based on statistical calculations, it’s important to understand how that decision works. 

An industry example of using hypothesis testing

The new director of marketing just completed the rollout of a new marketing campaign targeting the Hispanic market. Early indications showed that the campaign was successful in increasing sales in the Hispanic market. 

He came to that conclusion by comparing a sample of sales prior to the campaign and current sales after implementation of the campaign. He was anxious to proudly tell his boss how successful the campaign was. But, he decided to first check with his Lean Six Sigma Black Belt to see whether she agreed with his conclusion.

The Black Belt first asked the director his tolerance for risk of being wrong by telling the boss the campaign was successful when in fact, it wasn’t. That was the alpha value. The Director picked 5% since he was new and didn’t want to make a false claim so early in his career. He also picked 20% as his beta value.  

When the Black Belt was done analyzing the data, she found out that the p-value was 15%.  That meant if the director told the VP the campaign worked, there was a 15% chance he would be wrong and that the campaign probably needed some revising. Since he was only willing to be wrong 5% of the time, the decision was to not reject the null since his 5% assumed risk was less than the 15% actual risk.

3 best practices when thinking about hypothesis testing 

Using hypothesis testing to help make better data-driven decisions requires that you properly adhere to the testing procedure. 

1. Always use the proper nomenclature when stating the null and alternative hypotheses 

The null will always be in the form of decisions regarding the population, not the sample. For example, the null and alternate hypotheses for testing the difference between two means is written as:

Note that you use mu (population average) as the nomenclature for the mean, not Xbar (sample average). You are not interested in whether the sample means are different since you already have those calculations. You are interested in if the means of the two populations the samples came from are different. 

2. Select an appropriate sample size

The statistics and final results will depend on you determining the appropriate sample size, and then collecting the data in an appropriate manner. Use statistics to determine sample size, not intuition. 

3. Pick a reasonable alpha risk so you’re not always failing to reject the null hypothesis

Being too cautious will lead you to make beta errors, and you’ll never learn anything about your population data. 

Frequently Asked Questions (FAQ) about hypothesis testing

What form should the null hypothesis be written in?

The null hypothesis should always be in the form of no difference, or zero, and always refer to the state of the population, not the sample. 

What is an alpha error? 

An alpha error, or Type 1 error, is rejecting the null hypothesis and claiming a significant event has occurred when, in fact, that is not true and the null should not have been rejected.

How do I use the alpha error and p-value to decide on what decision I should make about the null hypothesis?

The most common way of answering this is, if the p-value is low (less than the alpha), the Null should be rejected. If the p-value is high (greater than the alpha), then the null should not be rejected.

Summing up hypothesis testing

Hypothesis testing is a powerful statistical technique for confirming whether something is as you believe it is. You first state your basic or null hypothesis followed by your alternative one. You decide on the risk you are willing to take of being wrong. Collect your data, and use the appropriate statistical test based on your statistical question and type of data. Interpret your results, and draw a conclusion as to whether your null hypothesis was correct or not.