Making a decision based on sample data always has an inherent risk of making the wrong decision. We will explore the concept of beta risk as it applies to doing hypothesis testing asl well as how you can reduce the risk and the consequences of being wrong.

Overview: What is beta risk? 

You have made a change in your process. Now you must determine whether there actually was a statistically significant change. You collect data and now must make a decision of whether there was change or not. 

Hypothesis testing is the tool you would normally use.

This tool requires you to state two hypothesis statements. The first, called the null hypothesis, states that there has been no statistically significant change in the process. The second hypothesis, called the alternate or alternative hypothesis, states there was a meaningful change. 

Once you have done the proper calculations, you must decide whether to reject the null and accept the alternate, concluding there was change. Or you can decide not to reject the null and conclude there was no change.

In both cases, you have a risk of making the wrong decision. 

• Alpha risk: The risk of claiming there was change when there wasn’t is called an alpha, or type 1 risk or error. 
• Beta risk: Failing to conclude there was a change when there was is called a beta, or type 2 risk or error. 

In the case of beta risk, you run the risks and consequences of inaction. For example, when interpreting the results of your hypothesis test, you conclude the new marketing program was ineffective and did not increase sales. 

Unfortunately, the truth was that it did increase sales. You now have an opportunity cost associated with failing to implement the new marketing program. 

Once you discover your mistake, you may have a short-term cost, but nothing prevents you from implementing it in the future – although you will have lost potential sales, and your competition may have penetrated and stolen some of your market share.

The good news is, you can set your beta risk beforehand based on your risk profile. Since there is possible recovery from your bad decision, the beta risk is typically set between 15%-25%. This is the assumed risk you are willing to take for failing to reject the null and claim change when you should have.

An industry example of beta risk 

During clinical trials of a new drug, a pharmaceutical company statistician did a hypothesis test to determine whether the new drug was more effective than the existing drug being used. This was done early in the trials, when there was limited data. As a result, the statistician concluded that the new drug was not more effective than the old drug and recommended the trials be stopped. 

As more data came in from ongoing field studies, the outcomes of subsequent hypothesis testing clearly showed the new drug to have more efficacy. Unfortunately, because of the delay, a competitor was able to bring a similar product to market, and the company suffered significant lost opportunities because of their beta error. 

Frequently Asked Questions (FAQ) about beta risk

1. What is beta risk? 

A beta risk is failing to reject the null hypothesis when you should have. It is erroneously determining there was no change in the process when there was.

2. How is beta risk related to the concept of power? 

Power is defined as 1.0 minus beta. Power is the probability of seeing a change if there is one. If we assume beta to be 0.20, or 20%, power would be calculated as 0.80, or 80%. Specifically, your hypothesis test has an 80% chance of seeing a change if there really is one. This is typically a minimally accepted level of risk. 

3. How is beta risk different from alpha risk? 

Beta risk is failing to reject the null hypothesis when you should have rejected it. That is, claiming nothing happened when it did. Alpha risk is rejecting the null hypothesis and claiming change when there was none.