Designed experimentation is the gold standard in proving causal relationships between factors and a response of interest, as well as quantifying those relationships. While some of these can seem quite complicated and specific to unique situations, the most basic is also often the most powerful and efficient: the balanced experiment. This involves “balance” between the factors by ensuring that the set points of each occur the same number of times for any set point of the other factors.
Overview: What are balanced experiments?
In a controlled experiment, the set points, or levels, of each factor are determined ahead of time, and the experiment is designed to be efficient and effective at quantifying the relationship between those factors and any responses or outputs of interest.
While any number of levels can be utilized, most commonly two are used — a “high” and a “low” value. In a balanced experiment, the number of times the high or low value for any factor occurs is exactly the same when any other factor is at its high or low value.
For example, consider the following balanced experiment with three factors: A, B, and C:
| Run | Factor A | Factor B | Factor C |
| 1 | Low | Low | Low |
| 2 | Low | Low | High |
| 3 | Low | High | Low |
| 4 | Low | High | High |
| 5 | High | Low | Low |
| 6 | High | Low | High |
| 7 | High | High | Low |
| 8 | High | High | High |
Examining either the low- or high-level runs for Factor A, you can see that both Factors B and C have two low runs and two high runs within that level of Factor A. The same can be said for any factor. Thus, this experiment is considered balanced.
3 benefits of balanced experiments
There are many advantages to balanced experiments, which is why they remain very popular even as research expands the number of standard experimental designs available to practitioners.
1. Simplicity
While numerous types of experimental designs exist and were developed to provide the most optimal experiment for very specific situations, these often are complex to teach to practitioners. Balanced experiments are typically very simple designs that intuitively make sense and can be taught to anyone quickly, especially factorial designs.
2. Independence
While not the only designs that achieve this, balanced designs result in the factors being independent of one another. In statistical terms, there is no correlation between factors, and the condition is referred to as orthogonality. Aside from improving the estimation of the mathematical relationship between each factor and the response, this also means as the model is reduced by removing factors, the estimated relationship with the remaining factors does not change.
3. Efficiency
Many, but not all, balanced experiments are very efficient at estimating the relationship between the inputs and the output. In other words, this relationship can be determined with a small number of experimental runs. One popular example is the fractional factorial design.
However, efficiency cannot be assumed just because an experiment is balanced; a general full factorial experiment where factors have more than two levels may be quite inefficient.
Why are balanced experiments important to understand?
Traditional data collection, based on simple observation of a process, is inefficient and often results in inaccurate models that do not result in an optimized process.
DOE versus OFAT
One-factor-at-a-time analysis, where each input is analyzed for a relationship with the output, lacks the ability to identify and estimate interactions between multiple inputs and is less powerful at identifying statistically significant relationships. Designed experiments, and specifically balanced experiments selected to estimate interactions of interest, do not have this drawback.
Factor independence
Dependence between factors, where two or more are correlated with one another, is known as multicollinearity, and it can result in not identifying important relationships as well as inaccurately estimating them.
By utilizing a balanced experiment, there is complete independence between factors and the relationship between each and the output cannot be hidden or impacted by another.
An industry example of balanced experiments
In designing an email marketing campaign, a company wants to understand how each of three different subject lines, two different banner styles, and three different highlighted promotions affect click-through and conversion.
Before deciding on one email to send to their customer base, a subset is selected for a trial where different combinations will be utilized for randomly selected customers, and the best combination then used for all other customers.
A balanced experiment is utilized with the following design:
| Test Group | Subject | Banner | Promo |
| 1 | A | 1 | A |
| 2 | A | 1 | B |
| 3 | A | 1 | C |
| 4 | A | 2 | A |
| 5 | A | 2 | B |
| 6 | A | 2 | C |
| 7 | B | 1 | A |
| 8 | B | 1 | B |
| 9 | B | 1 | C |
| 10 | B | 2 | A |
| 11 | B | 2 | B |
| 12 | B | 2 | C |
| 13 | C | 1 | A |
| 14 | C | 1 | B |
| 15 | C | 1 | C |
| 16 | C | 2 | A |
| 17 | C | 2 | B |
| 18 | C | 2 | C |
By utilizing this experiment, the company is able to determine how each level of each factor performs relative to the other levels of that factor and do so independently of the other factors. In other words, within each level of a factor, the remaining factors utilize the exact same combinations of levels of the remaining factors so the estimated effect of that factor is not influenced by the others.
3 best practices when thinking about balanced experiments
While balanced experiments are powerful and result in some desirable properties, they are not perfect, and certain best practices can be helpful in avoiding pitfalls.
1. Don’t assume full factorial
A design in which every combination of factor levels is utilized is known as a full factorial experiment. While this results in information about every factor and every interaction among factors, often, these are not all needed and the experiment can be larger than necessary with little benefit. By considering fractional factorial or other smaller designs that still achieve the experimental goals, resources can be saved.
2. Consider more efficient designs
In some cases, experimental runs are extremely expensive. Depending on the goals of the experiment, utilizing a more efficient design may be desirable even if doing so results in losing independence among factors or simplicity of design.
3. Don’t sweat missing runs
Often when running a designed experiment, one or more runs may be “botched” or result in missing data for a variety of reasons. While sometimes this will prevent you from estimating the impact of one or more factors, in many others, the only negative impact is a moderate loss of independence among factors.
While this is less desirable than full independence, typically the impact is not practically significant, and it is better to simply analyze the results rather than repeating the experiment.
Frequently Asked Questions (FAQ) about balanced experiments
1. Do balanced experiments use all combinations of factor levels?
While an experiment in which all factor level combinations are utilized, known as a full factorial experiment, is a balanced experiment, there are numerous balanced experiments that do not meet this criteria. For example, fractional factorial designs are all balanced but by definition do not include all combinations of factor levels.
2. How do I create a balanced experiment?
Essentially all commercial statistical software that includes designed experiments will create balanced experiments, including various factorial designs.
3. What are some examples of balanced experiments?
Factorial designs, including both full factorial designs and fractional factorial designs, are the most common balanced experiments.
The gold standard
Balanced experiments remain the gold standard for designed experiments given their simplicity, independence between factors, and potential efficiency. While other experimental designs may sometimes provide greater efficiency or other desirable properties, in many (if not most) cases, a well-chosen balanced design is an excellent choice and will not fail you.
