When it comes to mastering design of experiments (DOE), many quality professionals experience a steep learning curve. DOE can be complex, and to fully understand and apply it, professionals need to learn when to use it, how to set up the experimental design and how to interpret their results.

Service quality professionals face an even greater challenge because they often cannot relate to the methods many instructors use to teach DOE.

Six Sigma trainers typically ask trainees to work through manufacturing-based scenarios such as identifying the ideal conditions for using a catapult to attack a castle, which can take up to a full training day or longer to execute. And because many of these Black Belts in training already believe they will never apply DOE, they do not want to spend the class time to learn it, and instructors are reluctant to make them. The challenge is to give Six Sigma trainees firsthand experience working through a designed experiment that addresses a relevant quality improvement problem and can be executed quickly. With the help of a macro (named the GO macro) written for a standard statistical software package (Minitab), here is a solution.

Training Begins with DOE Outline

DOE training for Black Belts should begin by following the outline for designing an experiment suggested by D.C. Montgomery in his book Design and Analysis of Experiments:

1. Recognition of and statement of the problem.
2. Choice of factors, levels, and ranges.*
3. Selection of the response variable.*
4. Choice of experimental design.
5. Performing the experiment.
6. Statistical analysis of the data.
7. Conclusions and recommendations.
   (*In practice, Steps 2 and 3 are often done simultaneously or in reverse order.)

Also all essential DOE topics need to be covered, such as the scientific method, one factor at a time experimental approach, DOE terminology (e.g., factors, levels and response variables), screening designs, resolution, randomization, replication, analysis of variance (ANOVA), residuals, F tests, main effects and interaction plots.

The most critical part of the training, however, is to give trainees the opportunity to apply these concepts firsthand using a DOE simulation.

To accomplish this, the group of trainees should be divided into teams and given a problem statement. For example, at PricewaterhouseCoopers, one of the largest professional services firms, the problem statement that has been used is: “A professional services company has a new tax preparation service they developed for people who are 55 years or older and have ‘wealth accounts.’ The company wants you to design an experiment that will identify the direct marketing process that maximizes the number of customers they can acquire.”

Applying Concepts in DOE Simulation

Trainees are told that the response variable is the number of customers who purchase the service, then given the seven potential factors and their corresponding levels in the table below:

The trainees’ task is to choose which factors they think may affect the number of acquired customers and determine the corresponding factor levels that they want to test. Following these steps, they are ready to choose their experimental design.

The parameters for the DOE include a budget of $40,000 and experimental runs of 100 calling hours that cost $2,000 each. Based on these parameters, there are several full factorial and fractional factorial designs that could be run within the allotted budget. For example, some students may decide there are only four potentially important factors and setup a 2⁴ full factorial. Another group may decide they need to explore all seven factors and setup a 2^7-3 fractional factorial.

The Importance Is Learning about DOE

After the trainees create their experimental design, they are ready to perform the experiment. Because it is of greater valuable to spend more time learning about DOE than actually running the experiment, here is where the macro was employed to generate the response data.

The data collection process begins with the FORM worksheet in Figure 1.

Trainees input the factor level settings for their first run into the Condition column. If they have excluded a factor(s) from their design because they determined it was not necessary to measure it, they enter an ‘x’ for that factor(s).

After they have entered the levels for all factors, they run the macro to generate the number of corresponding customers the run would acquire after 100 calling hours. As illustrated in Figure 2, the macro:

• Outputs the response value to the Customers column.
• Transposes the specified factor levels into the Price, Mailtime, Script, Training, Gender, Age and Day columns.
• Tracks the cumulative amount of money spent on the runs in the $Spent column (Figure 2).

Macro Makes Running Experiment Easier

Trainees then repeat this process for each run of their design by entering new factor level settings into the Condition column and running the macro to generate the corresponding number of customers acquired. The macro adds the information for each run in a new row in the worksheet, and within minutes the trainees have “performed their experiments” and the data they need to evaluate their designs is ready for them to analyze (Figure 3). Columns C4 through C10 list the factor levels for each run, and C11 identifies the run number. C3 lists the number of customers acquired and C12 presents a running total of the money spent to perform the experiment.

Once the teams have conducted their analyses, they can present their findings to the entire group. The graphs they produce with their statistical software package are used to explain what factors they think significantly impacted the sales process and why (Figure 4). The learning continues as trainees discuss how they chose their experimental designs, whether they omitted important factors from the design and selected reasonable levels, and the possibility of confirmatory runs, if the budget allowed for them.

This approach for teaching DOE follows Montgomery’s outline and makes the process engaging for trainees. All of them understand the need to design an effective sales process, which makes it easier for them to relate to and retain the material. And the macro gives them meaningful data to work with in far less time than it would take to run a typical experiment, which leaves more time to focus on the practice of DOE itself.

Feedback on this training has been positive, with trainees frequently commenting on how the simulation brought DOE to life for them. And the benefits seem to endure. Many who entered this part of their training dismayed at having to learn something they thought they would never use, later report they are beginning to plan their own DOEs – a response that Master Black Belts everywhere can appreciate.

The authors acknowledge the help of Michelle Paret and Tony Coray, both of Minitab, on the GO macro and this article.