A common testing mistake: A/B testing for multiple factors

In a previous blog post, we highlighted some key rules to consider when implementing a valid A/B test. The strength of A/B testing lies in its simplicity and in its ability to identify changes in performance across a singular factor. However, when a testing design becomes complicated by more than one factor, new methods need to be applied.

Marketers often still try and fit the A/B testing mold into situations where it is not well suited. Here, we will demonstrate one of the more common misuses of A/B testing and introduce a better option when testing with multiple factors.

Sequential factor testing

A common mistake when implementing A/B testing is when sequential A/B tests are performed in an effort to arrive at optimal levels for multiple factors. These experiments often start with the standard or status quo settings of the key factors to be tested. The levels of the factor that is believed to be the most responsible for performance are tested first, while the other factor levels remain constant. After the responses have been gathered and the optimal level for the first factor is determined, the factor regarded as the second most influential is tested next, with the ‘optimal’ first level factor remaining fixed for the rest of the experiment. This process continues to repeat itself until each factor level has been individually tested.

To better illustrate this flawed process, consider the following example with 2 factors, each with just 2 levels (the simplest case possible). Suppose an organization wants to determine the best image and ad copy to put in an email with the click-to-open ratio being the metric to maximize. We will denote the different images as I1 and I2 and the different ad copy as C1 and C2. In their first email blast to 20,000 customers, the company decides to send half of these customers an email with the combination (I1, C1) and the other half with the combination of (I1, C2). The results are as follows:

Click to Open %

(I1, C1) = 7.5%

(I1, C2) = 8.5%

Based on these results the company believes that Copy 2 is the preferred ad copy and fixes the next email blast to 10,000 more customers at this level so that the next test in the sequence will only vary the image level not already tested.

(I2, C2) = 9.5%

Seeing these results, the company decides that (I2, C2) is the optimal combination in terms of being able to generate the highest click to open ratio.

The problem is that the company may have missed out on finding the global optimum by not testing the fourth combination of (I2, C1) which may have yielded an even greater click to open ratio than 9.5%.

A/B tests assess one level of one factor versus the control group, but cannot measure the interaction effect across factors.

The reason why this method of sequentially testing one factor at a time fails to find the optimal factor levels is that an interaction effect exists between factors 1 and 2. Meaning, the factor effects are not additive, but rather the combinations among different factors and their levels produce an additional effect (interaction) when used simultaneously. By not being able to capture interaction effects, this sequential approach may miss the optimum altogether.

In situations such as these, it is more appropriate to perform a multivariate test (factorial test to be exact), where all factors are changed together and all combinations are accounted for. There is a wide array of different kinds of multivariate tests available, but when the number of factors and the number of factor levels to be tested are limited, the full factorial approach is the best option, as it retains the most amount of information about the factors.

The example here of 2 factors each with 2 levels is the simplest case, but the same reasoning applies when testing with more than 2 factors or more than 2 levels so think ahead and plan your testing strategies with the care they deserve. If you don’t, you may end up drawing the wrong conclusion about what approaches work best for your company’s marketing efforts.

For more information on testing, Experian CheetahMail’s strategic services team can assist you in choosing the best testing approach given your organization’s unique marketing challenges.