As marketing is often a science, all tests conducted with email marketing campaigns should follow scientific rules. Even if you are a novice to testing, these rules will help make your A/B test a success.

1. Become a “test advocate” in your organization: It is critical to have a test advocate who supports and defends test practices across the organization, as tests require discipline, resources and communication. Implementing a testing process often spans across several teams or groups, so it is also important that the test advocate communicate effectively cross-functionally.
2. Change is good! Time is always a key factor with any business initiative, and the same is true with testing. Tests require time to be conducted properly, but last minute changes will occur. Testing should help and not hinder your organization’s ability to execute effective marketing programs. Plan ahead and allow yourself a little more time to have the flexibility to postpone or extend a test.
3. The longer the test the better: On average 4 to 6 weeks in a row is enough to get a good response rate so the results are significant.
4. Avoid testing during times that are not “the norm”: Understand the seasonality of your customer base. A test during the holidays, for example, may not be as ideal as customers often behave differently during that period. If you sell cosmetics, and you launch tests in May, Mothers’ Day might be an external factor that could influence or skew your results.
5. Do not consider more than one factor in an A/B test: For the same reason as number four, above, make sure the differences you see in your results are driven from the different settings of your test. In a subject line testing for example, change only one word and avoid tweaking the language and the tone. This will help you rule out any other contributing factors to differences in your results.
6. Always have a control group: It is absolutely necessary to measure your test segment results against the control group – those who receive the “as usual” version of the test.
7. Always select random segments: Make sure the recipients in each segment are mixed randomly so test results are not impacted by the actual group composition. Gender, age, customer status, etc. can all factor into a results from a sample.
8. Sample size calculation is math: In order to make an appropriate decision, it must be based upon valid test results. At Experian CheetahMail, we have built a calculator to help our clients not only calculate the sample size needed, but also to check significance levels on results (to rule out the possibility that the results arose by chance).
9. Response rates influence the test significance: Depending on how large your sample size is, response rate gaps across your segments can influence testing significance. You may need to extend the test for a week or so to get enough responses to be statistically significant.
10. Testing should be an ongoing process: As marketing environments continuously evolve, any decision from a test result should be re-considered regularly. Consider an ongoing test plan and a test cycle. For example, once you have determined the best day of the week, you can start testing other factors. However, it’s a good practice to test the day of the week again either later in the year or one year after your initial tests tests.

Always follow a procedure.
Be prepared for flexibility!
Get results you can use.