Testing your online properties? Focus on the user

Data-driven marketing is the Holy Grail for success in marketing today. No one is happier than a marketer who has been given the OK to test, fail and iterate based on real-time data. Add in the velocity and volume of big data and you can’t fail! Or can you?

As more consumers move online to digital channels, marketers are relying more on A/B testing to collect user interaction data and make decisions. Based on how individuals react to a given change online, marketers can make adjustments that will yield the best conversion results.

The problem we are seeing is that companies are acting too quickly to action their A/B test results, without regard to given business goals and objectives. When marketers move on results too quickly or without a plan, this introduces scope creep, or uncontrolled changes that are outside of the original project plan.

While marketers need to be agile, they need to make sure they have a coordinated strategy that does not sacrifice the long term product and company vision. By making changes too quickly, they may adjust an online property without appropriate review.

Secondarily, too many tests introduce a fragmented experience for users. Often, companies will conduct A/B tests based on individual hits rather than considering each user as unique. This means someone may hit an online property one time and have one user experience, come back and see a totally different site.

Techcrunch recently pointed out Twitter’s “new” user interface which differed session to session to measure individual’s user interactions. You can note in the photo below the introduction of sharing icons underneath the posts on the left side.


As a marketer, A/B tests have value in their aggregated data results; but as the photo shows, you can’t forget the data represents individual’s online experience. By fragmenting the individual, you jeopardize user experience – a fatal mistake within a web-driven business.

Now, as a web marketer, I’m not saying you shouldn’t A/B test — they are very valuable and yield measurable results that can make a real difference. But, as you A/B test, you need to ensure two things:

  • You maintain the integrity of every user’s experience
  • The variables you’re testing (and proposing to change) align with your product and overall company strategy

As we know, “with great data comes great responsibility.” Just because you can test something, doesn’t mean you should … at least not without aligning the outcome to your business strategy.