Introducing EmaiLO, a new email marketing rating system

Long before I ever analyzed a single line of data for Experian or one of our clients, I was a sports fan.  As a lifelong nerd, it just makes a certain kind of sense that my fandom is largely driven by stats.  Winning percentage, passer rating, points per game – these were my beacons, the foundations to any great argument.

As I grew older, my love for stats grew from basic to more advanced analysis.  Sabermetrics, expectation models, and new methods for true evaluation were my muses, revealing deeper, formerly unknowable truths.  Since I’m both a self-described nerd and (sadly) a very much observed one, it was only a matter of time before I started performing advanced analysis on my fantasy football teams… and viola, my life as an analyst was born.

Though I moved into the marketing analytics world, my love for advanced sports stats has never abated, and soon enough, I began experimenting with some of the techniques from sabermetrics on marketing data.  Some have been successful, some of helped me create wholly different analytical techniques, and many have been abject failures.  Above all of them, one thing I’ve really been curious about was adapting the ELO rating system.  ELO was originally a system developed by Arphad Elo to rank chess competitors against one another, but it has since been adapted to soccer, the NFL, the NBA, and the MLB.  ELO’s main benefit is that it’s fairly straightforward, simple in the sense that it doesn’t require many inputs, and can be used to “objectively” rank competitors (and as an extension, predict how likely one is to best another in competition).  Those three features got me thinking… how can I adapt this for marketing?  How can I effectively rank marketing program strength based on a few inputs, in a manner that doesn’t share propriety information, and predict what likely metrics would be on a global scale?

And with that, I’m proud to introduce you to EmaiLO, our adaptation of the ELO rating system built specifically for email marketing!

What follows is a lengthy examination of what EmaiLO can and can’t do, its formulations, and why it has us excited.  Skip down for the gory details.  But to put it simply – we’re excited for EmaiLO because it gives us a high-level method for ranking our clients’ email marketing programs, and even more importantly, measuring if they performed better or worse than expected.  Since EmaiLO is a composite measure, it’s able to provide a more holistic view of email marketing performance, and since it essentially pits a marketer against themselves (more on that in the details), it allows us to provide a better sense of true benchmark performance.

In the upcoming days and months, we’ll be publishing some of the results of our EmaiLO efforts, showing how the new email marketing rating model works and highlighting clients that are doing well and why.  We also plan to use the model results to inform a broader commentary on the industry.  And since EmaiLO is so new, we might make some tweaks – anything that’s major, we’ll let you know.

Until then, ask your account representatives for more information, or shoot me a note on Twitter @davisj2007 (and read on…if you dare!)

The gory details, presented in a nod to FiveThirtyEight (and by extension, the New York Times) as a question and answer.

What is EmaiLO, exactly?

EmaiLO is our new adaptation of the ELO rating system, which is used to rank and forecast competitive events.  We’ve applied a similar methodology to measure the health of an email marketing campaign, and give a broad simple forecast for next month’s email performance (and to track performance vs expectation).

But if it’s for competitive events, how does that work for email?

Simple!  The biggest “competitor” for an email marketer isn’t really another email marketer, but their own historical email performance.  We’ve developed a process to scale the industry average performance in a way that says “based on global trends, you need to compete with yourself at x rate.”

That sounds hard, is there a lot of math?

While I can’t give up the precise details of how we scale the global trend to become the marketer’s “competitor,” I can talk about how ELO works itself.  To keep it as basic as possible, we use the logistic curve to produce an “expected score,” or in our case, an expected unique open rate, expected unique click rate, and expected session (total click) rate.   The logistic curve is expressed as

EmailLO, email marketing rating

where QA = 10RA /400, Q= 101500/(400m), m = our measurement of the industry average, and S = our proprietary scaler. R= our starting EmaiLO score. Every brand starts with an EmaiLO score of 1500 because… well, that’s what Arphad Elo decided. Our model runs from January 2013 through present, and if a brand became a client after January 2013, their first month starts at 1500.

I’m scared.

Don’t be – it’s really not that tough!  Besides, we did all of the math already.

Okay, so what does that funky equation give us?

Well, its most important output is an expected rate.  Since we run this equation with slightly different scalers for unique open rate, unique click rate, and session rate, we’re able to produce the expected KPI the brand should see next month… without the volume inputs we usually need.

Can you give us an example?

Sure thing.  This is a real example, but I’ll obscure the m and S factors (and the client name) so I don’t get in trouble.

Let’s say we have a brand, let’s call it Jake’s ELO Model Agency, and this brand had EmaiLO scores for open rate, click rate, and session rate of 1424, 1404, and 1389, respectively.  We’d normally blend these scores (1406) for most of our purposes, but let’s keep them separate for this exercise.  So we plug those three numbers as into our equations, and the output would look like 13.9%, 1.7%, and 2.3%, or our expected open, click, and session rates.  Make sense?

Sure – now what?

Well, here’s the fun part.  We can compare the expected rates to the actual outcomes one month later.  And check it out!  Jake’s ELO Model Agency actual email marketing rating performance is as follows: 13.4%, 1.7%, & 2.4% open, click, and session rates.  Pretty close, right?

Wow, Jake, you are really good at math!  You predicted the future!

Sure, but that’s not what’s so cool about this model.  The coolest thing is that it uses something called autocorrelation – normally a stumbling block when not taken into account in most statistical analysis – as part of the actual model.  I won’t get into the details of autocorrelation, but mainly it deals with the tendency of an observation to be highly correlated with the previous observation, which in our case is exactly the point.  See, if the model is correctly calibrated, then the actual performance should be what it predicted.  If Jake’s ELO Model Agency had a blended EmaiLO of 1406 to start, then we should expect it to have an EmaiLO rating of 1406 in the start + 1 month.  If we expected it to be higher or lower, well, then we should have rated it higher or lower in the first place!  This is where some other fun math comes in.

Why do your torture us??

In our EmaiLO model, we have to re-calibrate our ratings if things don’t turn out as planned.  In our example, since the expected outcomes mostly mirrored the actual outcomes, Jake’s ELO Model Agency came out with a 1406 – same as it started.  But what if something went wrong?  What if open rates did better than expected?

That’s where our k factor((s) – one for each metric) comes in.  It’s our useful guide on how to adjust ratings based on performance.  In our case, we can express the adjustment as

EmaiLO, Email marketing rating formula

This equation gives us our new RA and the process starts again.  I can’t give away our k factor(s), but I can tell you we’ve done a good job calibrating their predictive ability.

That wasn’t so bad.

And that’s it!  You’re now an expert in ELO models, and specifically our variation, EmaiLO.  I’m glad you made it to the end!  Keep on the lookout for some EmaiLO rating model related content in the near future, where we examine its predictive ability, highlight those clients that consistently beat expectations, and talk about self-calibration.

See you there!