Category Archives: Creative

Font colour change causes 74% click difference

A split test with identical results doesn’t mean there is nothing to learn. A case in point is the split test covered here. It shows how valuable lessons can still be found by looking deeper even though the split test gave identical overall unique clicks results.

By drilling into individual calls to action we found a 74% difference between the control and treatment.

This split test is the third test run as a follow up to two previous tests. The first test delivered a 61% increase in unique clicks. Tests two and three were designed to tell us more about what drove that difference so the principles can be carried forwards and re-applied.

All tests were made on the DMA Infobox monthly email marketing advice newsletter – if you don’t get the newsletter signup here, its free.

This test was to understand the value of the Editorial section. In a previous test the Editorial section was in the control and not the treatment. Could adding the Editorial to the treatment increase response or would it just change reader behaviour?

Here are the emails tested, control on the left and treatment on the right.Infobox split test 3

The summary result was identical click through performance. What was there to learn from this?

I analysed which links were clicked. Each main article is promoted both in the Editorial section and in the main body following the Editorial.

A big difference between the control (left) and treatment (right) became apparent. The table below shows the uplift on the treatment over the control for links in Editorial area vs main body. Uplifts have above 95% statistical significance (check your test results with this ab split test calculator).

  Editorial links Body links
Uplift 74% -70%

In short whilst the overall number of clicks was the same, where people clicked was changed by the design. The treatment received the majority of the clicks in the Editorial area and had weak performance in body. In fact body clicks reduced by 70% over the control.

Why does the Editorial perform badly in the control?

There are two obvious reasons why the difference occurred:

  1. People find scan reading of white font on a blue background harder than the higher contrast of black on white. So they tend to skip over the editorial and scan the body and large sub-heads to pick out content of interest.
  2. Readers are using the ‘In this issue’ bookmark links in the control right column to skip to the body content.

Bookmark links, which just jump the reader to another place in the email rather than to a landing page, are not trackable. So we don’t have data to confirm or eliminate the second possibility.

The test results imply that the treatment can be reduced to just the black on white Editorial and the remaining simple email will give good performance.

Three lessons from this test

  • Look at individual link clicks for more insight and not just top line metrics when doing split tests on designs.
  • Focus design on simplicity. Get to the point and don’t make your email over complex simply because it looks more designed.
  • Avoid use of white font and coloured backgrounds. Black on white is just fine.

Remember the real world isn’t perfect.

Device tracking was included in the test with the intention of drilling into behaviour differences between desktop and mobile users.

When I looked at the device results they were quite remarkable. In fact so surprising I had to question the data. After digging deeper it became clear that a technical gremlin had meant the device tracking information was incomplete and the data could not be used.

The lesson here is be prepared to question everything. Especially if something looks too good or too bad to be true. Working with fundamentally wrong data is at best a waste of time and at worse takes you in the wrong direction, leading to worse results.

What next?

We’ve another test planned in this series that will drill into a different element of the control and treatment. I’ll be sure to share the results so come back next month and find out.

Email design change driving a 61% click increase

You can’t have missed the many reports and stats about the growing use of mobile devices to read email. Brands I work with are seeing anywhere from 15% to 70% of their emails being read on mobile devices. The average is 45% and the most popular email client, rated by number of emails opened, is now the iPhone, source

The DMA have been testing a new mobile first email design. Its optimised for mobile with a skinny design approach of 400 pixels wide, as opposed to using another technique such as responsive design. If you’re not sure about the different mobile design methods then read this short primer that explains skinny, scalable, fluid and responsive email design.

The DMA use the same template across many of the DMA newsletters. The design format change was put to the test by the DMA production team with an A/B split for the DMAs ‘Infobox’ newsletter. A newsletter dedicated to email marketing.

The skinny design treatment delivered a 61% increase in clicks over the control, with statistical significance of 99%.

However, the conclusion is not the obvious one. Can you see why? Here are the two designs used in the split test:


We can’t conclude skinny rules just yet. In making a design change of this significance there are many factors at play. Potentially important changes in this design treatment include:

  • Mobile friendly width change to skinny 400 pixels
  • Single column layout
  • Removal of thumb nail images
  • Increase in font size
  • Large call to action buttons
  • Removal of editorial introduction
  • Reduced copy
  • Changed call to action copy
  • Removal of in this issue contents list

So the question arises as to what really drove 61% click increase? Drilling down into clicks on individual links gives more insight.

I would expect some of these factors to change click response evenly across all links, such as font size change.

However, I found one particular link in the treatment was responsible for the majority of the overall click increase in the split test. This implies the copy and call to action changes are driving the click increase and not the change to skinny design.

To put the theory to the test that copy changes were driving the difference a further test was run, this time further synchronising the copy between the control and treatment, in particular ensuring the same calls to action. These are the two emails in the split test:


Note how the control, the design on the left above, now has the same calls to action and lead in copy to the CTA button as the treatment.

On this test there was no click rate response difference between the two designs and in fact both performed well. The skinny mobile friendly design no longer looks to be the cause of the improvement.

It would have been easy to initially conclude that the mobile friendly approach is driving the results after the first spit test. However, a failure to truly understand key causes of change means you’ll make future decisions based on your wrong perception of what’s important.

There are always unanswered questions, in this test what about:

  • The impact of the introduction editorial copy in the email, does it increase clicks or just re-distribute them?
  • Is there a difference in response between desktop readers and mobile readers with the different designs?

The DMA has further tests planned to provide more answers and gain clarity as to how design can improve click response for the DMA . Watch out for an update on this blog and I’ll be sure to share with you as to what we find next.


Email welcome campaign lessons from Sainsbury’s DM


A few weeks back I got a new Nectar card and used it for the first time last Saturday, clocking up my very first Nectar points.

Impressively on the following Tuesday the postman delivered me a piece of DM from Sainsbury’s.

This was no co-incidence, the opening line of the DM is “As you’ve recently picked up your first Nectar points at Sainsbury’s…” – a great lead-in line that immediately makes the content relevant to the reader.

A slick piece of marketing, triggering a DM shot from first use of a Nectar card, the design is a great example of how to welcome new customers


The DM design and content shares many best practices with email design best practice

  • A distinctive circle attracts the eye as the starting point
  • The headline “We’re here to help you get started” has clarity and tells the reader immediately what this is about – no need for hype here
  • The first paragraphs speaks to the individual
  • The brown signs give visual clues to direct your eyes across the pages
  • Copy is chunked and easily scanned
  • Icons and images are functional and support the message, no pointless pictures of a stock happy shoppers

The DM format is a horizontal foldout but the same concepts for eye-path flow can be adapted to vertical email design.

The messaging of the DM has clear purpose addressing in a logical order

  • The different ways to collect Nectar points
  • What you can do with the points you’ve collected
  • A next step to finding out more


Welcome campaigns are about education, providing customers with key useful information so the relationship can continue to develop.

Finally when turning the last foldout of the DM there are vouchers that incentivise the Nectar point collecting habit.

So there it is; great design, brand education and incentives to continue the habit that’s just started.

The key ingredients of every Welcome campaign, whether in DM or Email.


Are high open rates holding you back?

Our findings last month on the Obama campaign caused a lot of debate but the bare facts of our analysis still stand – had Obama’s team optimized for improved open rates, their send volumes would have dropped and their all-important donations would have followed.

Open rates remain a widely used and hugely misleading measure of performance and engagement in the email industry. At best they give you an idea of a campaign’s performance in isolation but at worst they lead email marketers to focus on optimizing the wrong strategies for their email program.

Here we discuss how to identify if maximizing open rates is holding you back and how to go about identifying the strategies that will have the biggest impact on your results.

The open rate paradox

Using EDS Analyst we examined the relationship between open rates and total unique opens for the top 200 email senders by list size in the US for 2012.

We were confident that, like the Obama campaign, there would be an inverse relationship between rates and totals – so as rates increase, totals decrease and vice versa. We call this the open rate paradox or to paraphrase a popular sports trusim: rates are for show, totals are for dough.

Each dot on the graph below represents a single sender and we picked out some well-known brands as reference points.

Sure enough, the graph shows that for most large senders, there is an inverse relationship between open rates and the total number of opens – the higher the open rate, the lower the number of total opens. Rates are for show.

It’s also no coincidence that nearly all of the brands with the biggest lists (orange dots) also have highest number of total opens because they are sending more opportunities to open.

Although opens don’t directly correlate to revenue, even the most avid fans of open rate maximization would agree that the more people that actually open your emails, the more engaged your database and the more revenue or conversions you are likely to generate. Totals are for dough.

Keep it simple – focus on just three strategies

If your goal is only to improve open rates, then your strategy is simple: halve your list by suppressing your less active subscribers and watch those rates soar… and those total opens plummet! But if your goal is to increase total opens, then the bell curve in the graph above helps define three clear strategies:

  1. List size:
    Has the biggest impact on totals and can be improved independently of the other two.
  2. Increase send volume:
    Significantly increases total opens for relatively little effort (low effort to gain ratio).
  3. Optimize for rates
    Increases total opens but requires the biggest effort (high effort to gain ratio).

Most brands are clustered towards the lower middle of the curve because it’s the easy place to be. By and large, they all put a similar amount of effort into their program and use the same undefined strategies.

The outliers, however, go above and beyond in one of three ways – those to the right have very high open rates, those to the left have high send volumes and those at the top are combining high send frequency with very big lists to produce massive send volumes.

In effect, this is the three different strategies implemented to their extremes.

Of course, there are limits to the effectiveness of each strategy and these are defined in the graph above by the orange line to the left (frequency cap) and green line to the right (optimization cap).

These boundaries exist because for any given list size there is point at which diminishing returns kick in for both frequency and open rate. And, as the big empty space to the right of the green optimization cap shows, it’s very hard to send a large volume of email while still achieving a high open rate.

So the basis of a successful email program is to continually grow your list while finding a balance between increasing send volume and maximizing open rates with better offers, targeting, subject lines, etc.

And you find that balance by ignoring your open rates…

Define your strategy by ignoring open rates

To illustrate the effect these strategies have on an email program, we have created a simple optimization chart, below. The green curves represent the impact of send volume on total opens and the brown lines represent the impact of open rate on total opens.

Each intersection represents a hypothetical 10-hour unit of resource, as a means of comparing the effort required to implement each strategy. As you get closer to each cap, the effort required to improve your totals with your chosen strategy increases exponentially.

Imagine your brand is the star in the middle of the curve and you want to take on your leading competitor, the lightning bolt.

If you use open rates to define your strategy, then you focus your resource on maximizing those, route A. Your open rate may now be much better than your competitor’s but they are out-mailing you, so they are still creating twice as many opportunities to buy or convert.

If you choose to increase your send volume, ‘route B’, then your open rate drops but your total opens more than double. However, as you approach the frequency cap, the impact of your strategy diminishes and you still trail your competitor.

If you use totals to define your strategy, then you take ‘route C’, which balances resource between increasing send volume and maximizing open rates. Your open rate drops but you are finally creating more opportunities to buy than your competitor.

Smart email marketing is not just a case of increasing send volume indiscriminately or of only focusing on ever tighter targeting. There is a balance that exists for each brand, you just have to find your own sweet spot.

Total opens the key to optimizing your program?

In this instance, we have highlighted the open rate paradox using total opens because that was the data available. However, we’re confident you will find the same inverse relationship in your own campaigns with total clicks and, more importantly, revenue. And in the end that’s the only metric that matters!


Stop everything – we need to re-design for mobile devices

There has been a lot of talk over the past few weeks among various groups about whether or not mobiles have become so important that we should all be designing new templates for mobiles either optimised or using responsive design. A recent study by Blue Hornet says that 80% of people will delete an email on a mobile device if it doesn’t look good so this suggests that we should.

Now, a survey is only as good as the questions it contains and how these are phrased so let us look at the question. “If you get a mobile email that doesn’t look good then what do you do”. So what constitutes “does not look good”

Lets go back a few years to the days when Blackberrys where the only smartphones we had to worry about. In these days the html renderer on a Blackberry was pretty terrible and only the simplest of html would be readable and older versions would only give you the text version of the message. If was pretty safe to say that any html you sent would not look good.


Nowadays the html renderers on smart phones are excellent and they will pretty much render anything that a desktop email client will. Just because people are opening your email on a mobile device and you haven’t fully optimised your campaign for mobile it doesn’t mean these people aren’t engaging with your campaign.

So, everyone can relax a little. Without doing anything, you have a mobile strategy because people can read your emails on a smart phone and engage. It maybe harder to navigate around the email on a mobile device and the calls to action a little difficult to click but if your email looks good on a desktop then the odds are it will look ok on a mobile device.  If you want to improve the user experience on a mobile device then it is not about getting your campaign to render on a mobile device but about optimising it for mobile devices.

The question is will this improve your campaign results? Every campaign is different and just because Litmus say that 43% of emails are read on a mobile device it doesn’t necessarily translate to your target audience. Tim Watson from Zettasphere analysed the data and found there are still campaigns at both ends, some with almost no mobile activity and some with almost only mobile activity. If you have historical campaign data to your subscriber base then look at your open stats and see what your mobile open rate is across a variety of campaigns.

However, don’t be misled by your mobile open rate. Just because someone opens your email on a mobile device it doesn’t mean they don’t then open it on their desktop as well. If I receive an interesting email on my mobile device but find it difficult to read I just wait until I’m in the office and read it on my desktop.

One thing that does generally improve campaign performance is re-designing an old template whether it is related to mobile or not. If you put the time and effort in to look at your existing template and re-design it with the mobile user experience in mind then I would expect your results to improve. No matter whether this is related to mobile opens or not.  Its a win win. The new template should get more opens on the desktop and mobile devices.

Everyone should be thinking about mobile devices and how this affects their campaigns but you don’t necessarily have to drop everything and rethink what you are doing. Look at your recipients and stats. If it has been a long time since you changed your template then it may be time to think about creating a new simplified template with mobile devices in mind.

What do your emails say about you?

In my last DMA blog I made the mistake of highlighting the “branding” gap in Apple’s emails. And I was duly punished by the “Apple gods”, who caused me to drop my new iPhone 5 in the loo. Actually, a recent YouGov survey showed that 21% of people check their emails in the toilet, and I can tell you that there are literally hundreds of web posts advising on how to repair iPhone water damage. So, though it probably wasn’t a personal punishment, I’m certainly going to be more careful about what I say in future. But I would still like to explore further one of the areas that this blog touched upon, and that’s the importance of keeping consistent and relevant branding in all email communications. There’s a profusion of advice on how to optimise your email marketing, a deluge of articles about how mobile is impacting email, and a large wash of commentary about new functionality – testing, responsive design, HTML 5 etc. And this is all good stuff. But there’s not much marketing advice on how to get your employee email up to scratch, for those one-to-one customer communications, and consequently there are still many email basics that businesses are not getting right. If you were to send a mail now from your work email address to a client, what impression would that give them of your brand? Would it help reinforce your business proposition or increase awareness of your company’s services? Well, I received an email today from one of the team at the DMA and I think this is a great example of how email signatures can be used as a marketing tool. (And I haven’t turned from criticism to sycophancy out of fear of retribution before you ask….). Here’s their current email signature:-


The email footer delivers the powerful message “180 days to save your industry”, with a strong call to action “Act now”, and this links directly to the event sign-up page for the DMA Data Conference. As well as linking directly to their site, and helping to promote registrations, it also helps to demonstrate one of the core activities of the DMA – that they lobby on our behalf, and ensure DMA members are informed about the implications of any proposed marketing legislation. And what’s good about their signature is that it also works on mobiles. So if, like me, you are reading their emails on the loo, you will still experience their branding. – And, actually, you’ll get an even better experience on your phone than on Outlook…they have used an animated gif in their signature, that looks particularly good on my (new) iPhone, creating a neat “countdown effect” that shows that time is running out, and increases the urgency of their message.


Having a good email signature can make all the difference to your communications. Here are my top tips of things to think about for your email signatures plus any headers or footers that you include, to help increase the marketing potential of the thousand individual emails that each employee in your organisation will send every month. • Define your objectives: Is it to achieve a consistent corporate look? To generate sales? To increase awareness? To distribute content? Do you need to measure the results? (These factors will impact the design style plus the kind of solution you’ll need to implement your signatures.) • Aim to keep image weight under 30KB • In terms of image size, don’t make your designs wider than 650 pixels or deeper than 100 pixels. • In signatures, use web safe fonts, so that they will render consistently for all recipients • Use basic HTML without nested tables as these can cause issues with reply mail chains • Try to avoid background colours in the HTML layout as these render differently in different email clients • Try to avoid using background images as many email clients do not support them • Minimum font size should be 8pt or size 1 for best legibility