Tim Watson has over 8 years experience in B2B and B2C Digital Marketing, helping blue chip brands with successful email marketing.
He is an elected member of the UK DMA Email Council, supporting the email marketing industry. Tim Chairs the Legal and Best Practice hub of the Email Council, authoring and reviewing DMA whitepapers and best practice documentation. He is also a frequent speaker and blogger on emerging email marketing trends.
Tim works as an independent email marketing consultant providing strategic support to email marketing teams.
A mega test for sure, Marketing Experiments took 300 subject lines suggested by their readers, picked what they felt were the best ten and ran a split test.
The table of results is below. Read the original post for details of how it was run but before you do read on here for a quick lesson in email split test strategy not mentioned elsewhere.
Notice that the top three had the same level of performance, within level of confidence. The next five subject lines, lines 4 to 8, also shared the same performance level too.
Had only the top three subject lines been tested there would have been no winner! Had the test been just the five subject lines, lines 4 to 8, there would have been no winner – in five tests.
The crucial lesson is that testing is a game of skill and quantity. A key ingredient to email split test strategy is simply doing enough tests and different treatments to discover a big gain. Doing occasional tests won’t do it.
If you are running A/B subject line tests then be prepared for several tests showing no performance improvement. Don’t get despondent but keep persevering.
As a rule of thumb expect out of ten test treatments for one to make a big shift to the needle, 3 provide some learning and improvement, 3 no winner and 3 under perform (but you’ll still learn).
When you’ve enough data to run multiple tests concurrently then it’s great to do as it accelerates learning and optimisation. However, remember your sample sizes need to be large enough for statistical significance. A 10,000 list split into ten test cells of 1000 each won’t give any valid result.
Stay on the straight and narrow with this split test calculator. to work out sample sizes and check statistical significance of results.
A split test with identical results doesn’t mean there is nothing to learn. A case in point is the split test covered here. It shows how valuable lessons can still be found by looking deeper even though the split test gave identical overall unique clicks results.
By drilling into individual calls to action we found a 74% difference between the control and treatment.
This split test is the third test run as a follow up to two previous tests. The first test delivered a 61% increase in unique clicks. Tests two and three were designed to tell us more about what drove that difference so the principles can be carried forwards and re-applied.
All tests were made on the DMA Infobox monthly email marketing advice newsletter – if you don’t get the newsletter signup here, its free.
This test was to understand the value of the Editorial section. In a previous test the Editorial section was in the control and not the treatment. Could adding the Editorial to the treatment increase response or would it just change reader behaviour?
Here are the emails tested, control on the left and treatment on the right.
The summary result was identical click through performance. What was there to learn from this?
I analysed which links were clicked. Each main article is promoted both in the Editorial section and in the main body following the Editorial.
A big difference between the control (left) and treatment (right) became apparent. The table below shows the uplift on the treatment over the control for links in Editorial area vs main body. Uplifts have above 95% statistical significance (check your test results with this ab split test calculator).
In short whilst the overall number of clicks was the same, where people clicked was changed by the design. The treatment received the majority of the clicks in the Editorial area and had weak performance in body. In fact body clicks reduced by 70% over the control.
Why does the Editorial perform badly in the control?
There are two obvious reasons why the difference occurred:
People find scan reading of white font on a blue background harder than the higher contrast of black on white. So they tend to skip over the editorial and scan the body and large sub-heads to pick out content of interest.
Readers are using the ‘In this issue’ bookmark links in the control right column to skip to the body content.
Bookmark links, which just jump the reader to another place in the email rather than to a landing page, are not trackable. So we don’t have data to confirm or eliminate the second possibility.
The test results imply that the treatment can be reduced to just the black on white Editorial and the remaining simple email will give good performance.
Three lessons from this test
Look at individual link clicks for more insight and not just top line metrics when doing split tests on designs.
Focus design on simplicity. Get to the point and don’t make your email over complex simply because it looks more designed.
Avoid use of white font and coloured backgrounds. Black on white is just fine.
Remember the real world isn’t perfect.
Device tracking was included in the test with the intention of drilling into behaviour differences between desktop and mobile users.
When I looked at the device results they were quite remarkable. In fact so surprising I had to question the data. After digging deeper it became clear that a technical gremlin had meant the device tracking information was incomplete and the data could not be used.
The lesson here is be prepared to question everything. Especially if something looks too good or too bad to be true. Working with fundamentally wrong data is at best a waste of time and at worse takes you in the wrong direction, leading to worse results.
We’ve another test planned in this series that will drill into a different element of the control and treatment. I’ll be sure to share the results so come back next month and find out.
You can’t have missed the many reports and stats about the growing use of mobile devices to read email. Brands I work with are seeing anywhere from 15% to 70% of their emails being read on mobile devices. The average is 45% and the most popular email client, rated by number of emails opened, is now the iPhone, source http://emailclientmarketshare.com/.
The DMA have been testing a new mobile first email design. Its optimised for mobile with a skinny design approach of 400 pixels wide, as opposed to using another technique such as responsive design. If you’re not sure about the different mobile design methods then read this short primer that explains skinny, scalable, fluid and responsive email design.
The DMA use the same template across many of the DMA newsletters. The design format change was put to the test by the DMA production team with an A/B split for the DMAs ‘Infobox’ newsletter. A newsletter dedicated to email marketing.
The skinny design treatment delivered a 61% increase in clicks over the control, with statistical significance of 99%.
However, the conclusion is not the obvious one. Can you see why? Here are the two designs used in the split test:
We can’t conclude skinny rules just yet. In making a design change of this significance there are many factors at play. Potentially important changes in this design treatment include:
Mobile friendly width change to skinny 400 pixels
Single column layout
Removal of thumb nail images
Increase in font size
Large call to action buttons
Removal of editorial introduction
Changed call to action copy
Removal of in this issue contents list
So the question arises as to what really drove 61% click increase? Drilling down into clicks on individual links gives more insight.
I would expect some of these factors to change click response evenly across all links, such as font size change.
However, I found one particular link in the treatment was responsible for the majority of the overall click increase in the split test. This implies the copy and call to action changes are driving the click increase and not the change to skinny design.
To put the theory to the test that copy changes were driving the difference a further test was run, this time further synchronising the copy between the control and treatment, in particular ensuring the same calls to action. These are the two emails in the split test:
Note how the control, the design on the left above, now has the same calls to action and lead in copy to the CTA button as the treatment.
On this test there was no click rate response difference between the two designs and in fact both performed well. The skinny mobile friendly design no longer looks to be the cause of the improvement.
It would have been easy to initially conclude that the mobile friendly approach is driving the results after the first spit test. However, a failure to truly understand key causes of change means you’ll make future decisions based on your wrong perception of what’s important.
There are always unanswered questions, in this test what about:
The impact of the introduction editorial copy in the email, does it increase clicks or just re-distribute them?
Is there a difference in response between desktop readers and mobile readers with the different designs?
The DMA has further tests planned to provide more answers and gain clarity as to how design can improve click response for the DMA . Watch out for an update on this blog and I’ll be sure to share with you as to what we find next.
A few weeks back I got a new Nectar card and used it for the first time last Saturday, clocking up my very first Nectar points.
Impressively on the following Tuesday the postman delivered me a piece of DM from Sainsbury’s.
This was no co-incidence, the opening line of the DM is “As you’ve recently picked up your first Nectar points at Sainsbury’s…” – a great lead-in line that immediately makes the content relevant to the reader.
A slick piece of marketing, triggering a DM shot from first use of a Nectar card, the design is a great example of how to welcome new customers
The DM design and content shares many best practices with email design best practice
A distinctive circle attracts the eye as the starting point
The headline “We’re here to help you get started” has clarity and tells the reader immediately what this is about – no need for hype here
The first paragraphs speaks to the individual
The brown signs give visual clues to direct your eyes across the pages
Copy is chunked and easily scanned
Icons and images are functional and support the message, no pointless pictures of a stock happy shoppers
The DM format is a horizontal foldout but the same concepts for eye-path flow can be adapted to vertical email design.
The messaging of the DM has clear purpose addressing in a logical order
The different ways to collect Nectar points
What you can do with the points you’ve collected
A next step to finding out more
Welcome campaigns are about education, providing customers with key useful information so the relationship can continue to develop.
Finally when turning the last foldout of the DM there are vouchers that incentivise the Nectar point collecting habit.
So there it is; great design, brand education and incentives to continue the habit that’s just started.
The key ingredients of every Welcome campaign, whether in DM or Email.
Ever heard someone say the purpose of the subject line is to get the open? This is short sighted and the purpose and impact of the subject line goes much deeper. The thinking behind a subject line should be more than “what will make someone read this email?”
A case in point is some work I’ve recently completed for Coinks Deals. I’d like to share with you what was learnt about subject lines and how to best communicate with a large dormant database.
Coinks Points introduced a new deals service for their members and wished to provide these deal emails to members who responded to an introduction email about them. Coinks have millions of members, including hundreds of thousands of who had been suppressed from contact for over 12 months. The challenge was how to message to their entire member database, including the dormant members.
The messaging strategy I developed was a four email sequence using a high degree of personalisation to make connection and re-establish trust with members. Several tests were developed to optimise each step of the sequence, testing a variety of elements, including of course subject lines.
As always with testing, the results were insightful and I’m going to focus on one of the subject line tests and what you can learn from it.
For the third email in the sequence to the dormant segment, one of the tests was of these two subject lines:
Subject A: Are we still welcome in our inbox?
Subject B: Was it something we said?
The email itself was a short mostly plain email with a few links and a couple of central buttons shown below
The subject line B gave a 67% higher open rate. However, what was interesting to show the impact of the subject line beyond the open was the ratio of clicks on the above two buttons.
For subject line ‘A’ the ratio of clicks on the first button to second button was 6.5 whereas for subject line ‘B’ it was 2.8. Customers with subject line ‘A’ were more inclined to click the first button. The test cell sample size was 12,000 and the difference in clicks was statistically significant.
The difference in ratio was due to the different subject lines, it changed how customers read the message and what they did as a result. In this case “Are we still welcome in your inbox?” prompted the customer to consider this very question and whether their answer is yes or no. Whereas “Was it something we said?” does not prompt the direct question and the more conciliatory tone creates more interest in deals.
In the many tests I’ve run over many clients I’ve time and time again seen that what happens in the email is skewed and changed by the subject line. The subject line should be designed to get the right people to open not the most people, the right people means those most likely to take the action you want. Plus the subject line should frame their thoughts correctly.
The subject line is used by customers to self-qualify, if the subject line does not accurately qualify the right people then customers who might have taken action do not open and conversely some open only to find it’s not the right message for them. In this case the risk is customers become less inclined to open again since they found they wasted their time previously.
Summarising two key learning’s:
When testing subject lines don’t stop evaluation at the open rate, get more insight by looking deeper at which individual links were clicked and the call to action of each, to learn why the subject line created a particular result.
Create subject lines with the call to action in mind. The power and impact of the subject line goes further than getting the read, it’s about getting the action and not just the read.
This was just one test out of many over a series of four emails. The compounded gain across the whole email sequence was an impressive 190%.
Next time you think about subject lines don’t focus on just getting the open but setting up the right thought sequence for the call action.
Acknowledgements: My thanks to Coinks Deals and Emailvision for permission to publish the results from this work.
Everyone wants it, but there is no industry consensus on the best way to measure it. I’m talking about engagement in the email channel.
Take an example, a fashion brand might send two or three emails per week. It’s not realistic to expect that most people are going to be interested in buying a new fashion item every week or even to review offers each week.
Just because someone is not in the mood to buy or look at current offers does them make them no longer engaged with a brand? Of course not, they gave permission to receive the emails, they showed engagement, ignoring a few emails does not mean a lack of engagement.
Classically campaign open and click rates are used to judge engagement. This was fine when brands sent one campaign per month. Email volumes have increased considerably in the last five years but metrics have not moved on.
A re-think is needed as the classic metrics measure campaigns not customers and as a result promote the wrong behaviour in email marketing.
Its customers that need to be engaged so measuring campaigns makes no sense, its customers that should be measured.
I’ve been working on a paper, along with my fellow DMA Email Council hub members, Dela Quist, Skip Fidura and Kath Pay. The paper goes to the core of how to measure customer engagement in the email channel and delivers a verdict, based on analysis of brand data.
The paper has been put together to kick-start the discussion in the email industry about just what should be measured and the debate is starting at Email Evolution 2013 conference in Miami this week. For DMA blog readers we’re releasing the paper to you ahead of the event.
I hope you download the paper and find it thought provoking. If you leave a comment one of the paper authors will reply.
We have been covering the proposed EU data legislation. The full changes to society and benefits from the internet are far from complete, but one thing is clear, in the future the strong countries and economies will be those that have the best digital capability. The EU Data Protection Regulation threatens to limit and slow the growth of the digital economy in Europe.
In this post we look at the changes to the rules as to when individuals can claim compensation if they have suffered damage due to non- compliance with the proposed Regulation.
The current data protection legislation, Article 23 of the 1995 EU Data Protection Directive (95/46/EC) allows for individuals to claim compensation for damage suffered due to non- compliance with the Directive .
The current article states:
1. Member States shall provide that any person who has suffered damage as a result of an unlawful processing operation or of any act incompatible with the national provisions adopted pursuant to this Directive is entitled to receive compensation from the controller for the damage suffered.
2. The controller may be exempted from this liability, in whole or in part, if he proves that he is not responsible for the event giving rise to the damage.
The proposed changes are in Article 77 of the draft Regulation and the current text would widen the rights of individuals to be able to claim compensation in the event of non- compliance with the Regulation as they would be able to make claims against data processors Iin the case of multiple controllers/processors they would be jointly liable for the full amount. The proposal in full is:
1. Any person who has suffered damage as a result of an unlawful processing operation or of an action incompatible with this Regulation shall have the right to receive compensation from the controller or the processor for the damage suffered.
2. Where more than one controller or processor is involved in the processing, each controller or processor shall be jointly and severally liable for the entire amount of the damage.
3. The controller or the processor may be exempted from this liability, in whole or in part, if the controller or the processor proves that they are not responsible for the event giving rise to the damage.
The points of concern the proposed text raises are:
• Data processors located within the European Economic Area ( the 27 Member States of the EU plus Iceland, Lichtenstein and Norway) (EEA) could be liable for failings by data controllers outside of the European Economic Area (EEA)
• The definition of damage and the risk that it fosters a compensation claims culture
1) Liability between data controllers and data processors
If the data controller is located outside of the EEA then it brings a risk for any data processor located within the EU to deal with such a data controller. A data controller located outside the EEA may not be familiar with the EU data protection legislation, and under the draft Regulation the data processor located within the EEA could be liable for for the actions of the data controller in cases of non- compliance with the legislation. Such actions of the data controller could be outside the data processor’s control. This may stem the ability of data processors located within the EU to grow and deal internationally. This is not in the EU’s economic interest..
What is the benefit of adding liability to the data processor for the actions of data controllers? Data processors are required to have data processing agreements in place with the data controllers for whom they are processing personal data. Under such agreements a data processor should be liable to the relevant data controller for any breaches of data protection legislation. Giving individuals the right to claim compensation from data processors as well as data controllers may mean more disputes between data controllers and data processors as to which one should pay the compensation for damage caused by non-compliance with data protection legislation
2) The definition of damage
What is damage? The UK Information Commissioner’s Office believes it is the European Commissions’ intent for damage to include ‘damage or distress’ suffered. Given the historical difficulty and subjectiveness of establishing whether distress has been caused and the monetary value of that distress, this is bound to lead to expensive court cases to establish these parameters. The loser will be the consumer as increased risks and legal costs will be passed on by business and ultimately paid for by the consumer. The alternative is some data processors will be forced to go out of business. To avoid this, clarity as to the meaning of damage in the draft Regulation is needed- does this mean “quantifiable monetary damage” as is currently the case in UK law?
Overall this change appears to bring in unnecessary additional legal risk, the costs of which will ultimately land upon the consumer and will limit the growth of the European digital economy.