Feeling testy

Harry Hill would put it well.
 

“I like your short, to-the-point one page letter. And I like your long, involving four page letter too. But which one’s better? There’s only one way to find out...

“FIGHT!”

One of the best things about direct response media is being able to pitch one approach against another, and see which works best.

Rather than just theorising about what might work better, we can test our theories – and get empirical evidence as to the way forward.

Having said that, I think testing can be a very dangerous thing.

Or at least, the way we interpret the results of a test can be – in the same way that a little knowledge can be a dangerous thing.

I shall explain my heresy, but first, here are five of the dozens and dozens of tests I’ve been party to (and these are just in mailings, let alone other media):

1: involvement devices / incentives
a) Whether or not they increase response (yes, most of the time, especially for acquisition)

b) Whether that increase in response outweighs the extra cost – ie does it improve or worsen return on investment (improves it most of the time)

c) do they buy poorer quality donors, who have a poor lifetime value and often lapse (sometimes difficult to tell, but I certainly think the involvement / incentive should be relevant, rather than just arbitrary ‘labels or pens’, so you’re developing brand engagement, not just creating guilt-induced reciprocation).

2. reply envelopes
Testing a reply envelope with the return address printed on it versus a window reply envelope so you have to insert the donation form correctly to display the return address (the window reply envelope won)

3. donation forms
Testing an A5 donation form versus an A4 donation form (the A4 version won)

4. mailsort light
Testing a mailsort light teaser versus a mailsort light follow-up (the follow-up won)

5. need or solution
Testing need-led versus solution-led creative (need won).

All pretty straightforward, simple stuff. The problem, I think, is when we extrapolate the results of a test to create The Way Forward For Every Campaign For The Rest Of Time.

The easiest example of where I’ve seen this happen is to leave fundraising for a moment and look at financial services, another sector I’ve a fair bit of experience in.

And in financial services, so many mail packs to existing customers have blank outer envelopes. Because it’s from a company you’ve got some kind of financial product with, you see the logo and you’re likely to open it just in case it’s something about that product.

So the pack has a very high ‘open rate’.

What’s more, they’ve tested blank outer versus a message on the outer. And the blank outer got a better response.

So now all their packs have blank outers. Because testing has ‘proven’ that blank outers work better than outers with a message on.

Except of course, it hasn’t.

Testing has proven that, at that time in history, a blank outer was more successful than that particular message you had on the outer. It doesn’t mean a blank outer is better than any message. What if – just imagine – you had something genuinely good to say? Maybe a better line, with a stronger promise, would outperform a blank outer.

You still might get fewer people opening the outer than a blank one but those who do open it are already warm to the message inside. They’re already interested in the story, which you can continue in the pack, rather than (in the case of the blank outer), beginning.

So, when it comes to testing, we've got to be careful:

1. Things change over time, and what didn’t work yesterday might work today.
2. Messages can lose efficacy with uniform repetition – if we keep saying the same thing in the same way, it’ll become wallpaper and be ignored – even if it was the 'best' way of saying it.
3. We mustn't overreach what the results of the test are actually telling us.
 
If it was as easy as testing something, then using it that way forever, by now we'd all be telling the same story in the same way and every communication would look identical.

But of course, they don't. There's still plenty of art in what we do, to go with the science. That’s what makes it so blimmin’ interesting.