Opinions Targeting

From A/B to A/Z Testing: How Machine Learning is Transforming Mobile Marketing

Posted by Glenn Pingul on October 17th, 2012 at 5:36 am

A/B testing has been used by marketers for decades.    The concept is simple.  Present two versions (A and B), then measure the response rate, determine ‘the winner’ and then target that offer to everybody.

Originating with direct mail before being applied to online, A/B testing continues to be used as a means for determining what is more effective in driving response rates.  But when are marketers going to stop shooting for the average?

A typical A/B test would help you measure the impact of certain elements, such as:

  • Does a percentage or dollar value discount drive customers to respond?
  • Does a concise or extended amount of product information lead to online purchase?
  • Does an immediate or extended call to action more likely to drive a purchase?
  • Does a confident or humorous tone lead to a higher response rate?

The assumption is made that if group A’s treatment results in a higher response rate, than this is the ‘winner’.   A perfect fit for all?   Not likely.  What about the people that actually preferred offer B?   Force offer A on them?  And what about offers C, D and G relative to offers A and B?  What about messages F, J and M?  Content elements G, H, and L?  Plans E and H?

Now sure, a marketer could test each of these different combinations to determine the winners.  But how long would it take?  And chances are that the marketer would have no way to determine the efficacy of each message by customer in a given context.

But if you could target the best offer at the right time and place to each customer, would you shoot better than average?  In a word, yes.

Contextual marketing is relatively new and it’s not something that can be done easily given the vast amount of data that exists and the difficultly associated with harnessing that data.  There are, however, new technologies in this area.  They involve real time data capture and analysis and sophisticated machine learning to determine, for example, the best context for engaging each customer and the most relevant message – whether it be promotional, educational, informative, billing related, etc.

Let’s look back at the A/B test examples above.   Now let’s take them a step further to incorporate context:

  • Does a percentage or dollar value discount drive a multi-purchase customer to respond when delivered 48 hours after a previous purchase?
  • Does a concise message or one with more detailed product information lead to online purchase for a customer who engages with our app at least three times a week while at home?
  • Is an immediate call to action more likely to result in a purchase for a recently-acquired customer who works within a five mile radius of your retail location?
  • Does a confident or humorous tone lead to higher response rates for customers who have decreased usage by more than 25% over past month?

Questioning if this is possible?   Most marketers have the data – purchase history, location information, usage, tenure, loyalty scores, etc. – but do they have the tools to easily act on it?

Recently some mobile marketers have discovered new technologies that allow them to test hundreds of treatments simultaneously to determine the efficacy of messages and contexts, learn through automated analysis and model generation, and then seamlessly iterate and optimize campaigns to ensure delivery of the best treatment to each customer.

Essentially what makes this move from A/B to A/Z testing possible is machine learning.  With machine learning capabilities, you can rapidly – that is, automatically – determine what is the best offer for a given customer and context.  The technology does all of the complex analysis to sort out what works best for whom, when, where, etc. and automatically identifies which factors drive a certain behavior or outcome, thus eliminating the guesswork.  As a result, marketers gain the ability to test an infinite number of combinations of offers and contexts, get quick learnings, and then based on the insights continually iterate and optimize.

Part of the glory of this type of technology is that the drilling in to the details of campaign performance is no longer manual.  Machine learning automatically identifies which factors among hundreds of behavioral and demographic attributes drive a certain behavior or outcome for an individual customer. What this means is that it virtually eliminates the manual nature of analyzing campaign performance and trying to figure out which rule or condition, for example, led to a positive response.  The result is that by targeting the best offer in the right context to each customer you’re able to maximize returns in terms of increased lift for every one-to-one communication.

So for those marketers who are still shooting for the average – that is looking for the winner in either A or B and blasting the winner to the entire customer base, I say it’s time you look at what is going on in mobile.  Mobile has forced marketers to figure out how to act in context.  Mobile marketers are making great strides in harnessing the vast amount of data that they have to be smarter marketers.  They know that being relevant requires understanding your customers’ behaviors, and actually monitoring behavior in real time to then deliver the right message in the right context to the right customer.

They are realizing that determining the right context and the right offer means more testing.   But using machine learning means less guesswork in determining what is working and why.  It means shorter cycle times for campaigns.  And it means less dependency on internal and external groups to develop and deliver campaigns.

Finally marketers can spend more time thinking creatively about what delivers more value to their customers rather than how to deliver it.  As a fellow marketer I say…that’s a winner.

One Response to “From A/B to A/Z Testing: How Machine Learning is Transforming Mobile Marketing”

  1. Holden Caulfield says:

    I'm sorry, but I'm not impressed with this article. Beat me if I'm wrong, but it looks to me that big claims are made about what "machine learning" can do without the slightest explanation of how it would work.

    For example, I think that anyone with a firm grounding in statistics would agree that this claim is foolish: "Machine learning automatically identifies which factors among hundreds of behavioral and demographic attributes drive a certain behavior or outcome for an individual customer."

    By automatically, I hope you don't mean that "machine learning" achieves this identification with one single observation of a single customer. A statistician will tell you that, in order to do so, a sample is needed - i.e. many observations. The greater the number of attributes to judge, the greater the sample needs to be. This experiment can be achieved faster if the sample is taken across a multitude of customers. But to discriminate at the customer-level, the sample needs consist in observations of a single customer - and that is likely to take much, much longer to obtain.

    My knowledge of machine learning is limited, so perhaps machine learning has nothing to do with statistics and actually does fulfill your claim - in which case please go into detail. Otherwise it makes the article sound like the work of a journalist who understands nothing and is trying to sell a big fake story to a mass market.

Leave a comment