BLOG

6 Types Of Marketing Testing Methods To Try

Published: May 7, 2020

There are many marketing testing methods out there, but which ones should you try? Here we look at 6 most popular ways to test various touchpoints across the entire user journey.

Marketing testing is probably one of the most misunderstood and thus, overlooked, areas in B2C marketing. Even when they are performed, marketing tests rarely influence any significant changes due to the fact that we either a) implemented the test wrong; or b) don’t know how to interpret the results that we’ve got.

We’ve all run an odd email subject line test at some point, or pushed our design team to A/B test different colors of a call-to-action button. But what really hides behind marketing tests and what mistakes should you avoid to enjoy data-driven insights? Join us as we dig deeper into the benefits of marketing and user journey testing.

What we’ll be covering:
1. What is testing in marketing?
2. Six popular marketing testing methods
2.1 A/B testing
2.2 Multivariate testing
2.3 User testing
2.4 Usability testing
2.5 Content testing
2.6 Incrementality testing
3. Why it pays to test the entire user journey

What is meant by testing in marketing?

​​Test marketing, also known as market testing, is a marketing technique that businesses use to evaluate the viability of their new products, services, or marketing campaigns before releasing them on a large scale. Businesses all around the world test multiple marketing scenarios to determine consumer needs and to find out whether the product, service or campaign meets and fulfils consumer requirements.

The test campaigns can be carried out in chosen test markets, such as in brick and mortar stores, online stores, or social media platforms. The experiment is generally done without customers’ knowledge to acquire non-biased feedback on the product or service.

After collecting test market feedback, the data is analyzed and data-backed insights are utilized to improve the product, service, or marketing strategy. To collect, aggregate and analyze the data, reporting tools are often used as these are able to automate data processes and showcase results in easy to understand visual reports.

A/B and Multivariate testing

Let’s kick off with the tests that most of us are already familiar with: A/B tests and multivariate tests. These are user experience research methodologies used to compare variables and determine which set of variables is most effective. Also referred to as split testing, they are somewhat similar to one another but offer rather different outcomes.

What is A/B testing?

A/B testing compares two variations of the same element to determine which one produces better results. It is called A/B testing because it measures a user’s response to Variant A vs Variant B.

A/B tests have risen to stardom in the early 2000s (interesting tidbit – first cases of A/B testing date back to the 18th century and were used to treat scurvy). They were quickly adopted by designers, marketers, and developers alike to test UI elements, copywriting, discount offers, you name it!

The nice thing about A/B tests is that they’re fairly easy to implement and they can be used at a variety of touchpoints in the customer journey to test placement, color, copy, size, and other variables. But easy doesn’t mean they are always done right. Here are the main pitfalls of A/B testing:

1. Testing more than two variants. The most common mistake is when you actually perform a multivariate test but judge the results as if it were an A/B test. To avoid this, remember this simple rule:
   
   [x] vs [x] = A/B test
   [y] vs [y] = A/B test
   [x] vs [y] – not an A/B test
   [x] vs [x+y] – not an A/B test

2. Insufficient sample size. For an A/B test to return significant results, you must think of your sample size before running the test. This is not a one-size-fits-all situation – you really need to think of the level of confidence you want to achieve as well as the uplift rate you’re looking to get. There are dozens of sample size calculators online so find one that works best for you!
   
3. Testing too short or too long. Again, it’s not easy to say what is a good test length for your A/B test but the general rule is that you should test for at least 7 days to reach statistical significance. If you haven’t, you can extend the test for another 7 days. However, if you run a test too long, there is a likelihood its results will compound with other tests and activities happening in the background.

What is multivariate testing?

Multivariate testing (MVT) compares multiple variations of several elements to determine which combination produces better results. The upside of multivariate tests is that you initially have more freedom and can make subtle changes rather than committing to versions A or B.

So what could be a multivariate testing example? Let’s say I want to increase conversions of my product page. Some of the elements I can use for a multivariate test are:

• Test different imagery: product shots vs lifestyle shots.
• Test button copy: ‘Buy now’ vs ‘Add to cart’.
• Test different trust signals: customer reviews vs dynamic social proof.

Or if you’re running an email marketing campaign to promote the release of a new product. You could test:

• Test sender names: brand vs individual.
• Test opening lines: with recipient name vs generic.
• Test visual formats: image vs video.

Note that MVT is a more advanced marketing testing method and is, therefore, most suitable for high traffic sites. Compared to A/B tests where traffic is usually split in two, in multivariate testing, it can be split in 3, 4, 5, and more groups.

User testing and usability testing

If you were ever involved in a product release, app launch, or even a website redesign project, you’ve probably heard of usability testing and user testing. Both of these terms refer to testing methods that involve real-life users and are often used interchangeably. However, they both offer different value and insights, and thus should be used separately.

What is user testing?

User testing, or user research, is an umbrella term that involves a variety of user testing methods: interviews, surveys, field studies, focus groups, and others. Its purpose is to determine the potential demand for an idea or a product by investigating user needs and behaviors. A typical outcome of a user test is a user persona or a target audience segment.

Under this definition, user testing would include usability testing.

What is usability testing?

Usability testing is a technique used to evaluate how easy or difficult it is to use a product or service by testing it on real users. Usability testing lets you understand how real users interact with the product or service, measure the user experience, and improve design based on your findings. In a typical usability test, participants are asked to complete a set of tasks while being observed by a research team.

“User Testing is about ‘Will this user use my product?’ Usability testing is to figure out ‘Can this user use this product?’ – the UX Blog.

There are multiple benefits of usability testing:

• Usability testing is easy to implement. All you have to do is gather some participants, find a room and ideally record the process.
• Usability testing saves money. It removes guesswork and enables you to focus on trouble areas instead of wasting away budget.
• Usability testing clarifies your USP. It enables you to observe honest and true reactions to your product or service, determine the ‘wow factor’ and capitalize on it.
• Usability testing uncovers missed opportunities. It showcases real-life scenarios and may uncover use cases that you haven’t thought of before.
• Usability testing can be used throughout the product life cycle. It is not just for launching new products – usability testing helps you iterate current versions and remove existing pain points.
• Usability testing makes everything better. Ultimately, it is here to improve user experience, which is essential for any project to succeed.

So what does this translate to for eCommerce marketers? Well, your websites and apps are your real estate, so focus on testing those! See how easy it is for users to find a product/offer they want, how they interact with ancillary products/offers on your store, how they navigate to the checkout, how they fill out the shipping form, etc. Usability testing can also work wonders for testing loyalty programs. 

Also – remember it doesn’t just have to be your product or service. You may as well test a competitor’s website to see where they might struggle or outshine you, gaining you a competitive advantage. 

There are several usability testing methods at play. Most commonly usability tests are split into: moderated vs. unmoderated; remote vs in-person; and explorative vs. assessment vs. comparative. The combinations go from there as you can have a moderated in-person assessment test, an unmoderated remote comparative test, and so on.

Content testing

Arguably the most self-explanatory of all the marketing testing methods listed here, content testing can be considered part of usability testing. Its primary purpose is to determine whether a set of real-life users can understand and comprehend the content that you’ve provided.

Again, the objective is to measure user experience: content testing is not to determine whether users like your content, it is to determine if they can read and understand it. A good example would be the user manual for assembling a piece of furniture: you can test if the text is legible, written accurately and whether it provides the information necessary to use the product. 

When thinking of eCommerce websites, content testing comes in handy when you need to determine the names of your product categories, phrase a discount offer and its terms and conditions or feature your headline or slogan on the page – does it catch attention and explain what the page is about?

Incrementality testing

One of the marketing testing methods that we like to talk about a lot at Yieldify is incrementality testing. 

Incrementality testing is a technique that measures the impact of a single variable on an individual user’s behavior. In other words, it helps to identify how effective a particular interaction is in influencing conversions. Incrementality tests are always performed with a control group to determine the uplift an interaction brings vs. a ‘no-change’ scenario.

Incrementality testing falls under the A/B testing umbrella. However, whereas A/B testing is used mostly on creative elements and measured by click-through rate (CTR), incrementality testing focused on sales uplift and is measured by conversion rate (CVR).

Widely used in digital advertising, incrementality testing found its way into conversion rate and customer journey optimization processes as well. At Yieldify, we run incrementality tests to help our clients understand the value of the campaigns that we run on their websites. Here’s an example of what an incrementality test report could look like:

Why it pays to test the entire user journey

The marketing testing methods we’ve discussed above are incredible ways to finetune your product, software or service. By employing some of them together you’re bound to collect invaluable data about your users and make necessary changes to, depending on your industry, sell more products, increase ticket sales or secure more customer sign-ups.

Nevertheless, conversion rate optimization (CRO) is just one side of the coin. If you really want to see significant outcomes of user testing, you have to employ these practices across multiple user touchpoints. In other words, you must test the entire user journey, from initial marketing channels, all the way through to how potential customers navigate your website.

The main difference between CRO and customer journey optimization (or CXO – customer experience optimization) is that CRO tests individual element sets removed from the rest of the customer journey. Whereas CJO (or CXO) runs across multiple touchpoints providing a ‘big-picture’ look at the different levers you can pull to improve your customer journey as a whole.

For that purpose, marketers exercise customer journey mapping. This allows them to not only develop empathy for the customer, but also understand consumer behavior patterns that influence conversions, loyalty, and a number of other business metrics that are directly connected to revenue.

In conclusion

Regardless of the type of market test you are carrying out, the most important thing is that you are conducting tests, collecting copious market research, and thus, gaining access to all of the valuable insights and data that can better inform your strategy. 

For the best results, it is clear that the holistic, 360 degree, all-encompassing, customer journey map reigns king. With high-impact benefits such as higher conversions, loyalty and revenue income, the marketers that can include this potent weapon in their arsenal will ultimately reap the best rewards and edge ever further in front of the competition by better understanding their target market.

Marketing Testing FAQs

Click to download