Mastering Creative Multivariate Testing at Scale

In a world of constant change and fierce competition, the key to sustained growth lies in the ability to adapt and innovate. Unlocking growth has become a fast-paced moving target, and the key to keeping a brand in the limelight is not about chasing those trends but setting them.

A cornerstone of our approach is rapid experimentation, and more specifically, multivariate testing, which allows us to improve processes, products, and future campaigns to enable partners to make informed, data-driven decisions quickly and iteratively based on results.

Rapid experimentation is a broad concept that encompasses various testing and experimentation methods such as A/B testing, pricing experiments, customer feedback loops, multivariate testing, and more. Within growth marketing, being willing, open-minded, and strategically set up to constantly refine and optimize marketing strategies based on data, from ad creatives to website design, is the key that will set a brand apart.

A few years ago, launching A/B tests was a novelty and could provide the dirty, quick, and rough insights needed to pivot and see results. Though A/B tests are still helpful, taking the test one level deeper and launching multivariate testing is what’s now required to make data-driven decisions that will move the needle.

Multivariate testing is the process of simultaneously testing multiple variables across numerous growth channels and campaigns to determine which combinations yield the best results. Multivariate testing is like conducting a symphony of experiments, each with its own set of variables. It’s not just A/B testing; it’s A to Z testing. For each element of a marketing campaign—be it ad copy, visuals, landing pages, or CTAs— test multiple variations simultaneously. The significance lies in its ability to uncover hidden insights, optimize user experiences, and boost conversion rates.

Applying multivariate testing throughout an organization

Whether the brand is a startup or an established market leader, there is always an opportunity to leverage experimentation as the playground where ideas are tested, refined, and transformed into growth engines.

• The playground for startup founders: At its core, multivariate testing is an opportunity to discover what makes the brand and product shine. Leverage multivariate testing across a variety of growth channels and within customer surveys (zero-party data analysis) to uncover the golden nugget of what customers feel makes the brand and product stand out.

• The strategic vision for CMOs and growth execs: This role defines the macro vision of audience and persona testing, messaging and value proposition testing, and other experiments that will chart the course of the brand’s growth trajectory. Multivariate testing empowers the growth leader to pivot when necessary and double down when a winning formula has been identified. This experimentation approach provides data-backed insights needed to craft a growth strategy that’s not just effective but exhilaratingly efficient.
• The battle plan for individual contributors: This group is full of technical and tactical masterminds who execute the vision. They are the data explorers who dig deep into the numbers to unearth the hidden gems and rumblings of a shift in the macro environment. Their battlefield is the digital landscape within creative, messaging, landing pages, and campaign objectives. This role uncovers the precision and firepower required to conquer every stage of the customer journey.

3 Core wins of multivariate testing

• Precision optimization & performance insights: Multivariate testing provides the opportunity to analyze the performance of multiple variations of different elements within a campaign, landing page, or targeting strategy.
  • Benefit: This precision provides valuable insights into which specific elements contribute most to the campaign’s success. This makes fine-tuning simple and effective so the final product can achieve maximal impact.
  • Impact: By pinpointing these elements, we’re able to make targeted optimizations, which can lead to substantial improvements to the North Star and the bottom line.
• Efficiency at scale: No more guesswork or one-size-fits-all strategies. Test multiple elements concurrently and discover what combinations of elements truly resonate and are most effective in driving results.
  • Benefit: Focus budget, time, and creative efforts on strategies and combinations of winning elements that have the highest likelihood of success.
  • Impact: This efficiency can lead to cost savings and a better ROI. Avoid spending resources on strategies that are less effective and instead allocate those resources toward initiatives that are more likely to drive growth.
• Continuous improvement: Multivariate testing is not a one-and-done process, but rather it encourages ongoing experimentation and refinement of digital marketing elements.
  • Benefit: Multivariate testing fosters a culture of continuous optimization and enables frequent refinement of the macro strategy. In the short term, this keeps a brand relevant, dynamic, and flexible.
  • Impact: In the long term, a brand’s commitment to continuous improvement sets them up for success when sudden, drastic pivots are needed. A brand that is nimble, adaptable, and open-minded will always outperform the dinosaur. The cumulative impact of ongoing optimizations will likely result in sustained growth and a competitive edge within the brand’s niche.

Getting started with multivariate testing

The end goal is to optimize every step of the customer journey to transform customer intent into loyalty. Before diving headfirst into multivariate testing, it’s crucial to set the stage for success. Start by defining your goals, whether it’s increasing click-through rates, reducing bounce rates, or boosting CVRs. These objectives will serve as your North Star throughout the experimentation process.

Building a solid foundation is just the beginning. To make the most of multivariate testing, especially within creatives, you’ll need to master the art of crafting winning hypotheses.

The anatomy of a winning hypothesis

A hypothesis isn’t a wild guess; it’s an educated prediction backed by data and insights. Crafting a well-structured hypothesis is crucial for conducting effective experiments and deriving actionable insights. Here’s what a winning hypothesis should include:

1. Observation: This is the basis for recognizing there is an opportunity for growth and begins with collecting data to support the issue at hand. This data can be either quantitative (low CVRs or high bounce rates) or qualitative (low NPS score or user feedback indicating dissatisfaction with the product).
   1. Example: The landing page’s conversion rate has decreased -20% this quarter which has driven a +20% increase in CPA.
2. Statement of the problem: Clearly define the challenge the experiment should attempt to resolve.
   1. Example: The decreasing conversion rate suggests that the value proposition (Convenience & Simplicity of Use) does not resonate with the audience on a larger scale.
3. Identification of the variable: Choose the specific elements that you plan to change or test. These variables are the ones that you suspect may have an impact on the problem you’ve identified.
   1. Example: We will test different versions of a landing page, which will highlight unique value props for the product. 
4. Expected outcome: State the expectation about how the changes to the variable will impact the problem. This should be a prediction, based on data from the observation above, on what you anticipate will happen if you make these changes. This is the heart and boiled-down version of the hypothesis.
   1. Example: We hypothesize that revising the landing page with a new value proposition will resonate more with the target audience and will lead to an increase in the landing page’s conversion rate.
5. Rationale or reasoning: Leverage data, insights, and observations on the context behind why you believe the hypothesis is valid.
   1. Example: Our hypothesis is based on the analysis of user feedback, which indicated that the current value proposition, Convenience & Simplicity of Use, isn’t compelling enough to differentiate the product against competitors.
6. Specify variations: Clearly define the variations that will be tested. Since this is multivariate testing, this will involve multiple versions of the variable you’re changing.
   1. Example: We will test three different value propositions: one emphasizing the product’s effectiveness, one emphasizing the physical health gains from the product, and one emphasizing mental health gains from the product. We will also perform a heatmap analysis of the existing page & ensure all AIDA best practices are being met on the landing pages.
7. Audience: Specify the target audience for your experiment. This can be determined via demographics, behaviors, or preferences, or pre-defined personas that will be tested across the board. It’s also possible
   1. Example: We will test the three landing pages to our 3 core target personas: the money-conscious buyer, the health-conscious buyer, and the time-conscious buyer.
8. Metrics for evaluation: Specify the OKRs or KPIs that will be used to measure the impact of the changes. The results of these metrics will determine whether the hypothesis is valid or null.
   1. Example: The key indicator of success for this experiment will be the impact on conversion rate, though we will also use avg. time on page and CTR to sign-up page as secondary metrics and performance indicators.
9. Timeline and testing plan: Identify how long the test will run and the number of samples or users you aim to include. This should also include minimum conditions for success, such as an impression, spend, or visitors threshold, which ensures statistical significance.
   1. Example: We will run the multivariate test for four weeks with an automatic, randomized 33% split between landing page variations. We aim to gather data from at least 5,000 unique visitors.
10. Success criteria: Define what success looks like for the hypothesis to be categorized as either valid or null.
    1. Example: We will consider the test a success if any of the value proposition variations result in a statistically significant increase of at least +10% in conversion rate compared to the existing landing page. 
11. Contingency plan: Provide clear next steps if the hypothesis is proven to be null/incorrect.

a. Example: If none of the variations show a significant improvement in conversion rate, we will reevaluate our approach and explore other value propositions to test.

Now, let’s explore a case study on how to effectively apply multivariate testing to creatives.

Case study on multivariate testing within creatives

There is an AI-driven B2B brand that specializes in SMS marketing. They have a few theories on who their target customer and winning value proposition will be; however, they lack data to inform these macro-level decisions.

As part of the NoGood growth strategy development, we outlined the following macro-level multivariate experiment that will run across paid social platforms, paid search, within core landing pages for paid, on organic social, and within incoming lead nurturing email flows.

Within paid media specifically, we outlined the following hypothesis:

1. Observation: After completing the value proposition and persona exploration with the client, we have identified 2 personas to test against 4 value propositions. There is no prior data to inform performance.
2. Statement of the problem: There is a lack of concrete data, learnings, and internal alignment about the brand’s ideal customer and the value proposition that resonates best with their needs.
3. Identification of the variable: We will test four versions of a paid social creative that will run on LinkedIn and Facebook to the 2 target audiences (personas). Each version of the creative will highlight the 4 potential value props for the product. From a visual perspective, each asset needs to reflect the value prop in a way that is unique and visually different from other assets. For example, the main creative element (comparison chart) cannot be the same across creative variations because the algorithm is not intelligent enough to differentiate them as different assets.
4. Expected outcome: There are three micro experiments at play as part of this multivariate test. We hypothesize that we’ll be able to identify:
   1. The top-performing persona by overall conversion rate
   2. The top-performing value proposition by overall conversion rate
   3. The top performing value prop for each individual persona by conversion rate
5. Rationale or reasoning: Our hypothesis is based on the result of the value proposition and persona exploration with the client, which is based on the brand’s strategic vision and goals.
6. Specify variations: We will test four different value propositions: Personalized Messaging at Scale, Simplify & Streamline, Increase Customer LTV, and Decrease Churn within creative assets on paid and organic social against two personas: the Acquisition Marketer and the Retention Marketer.
7. Audience: We will test the four creative assets against two target personas: the Acquisition Marketer and the Retention Marketer. We will leverage behaviors, interests, and in-platform engagement to build these audiences. Geographic and age targeting will be consistent across both groups.
8. Metrics for evaluation: The key indicator of success for this experiment will be the impact on the lead conversion rate on the website. We will also use in-platform CTR (based on all clicks) and the ratio for users who initiated the lead form and users who completed the lead form as secondary metrics and performance indicators.
9. Timeline and testing plan: Identify how long the test will run and the number of samples or users you aim to include. This should also include minimum conditions for success, such as an impression, spend, or visitors threshold, which ensures statistical significance.
   1. Example: We will run the multivariate test for four weeks. Each persona (ad set) within paid social campaigns will receive the same level of ad spend and launch timeline. For organic social, we aim to gather data from at least 1,000 post views per asset, given the existing follower size of 3,000.
10. Success criteria: We will consider the test a success if any of the value proposition variations result in a statistically significant increase of at least +10% in conversion rate compared to the existing landing page. All creative variations within each audience ad set need to achieve 50K impressions to ensure statistical significance for paid campaigns.
11. Contingency plan: If none of the variations show a significant improvement in conversion rate, we will reevaluate our macro-level strategy for persona and value proposition testing. We will also evaluate the performance of the core landing page using a heatmap and click behavioral analysis to determine the next steps for the website.

Harness the power of multivariate testing tools

For landing page experimentation, we recommend tools such as Optimizely, VMO, and CrazyEgg, which make multivariate testing a breeze. These platforms enable the growth marketer to set up experiments, track results, and draw insights, all while ensuring statistical significance that will propel your growth strategy forward.

For creative experimentation, there are tools such as Marpipe that allow brands to do relevant multivariate testing to the nth degree. However, we have seen mixed results with these approaches because tools like these allow for such a detailed level of customization that detracts from achieving statistically significant findings. In general, we recommend leveraging multivariate testing for macro-level experiments such as persona testing, value prop testing, and visually different creative themes (i.e. comparison chart vs. testimonial). This will restrict the team from going overboard on launching too many micro variations that can muddle results on a larger scale.

Key takeaways

Mastering creative multivariate testing at scale is vital for growth marketers and sets the stage for innovation, adaptability, and sustained growth and success. It involves simultaneously testing multiple variables across various growth channels and campaigns to optimize user experiences and boost conversion rates. This approach can benefit startups searching for their unique value proposition, CMOs shaping macro-level strategies, and individual contributors executing the vision.

Multivariate testing offers precision optimization and efficiency and fosters a culture of continuous improvement, resulting in cost savings and sustained growth. To start, define clear goals and craft well-structured hypotheses based on data and insights. However, use caution when crafting multivariate tests, as overly detailed customization can dilute results, especially within creative experimentation. Instead, focus on macro-level experiments such as persona and value proposition testing, ensuring statistically significant findings that drive growth.