Business

A/B Testing With Snapchat Ads: An Introduction

Which audience should I target? What should my ad creative look like? Does it matter which goal I optimize toward? When answering these tough questions, the most successful marketers don’t guess; they test!  
Launching effective campaigns and optimizing on Snapchat starts with making the right impression with your key audiences on Snapchat. Globally, Snapchatters hold $4.4 trillion in spending power1, and that number continues to grow - a key reason why brands and businesses are investing in Snapchat as an efficient and scalable way to grow. 
 
But how do successful advertisers reach this valuable audience effectively, while staying within budget? The answer starts with a strong ad testing strategy.   
 
A/B testing shows you what type of ads are resonating with your target audience, and why - giving you the ability to unlock maximum return on investment for your budget so you can grow your business. 
Throughout this blog post we’ll talk about why you should incorporate A/B testing into your campaign strategy on Snapchat, how to perform A/B testing on Snapchat, as well as A/B testing best practices on Snapchat.

An Introduction to A/B Testing

A/B testing is the process of putting different ads in front of a sample of viewers from your target audience, and analyzing the performance of those ads to determine what performed better. On Snapchat, you can run creative tests on an entire ad or specific aspects of it, then collect feedback to determine what creative, message, or strategy helps you reach your advertising goals. Testing to determine what ads resonate most with your key audiences unlocks the ability to fully tap into your target audience and get the most return on investment for your advertising campaigns. 
Some additional benefits of testing ad creative: 
  • Make data-driven decisions: Testing ad creative gives you data to back up your decisions. If you can prove which ad concepts are the best, you should have an easy time persuading colleagues to run with the winning ads.
  • Innovate over time: Winning ads might not always be your champion ads - consumer sentiment can change over time. Testing different messaging, around different key moments throughout the year can turn your standard, go-to copy into compelling copy depending on the season, time of year, or moment in time. Testing helps your ads evolve with the time and bring in new messages to current and future customers. 
  • Understand and segment your audiences: Once you have test results, you can dig into these performance results to see how different groups engage with your ads. These insights help you tailor your ads for specific groups, or make a single ad or campaign  that includes elements that each group responds well to.
  • Iterate quickly: Testing your ads allows you to update your ads and campaigns quickly so you don’t waste time or advertising budget on non-effective approaches. Increasing the frequency in which you collect insights empowers you to make quick decisions that are backed by data, so you can rely less on uninformed perceptions. 
So, now that you know the benefits of testing your ads, let’s walk through the testing methods you can use to A/B test your ads on Snapchat. 

Split Testing

Split testing is Snapchat’s testing feature that lets you run A/B tests on your Snapchat Ads, all through Ads Manager. With split testing, you can test different variables, such as creatives or audiences, to see which advertising strategy performs best. For example, you might want to see how targeting a Predefined Audience performs versus targeting a Custom Audience such as a Lookalike Audience. Once you’ve run your test, you can use insights from your results to create and optimize future campaigns. With Snapchat’s split testing, you can test the following variables:
  • Ad creative
  • Targeted audience
  • Ad placement
  • Advertiser goal
 
Split tests are easy to set up and cheap - as little as $20 a day for a week can help you understand what’s working, and what’s not. 

How Split Testing Works

When creating a split test within Ads Manager, ad sets will automatically be created for each variable you want to test. For example, for a creative test, these ad sets will have the same audience targeting, placement and delivery settings, but each ad set will have a different creative. Your budget is then split evenly across these ad sets for the duration of the spilt test.  
Your winning ad set will have the lowest cost per goal after the test is completed. After your test is complete, Snapchat will provide a confidence level percentage which is the probability of finding the same winner if you want to run this test again. To put it simply, the higher the percentage, the more confident Snapchat is in the results.
 

Where do I Start?

Start with an hypothesis. What do you think will drive results? For example, you might think that targeting all of the US will perform better than lifestyle targeting. Sounds like an audience test! 
Consider dedicating a portion of your budget to testing alongside a “business as usual” campaign. You can then use split testing to compare your business as usual creative, audience, goal, placement against some newer ideas. If your new ideas out-perform, “graduate” this approach to your “business as usual” campaign. 
 

How should I sequence tests?

Try starting with an audience test. Compare your default audience (e.g. all of US) to some lifestyle segments and/or a custom audience.
From there, move to creative testing. Try a few different creative approaches - some ad experts believe creative quality accounts for 65%+ of an ad’s impact. Additionally, seeing the same ad a number of times can lead to “creative fatigue,” so mixing it up can help your ad stand out. Consider running weekly creative tests to compare your current, best-performing creative to some new, wild ideas. 
Goal tests (e.g. optimizing for Impressions versus Swipe Ups when trying to lower your Cost per Site Visit), are great for squeezing every last ounce of performance from your target audience. What bidding strategy performs best? How does performance change with different values for target cost or max bid? Test it with split testing. 
Placement tests can help you understand how your ads perform when shown alongside specific types of content (e.g. between user-posted stories, content from publishers and creators, or specific types of content like beauty and fashion or gaming.)
 

How to Create a Split Test in Ads Manager

To create a split test campaign:
  1. Click the menu in the top corner and select Create Ads.
  2. Select Advanced Create.
  3. Choose an objective and toggle Create Split Test to Enabled. 
  4. Select your test variable, Creative, Audience, Placement or Goal.
  5. Input your split test details:
    1. Creative split tests: Select your ad format, placements, targeted audience, and delivery settings. Next, choose at least two creatives you would like to test. To add additional creatives to test in the Ads Queue, click Test Another Creative and input the details for each creative you’d like to add to this split test. Each creative will be placed in its own ad set.
    2. Audience split tests: Select your ad format and placement then select at least two audiences you would like to test by selecting each ad set in the Ad Set Queue. To add additional audiences, click Test Another Audience and input the details for each additional audience you’d like to add to your split test, then input your ad details. Each ad set will contain all ads implemented.
    3. Placement split tests: Select your ad format and placements then choose the audience you would like to test by selecting a different placement for each ad set in the Ad Set Queue. To add additional placements to test, click ‘Test Another Placement’ and input the details for each additional placement you’d like to add to this split test. Then input your ad details. Each ad set will contain all ads implemented.
    4. Goal split tests: Select your ad format, placements, targeted audience and delivery settings. Next, choose the goals you would like to test by selecting a different goal for each ad set in the Ad Set Queue. To add additional goals to test, click 'Test Another Goal' and input the additional goal you’d like to add to this split test. Then input your ad details. Each ad set will contain all ads implemented.
      Please note: You can only test goals that are available for the objective and ad format selected. 
  6. Submit your split test campaign for review.
 

To View the Progress of Your Split Test

  1. Log in to Ads Manager.
  2. Select an Ad Account from the dropdown in the top right corner.
  3. Click the menu in the top corner and select Manage Ads.
  4. Each test is represented as a campaign with an icon and each group is an ad set with its own ad(s).
Please note: other than the campaign, ad set, and ad name, split tests cannot be edited after they are published. If you need to end a split test early, you are able to pause the campaign but you will not be able to resume delivery.
 

Understanding Your Test Results

Now that you know how to run a split test on Snapchat, let’s go over how to interpret test results and how to run your winning ads. When it comes to your split test results, Snapchat compares the cost per goal across each of the ad sets, and the ad set with the lowest cost per goal is crowned as your winning ad set. Please note, it can take up to 24 hours after your split test has completed for Snapchat to name the winner and calculate the confidence level.
Next, let’s walk through how to interpret confidence levels associated with your split test results.

Interpreting Confidence Levels

Snapchat considers any test with a confidence level above 80% as strong evidence of a winner among the alternatives tested, regardless of the number of variations considered in the study. For confidence levels < 80%, the below thresholds can be used to determine if there is high confidence that the best performing variation in your test would continue to out-perform other variations, if the split test were re-run.
Recommended thresholds for ‘high confidence’ results by number of variations (confidence level):
  • 2 variations (≥ 70%)
  • 3 variations (≥ 60%)
  • 4 variations (≥ 55%)
  • 5 variations  (≥50%)
 

Run Your Winning Ad Set

To view your result confidence level, click the badge icon in the Ad Sets section of the Manage Ads table. To create a new campaign using your winning ad set and ad(s), click Run Winning Creative.
You'll have two options to run your winning creative:
  • Publish Campaign will immediately publish a new campaign using the winning ad set and ad(s). The ad set will use the same budget and start/end dates specified and will copy all of the other audience targeting, placements and delivery settings from your winning ad set.
  • Edit Ad Set will pre-populate the details from the winning ad set and allow you to edit before publishing the campaign. 
 

How much testing is enough?

Some of our most successful advertisers dedicate 10% or more of their advertising budget to ongoing testing. Once you get comfortable experimenting, you will continually find new variations of creatives, audiences and more that can make the difference between breaking through and not hitting the mark.
And that’s it! Now that you understand the importance of A/B testing, and have learned how to run your own A/B tests and act on your results, you’re ready to further optimize your campaigns and have more confidence in your advertising strategy on Snapchat.  
For more information on testing, check out these guides from Harvard Business Review and the Boston Consulting Group on how successful businesses adopt a test-and-learn mindset.
Become a data-driven advertiser with split testing today!
1 2021 Global Cassandra Study commissioned by Snap Inc. | Base: Those with a monthly income N=12,710 respondents. Cumulative total of average by market. Spending power for the Snapchat Generation was calculated as follows: We obtained the average monthly income for each respondent. The sum of any expenses related to rent or mortgage, debt, insurance, medical expenses/healthcare, spending money given to others, and utilities was subtracted from this average monthly income. We then multiplied each respondent’s average monthly discretionary spending by 12 to obtain their average yearly discretionary spending and averaged these results within each country across the Snapchat Generation.