A/B Testing For Beginners: Everything You Should Know To Get Started

16 min read

If you’re running a small business, then you know that every penny counts. You can’t afford to waste money on ad campaigns that don’t work, or settle for a website that’s not converting visitors into buyers.

That’s why A/B testing is so important—it helps you make decisions about your website, email campaigns, and ad campaigns that could lead to more sales with minimal investment.

In this article, we’ll explain what A/B testing is, how to get started, and some of the benefits of using this simple but effective marketing tool.

What Is A/B Testing?

Although it sounds rather technical, A/B testing isn’t overly complicated or difficult. As long as you know what it entails, you should be able to do effective testing in no time.

So, what is it, exactly?

A/B testing, also known as split testing, is a method of comparing two versions of a web page or content to see which one performs better. You usually only want to change one aspect of your website to get an accurate test.

First, you want to develop a hypothesis about whether a specific element of your website or content would affect your ability to retain customers. Then, you run tests to determine if your hypothesis was accurate.

For example, you could create two different designs for a landing page and send traffic to both pages equally. By tracking how each version performs, you can determine which one is more effective. You can then make decisions based on the data you collected.

Source: Towards Data Science

A/B testing helps identify the effective elements in your marketing strategies. From your website design to your email marketing, it is the best way to find what works for your target audience.

Once you’ve done your tests, you will analyze the results to determine where you stand with your various ToFu strategies (aka top-of-the-funnel strategies, or strategies used to draw in traffic from search engines and social media).

What Can You Use A/B Testing For?

You can use A/B testing for just about anything marketing-related. It allows you to better understand consumers’ experiences and use data to connect with your audience.

After testing, you will have the information you need to make improvements to your marketing. For example, depending on the results, you’ll know how best to change your website’s layout, adjust email content, or switch to a color palette that your audience finds more attractive.

A/B testing helps you improve your conversion rates by providing in-depth data that can inform your decision.

A/B testing can also help you:

Choose the best appearance (color, placement, word style) of a call-to-action (CTA) button
Select the right subject line for an email announcing a sale
Decide which product description best incentivizes a purchase
Find the right place for your “Share to social media” buttons for maximum engagement
Select which photos to use for Instagram ads
Test your slogan or catchphrase to determine if it’s representing your brand

An example of testing different subjects for a promotional email campaign

A/B testing is incredibly flexible and can help you assess a variety of marketing techniques to best suit your business.

How to Conduct an A/B Test

The following steps will guide you on how to start A/B testing. You can use these steps to make your own tests and apply the results to your business.

Step 1. Define Your Variables

The very first step of an A/B test is clearly determining what you want to assess. The first question is, will this be an off-site or on-site test?

On-site tests include all the elements of your website that are sales-related. For example, you can test your CTA text, the placement of your CTAs, headlines, images, video content, pop-ups, potential domain names, and more.

Off-site tests look at the effectiveness of advertisements and sales emails. You will do this kind of test to determine if your ads and emails drive traffic and result in conversions.

Deciding what exactly you need to test depends on your current goals. What do you want to improve? For example, if you’re not satisfied with your last advertising campaign, you can test new ad creatives to improve the performance of your marketing campaigns. Or, if you’re redesigning your website, you can test different home pages to see which one makes visitors spend more time on the site.

Step 2. Come Up With a Hypothesis

Now that you know what variables you’re going to test, it’s time to create a hypothesis. Think about what changes you can make to get the results you want.

Make a list of everything you think you can do better and the ways you can improve. Should you write better CTAs? Can your emails use more images? Should your website have a different layout?

After you come up with different hypotheses, you need to prioritize them. Identify the best and most important ones. Think about how you can execute your A/B tests to test them. Also, consider how difficult they will be to implement and their potential impact on customers.

Finally, you need to decide how your A/B test will run. For example, when testing emails, you’ll need to send out two different versions and track which version gets the best results. For this, identify which email elements you’re going to test, such as the subject line, copy, images, etc. Then consider measurement metrics like open rate or click-through rate (CTR) to differentiate and compare versions.

Step 3. Set a Time Limit

You also have to decide how long to run the A/B test. This isn’t something that someone else can decide for you — you’ll have to learn on your own intuition and find the time frame that works best for you.

Generally, A/B tests for email campaigns can run from two hours up to a day, depending on how you determine a winning version—typically based on opens, clicks, or revenue. It is recommended to wait at least two hours to determine a winner based on opens, one hour to determine a winner based on clicks, and 12 hours to determine a winner based on revenue.

An example of setting up a test in Mailchimp to compare what email content drives more revenue

For ads, you should run the campaign for a minimum of 7-day tests, because shorter tests may produce inconclusive results. For Facebook ads, you can run A/B tests for up to 30 days.

When it comes to websites, recommendations vary, suggesting you should run A/B tests for one week up to a month. Keep in mind the difference between shopping behavior during the weekend and weekdays before making a decision.

If you’re just getting started with A/B testing and are not sure how long your test should run, you can use an A/B test duration calculator like this one. After you run a few tests, you will get a better idea of the ideal time limit for each type of test.

Step 4. Test Each Variable Separately

Once you have determined which variables you want to test, you should narrow it down to only one. You will test the variable by creating two alternatives. You will test these against each other.

If you have multiple elements of a campaign or website to test, always run one test at a time.

It’s better to run A/B tests separately rather than running them all simultaneously. Testing too many variables at once will make it difficult to determine which parts were successful or not. By only changing one variable while keeping the rest constant, the resulting data will be easy to understand and apply.

Step 5. Analyze Results

Your goals will determine how you analyze the results of your A/B test. For example, if you want to test ways to increase your website traffic, you should test blog post titles and webpage titles. After all, titles should grab someone’s attention and make them want to learn more.

Every variable you test for will have different metrics, and produce different results. Here are a few examples of potential goals and variables to change in your A/B test:

Conversion rate improvement (you can change CTA text, colors, and element placement)
Bounce rate reduction (test product descriptions, fonts you use in listings, and featured images)
Website traffic boosts (change the placement of links)
Lower cart abandonment rates (use various product photos)

You can also break down your results by different segments of your audience. You can determine where your traffic comes from, what elements work best for mobile vs. desktop users, how new visitors are attracted, and more.

Your options are almost limitless:

Source: Right On Interactive

Not sure about the test results you got? One way you can see the accuracy of your tests is through customer feedback. After changing your marketing based on your findings, embed a survey form on your website to receive feedback from your audience to see if they enjoy the changes you made.

Step 6. Adjust and Repeat

The work doesn’t stop once you’ve got all your analytics neatly laid out. Now, you have to test again. Make more changes, run more tests, and learn from the new data.

Of course, you don’t have to run A/B tests one after the other. Instead, give yourself time to learn from the data you’ve gathered and develop creative ways to adjust your approach before you release a new test.

A/B Testing Best Practices

Knowing how to run A/B tests and doing them right are two different things. We’ve briefly listed A/B testing best practices below so you won’t have to learn the hard way.

Use a Standard Time Limit and Sample Size

You can easily get lost while running an overly lengthy test, or collecting too much data. Results get muddled if you don’t have a standard time limit and sample size for all your A/B tests.

However, it’s a good idea to keep in mind that you should focus on an actionable number. For a small business, a solid number for conversions would be 1,000 subjects. But for some brands, it might be closer to 5,000.

Remember that this number will be split in half because you’re testing two versions. You shouldn’t let the number of test subjects for each version drop below 500 because you might not have enough data to analyze.

Recommended Tools for A/B Testing

Depending on which variables you choose to test, you can use these tools for A/B testing:

Google Optimize
Freshmarketer (part of Freshworks)
Optimizely
Omniconvert
Crazyegg
VWO
A/B Tasty
Adobe Target

If you’re just getting started with A/B testing for your websites, Google Optimize might be the only tool you need. It’s easy to navigate for first-time users and it synchronizes with Google Analytics. The free version is robust enough for beginners, with a simple setup and intuitive interface.

As for testing email and ad campaigns, most platforms have built-in tools for A/B testing, which is more than enough for beginners. For example, you can run A/B tests if you create your email in Mailchimp. Facebook and Instagram will also run tests for you if you choose to advertise with them. These services will notify you about the results, which will be stored in a report that you can check out later.

Testing different customer testimonials for an ad copy on Facebook

Learn more: How to Advertise a Business on Facebook for Beginners

Keep Your Data to Revisit Later

All the hard work you put into your A/B tests will be wasted if you don’t store the data in a way that you can access it again later.

Though you may not consider all test results useful, you might one day. For example, a marketing channel you didn’t consider important before may become an important lead generation tool in a few years. In that situation, you’ll want to have as much data about that channel as possible—even minor details that may have seemed irrelevant when you originally ran the test.

Save your test results so you can refer to them later. You could track the results from each test in Google Sheets, or any other spreadsheet program, like Microsoft Office, LibreOffice, or WPS Office. For safety’s sake, use cloud-based storage to store your data so you don’t lose it if one of your devices breaks.

No matter the application you decide to use, keeping track of your data is key to making a customer-approved website.

A word to the wise, use good data synchronization to avoid getting confused or risk data getting lost.

You, Too, Can Run Effective and Comprehensive A/B Tests

There you have it—advice to get you started with strong A/B tests that will quickly help your business. Remember that your business is unique, and the knowledge shared here only gives you a template to work from. Use our steps to build the best A/B tests for you and your goals, even if you’re not a marketing guru.

Sell online

With Ecwid Ecommerce, you can easily sell anywhere, to anyone — across the internet and around the world.

Get started

Marketing and Promotion

Apr 8, 2022

About the author

Mark Quadros is a SaaS content marketer that helps brands create and distribute rad content. On a similar note, Mark loves content and contributes to several authoritative blogs like HubSpot, CoSchedule, Foundr, etc. Connect with him via LinkedIn or Twitter.