What is A/B Testing?

A/B Testing

What is A/B Testing

A/B testing (also known as split testing or bucket testing) is a method of comparing two versions of a webpage or app against each other to determine which one performs better. AB testing is essentially an experiment where two or more variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal.

How A/B Testing Works

In an A/B test, you take a webpage or app screen and modify it to create a second version of the same page. Half of your traffic is shown the original version of the page (known as the control) and half are shown the modified version of the page (the variation).

As visitors are served either the control or variation, their engagement with each experience is measured and collected in an analytics dashboard and analyzed through a statistical engine. You can then determine whether changing the experience had a positive, negative, or no effect on visitor behavior.

Why You Should A/B Test

A/B testing allows individuals, teams, and companies to make careful changes to their user experiences while collecting data on the results. This allows them to construct hypotheses, and to learn better why certain elements of their experiences impact user behavior. In another way, they can be proven wrong—their opinion about the best experience for a given goal can be proven wrong through an A/B test.

More than just answering a one-off question or settling a disagreement, AB testing can be used consistently to continually improve a given experience, improving a single goal like conversion rate over time.

A/B Testing Process

The following is an A/B testing framework you can use to start running tests:

  1. Collect Data: Your analytics will often provide insight into where you can begin optimizing. It helps to begin with high traffic areas of your site or app, as that will allow you to gather data faster. Look for pages with low conversion rates or high drop-off rates that can be improved.
  2. Identify Goals: Your conversion goals are the metrics that you are using to determine whether or not the variation is more successful than the original version. Goals can be anything from clicking a button or link to product purchases and e-mail signups.
  3. Generate Hypothesis: Once you’ve identified a goal you can begin generating A/B testing ideas and hypotheses for why you think they will be better than the current version. Once you have a list of ideas, prioritize them in terms of expected impact and difficulty of implementation.
  4. Create Variations: Using your A/B testing software (like Optimizely), make the desired changes to an element of your website or mobile app experience. This might be changing the color of a button, swapping the order of elements on the page, hiding navigation elements, or something entirely custom. Many leading A/B testing tools have a visual editor that will make these changes easy. However, make sure to QA your experiment to make sure it works as expected.
  5. Run Experiment: Kick off your experiment and wait for visitors to participate! At this point, visitors to your site or app will be randomly assigned to either the control or variation of your experience. Their interaction with each experience is measured, counted, and compared to determine how each performs.
  6. Analyze Results: Once your experiment is complete, it’s time to analyze the results. Your A/B testing software will present the data from the experiment and show you the difference between how the two versions of your page performed, and whether there is a statistically significant difference.

Platforms:

Optimizely

  • #1
  • Supported by segment
  • Offers many products including Optimizely Web for A/B Testing, Optimizely rollout for feature flagging, etc.
  • Price: contact sales.
  • Behavior Testing
  • Multivariate Testing
  • Free starter plan: 1 website, 1 android app, and 1 IOS app.
  • Free 30 days trial.
  • Used by: IBM, HP, ebay, etc.

Google Optimize

  • Supported by Segment as part of google analytics
  • Free; however, Optimize 360 which is a paid plan is available too.
  • A/B testing
  • Behavior and Technology Targeting
  • GEO targeting
  • Supports mobiles via Firebase A/B Tesing
  • Multivariate testing (MVT)

VWO:

  • Supported by Segment
  • Offers many products including VWO Testing for A/B Testing, VWO Insights for analyzing, VWO FULLSTACK, etc.
  • Free 30 days trial
  • Starting from $200 for 10k monthly visitors.
  • Behavior Testing
  • Multivariate Testing
  • Used by: Ubisoft, ebay, etc.

Leanplum:

  • Supported by Segment
  • Price: contact sales
  • Free demo
  • Focus on content personalization for mobile
  • Used by: Grab, NBC, TED, etc

Apptimize:

  • Supported by Segment
  • Price: contact sales
  • Free demo
  • Mobile apps (mainly CRO)
  • Used by: runtastic, Hotel Tonight, glassdoor

To summarize:


Optimizely Google Optimize VWO Leanplum Apptimize
Segment Yes Yes Yes Yes Yes
Price Contact Sales Free – paid version: Optimize 360 $200 + Contact Sales Contact Sales
Trial 30 days Always 30 days ? ?
Demo Yes ? Yes Yes Yes
Web Yes Yes Yes No No
Mobile Yes Firebase A/B Testing Yes Only Mobile Only Mobile
Starter plan Yes Always free ? ? ?

All of the platforms are mature products with robust features, hence if you are looking for conversion rate optimization, go for Leanplum. But if you are looking for A/B testing for web app mainly and mobile apps, go for Optimizely or VWO; however, if you want to do them on mobile apps only, go for Apptimize. Optimizely and VWO are specialized for websites, while Apptimize is for mobile apps.

Read more:

Software Deployment

Docker Images

Continuous Integration & Continuous Delivery: CI/CD

Sources:

Leave a Reply

Your email address will not be published. Required fields are marked *