Explain Like I am 5 | Week 1 Summary
Explain Like I’m Five Series | Week 01 Summary | What is AB testing? What is randomization? What is A/B/n testing and multivariate testing?
In the series “Explain like I’m 5” we are breaking down A/B testing topics that can be understood by even a 5-year-old.
👋 Hey! This is Manisha Arora from PrepVector. Welcome to the Tech Growth Series, a newsletter that aims to bridge the gap between academic knowledge and practical aspects of data science. My goal is to simplify complicated data concepts, share my perspectives on the latest trends, and share my learnings from building and leading data teams.
In the last few posts we read about:
What is A/B/n Testing and Multi Variate Testing
Following is a quick revision of all three topics
What is A/B Testing?
A/B testing is an experimental method where you compare generally two versions of something (control vs treatment) to determine which performs better on a specific metric.
The objective of A/B testing is to make decision-making from data driven instead of intuition and get incremental lifts
How does A/B testing work?
The Control (A): This is your current version—the baseline.
The Treatment(B): This is the version with one specific change (e.g., a different button color, headline, or image).
The Split: Users are randomly assigned to see either A or B.
The Result: You measure which version led to more clicks, sign-ups, or purchases
Examples
A/B Testing can be useful in many areas like
To increase Click-Through Rate (CTR): A travel company can test changing a button’s text from “Submit” to “Find My Room” to see if it drives more search queries on their website.
To maximize revenue per user: A SaaS company can add a “Most Popular” badge to their Pro subscription plan to guide users toward higher-value purchases
To optimize conversion rates: An e-commerce app can compare a multi-step checkout against a single-page checkout to identify which flow minimizes cart abandonment.
To improve email open rates: Testing a personalized subject line like “{USER_NAME}, here is 20% OFF” against a generic one like “Get 20% OFF” can help determine the impact of personalization.
To reduce bounce rate: A/B testing a new homepage UI (User Interface) can help determine which design keeps users engaged for a longer duration.
The Importance of Randomization
In an A/B test, you want to compare Version A (the original) and Version B (the new idea). For the results to be fair, the two groups of people seeing these versions need to be as similar as possible. Randomization ensures that both groups have a mix of all kinds of people.
If you don’t randomize, say, if you showed Version B only to people who visit your site on weekends you wouldn’t know if the “win” was because of your new design or because weekend shoppers just behave differently.
Randomization Units (Who is being randomized?)
User-level (ID-based): Assigns a specific user to a variant across all sessions. Best for long-term behavior.
Cookie-level: Uses web cookies for tracking, common for anonymous, web-based tests.
Session-level: Randomizes each visit independently. Best for measuring immediate reactions.
Device-level: Uses device IDs, ideal for mobile apps or logged-out users.
A/B/n Testing and Multivariate Testing
Sometimes, two choices aren’t enough. What if you want to test more than 2 options or want to check the combination of changes of multiple options. Then you need upgrade from simple A/B Testing
A/B/n Testing
This is when you test more than two options at once (e.g., red vs blue vs grey vs yellow CTA button).
You split your audience into three or more equal groups to find the single best choice. It can be said as an extension of A/B testing.
The “n” in A/B/n refers to adding more versions (C, D, E, etc.). However, even with multiple versions, you are typically only changing one major variable at a time (e.g. changing colors of CTA button)
Multivariate Testing
Instead of testing one big change, you test multiple variables simultaneously to see which combination of elements works best.
For example, if you want to test two different headlines and two different hero images, an MVT will create four versions (2X2) to cover every possible combination.
It reveals interactions between elements. You might find that Headline A works great on its own, but it performs poorly when paired with Image B. MVT catches these nuances.
Few useful resources on AB Testing
What Is A/B Testing? | Coursera
What Is A/B Testing? – 365 Data Science
Few useful resources on Randomization
Understanding Randomization Units in A/B Testing for Online Experiment | by Jie Shen | Medium
Randomization: The ABC’s of A/B Testing
Few useful resources on A/B/n testing and Multi Variate Testing
What is A/B/n testing? - Optimizely
What is Multivariate Testing? - Optimizely
Multivariate Testing vs. A/B Testing: CXL’s Complete Guide
Stay tuned for more such topics!
If you’d like to dive deeper into experimentation, here are a few of our learning programs you might enjoy:
A/B Testing Course for Data Scientists and Product Managers
Learn how top product data scientists frame hypotheses, pick the right metrics, and turn A/B test results into product decisions. This course combines product thinking, experimentation design, and storytelling—skills that set apart analysts who influence roadmaps.
Advanced A/B Testing for Data Scientists
Master the experimentation frameworks used by leading tech teams. Learn to design powerful tests, analyze results with statistical rigor, and translate insights into product growth. A hands-on program for data scientists ready to influence strategy through experimentation.
Master Product Sense and AB Testing, and learn to use statistical methods to drive product growth. I focus on inculcating a problem-solving mindset, and application of data-driven strategies, including A/B Testing, ML, and Causal Inference, to drive product growth.
Not sure which course aligns with your goals? Send me a message on LinkedIn with your background and aspirations, and I’ll help you find the best fit for your journey.