Explain Like I am 5 | Day 03: What is A/B/n Testing and Multivariate Testing
Explain Like I’m Five Series | Week 01 | Lesson 03—What is A/B/n Testing and Multivariate Testing? One core experimentation concept, explained with clarity and practical intuition.
In the series “Explain like I’m 5” we are breaking down A/B testing topics that can be understood by even a 5-year-old.
👋 Hey! This is Manisha Arora from PrepVector. Welcome to the Tech Growth Series, a newsletter that aims to bridge the gap between academic knowledge and practical aspects of data science. My goal is to simplify complicated data concepts, share my perspectives on the latest trends, and share my learnings from building and leading data teams.
In the last post, we explained how randomization is important in A/B experiments and how it removes the biasness.
A/B/n Testing: More choices to test
But today, you have a new question for your mom: “What if I have more than two options? I have oatmeal cookies, chocolate chip cookies, and sugar cookies. How do I figure out which one the kids will like best?”
Your mom smiles at your curiosity. “Great question!” she says. “You follow the same A/B testing approach, but with more versions.” Here’s what you do:
Give 20 kids oatmeal cookies
Give 20 kids chocolate chip cookies
Give 20 kids sugar cookies
Observe which plate finishes first. When you run the experiment, you discover that the oatmeal cookies disappear fastest.
What you just did is called A/B/n testing. Instead of testing two options, you tested three. A/B/n testing is simply A/B testing with more than two choices. It’s still the same idea to compare multiple versions and see which one wins.
In Data Science Terms
A/B/n Testing is an extension of A/B testing where we test more than two variants simultaneously.
Instead of comparing Version A against Version B, we compare multiple versions to find which one performs best.
Example: Google tested 41 different shades of blue for hyperlink text, which generated an estimated $200 million in additional annual ad revenue.
Multivariate Testing: Interaction of multiple factors
A few days later, you have another thought. You realize that cookies have two different things that could matter: texture and flavor. Kids have multiple preference like soft chocolate cookies or crunchy chocolate cookies or soft oatmeal cookies or crunchy oatmeal.
How do you know what’s really preferred, the texture or the flavor? This time, you’re not just changing one thing. You’re changing two things at the same time.
Your mom appreciates the question and explains how to test this:
Give 15 kids soft chocolate cookies
Give 15 kids soft oatmeal cookies
Give 15 kids crunchy chocolate cookies
Give 15 kids crunchy oatmeal cookies
You run the experiment and discover something interesting: the crunchy chocolate cookies finish fastest. But the crunchy oatmeal cookies aren’t popular at all.
By itself, crunchy texture doesn’t always win. And by itself, flavor doesn’t always win. But when you combine crunchy with chocolate, kids love it. The texture and flavor work together in ways we wouldn’t have discovered by testing them separately.
When you test two or more features together like this where the changes can interact with each other that’s called multivariate testing.
In Data Science Terms
Multivariate Testing is used when we want to test multiple variables at once to find the best combination. It reveals how different changes interact with each other.
Example: Imagine an online store wants to improve their product page. They have two variables to test:
Variable 1 (Button Color): Blue or Red
Variable 2 (Font Size): Small or Large
Instead of testing button color alone or font size alone, they run a multivariate test with four
Blue button + Large font
Blue button + Small font
Red button + Large font
Red button + Small font
They split their visitors equally across all four versions and measure which combination gets the most clicks and purchases. They might discover that a red button performs best with a large font but doesn’t work as well with a small image.
Multivariate testing reveals that the variables interact with each other in ways you wouldn’t discover by testing them separately.
If you’d like to dive deeper into experimentation, here are a few of our learning programs you might enjoy:
A/B Testing Course for Data Scientists and Product Managers
Learn how top product data scientists frame hypotheses, pick the right metrics, and turn A/B test results into product decisions. This course combines product thinking, experimentation design, and storytelling—skills that set apart analysts who influence roadmaps.
Advanced A/B Testing for Data Scientists
Master the experimentation frameworks used by leading tech teams. Learn to design powerful tests, analyze results with statistical rigor, and translate insights into product growth. A hands-on program for data scientists ready to influence strategy through experimentation.
Master Product Sense and AB Testing, and learn to use statistical methods to drive product growth. I focus on inculcating a problem-solving mindset, and application of data-driven strategies, including A/B Testing, ML, and Causal Inference, to drive product growth.
Not sure which course aligns with your goals? Send me a message on LinkedIn with your background and aspirations, and I’ll help you find the best fit for your journey.