Explain Like I am 5 | Day 02: Randomization
Explain Like I’m Five Series | Week 01 | Lesson 02 | Randomization. One core experimentation concept, explained with clarity and practical intuition.
In the series “Explain like I’m 5” we are breaking down A/B testing topics that can be understood by even a 5-year-old.
👋 Hey! This is Manisha Arora from PrepVector. Welcome to the Tech Growth Series, a newsletter that aims to bridge the gap between academic knowledge and practical aspects of data science. My goal is to simplify complicated data concepts, share my perspectives on the latest trends, and share my learnings from building and leading data teams.
In the last post, we explained A/B testing through the example of finding which cookie kids prefer plain or chocolate chip.
You learned that A/B testing helps you make decisions based on evidence, not intuition. But the evidence can also signal a false conclusion if the concept of randomization is not taken care of during the experiment.
Today we’ll cover randomization as a concept that a five-year-old can understand.
In the last post, you learned about A/B testing by distributing chocolate chip cookies and plain cookies to your classmates on your birthday celebration. You concluded that since chocolate chip cookies finished first, they are preferred more among kids.
Now it’s evening, and you’re about to throw a party for your friend from next door. Your mom asks, Should we serve plain cookies or chocolate chip cookies? You answer confidently, “Chocolate chip cookies!” You’re sure about this because you learned from A/B testing that chocolate chip cookies are the favorite.
During the party, your mom offered
The plain cookies plate to the older kids
The chocolate chip cookies plate to the younger kids.
You observed that the plain cookies get finished first.
Now you’re confused. This is the opposite of what you learned this morning! You asked why did this happen? Your mom smiles and explains a new concept: Randomization
Here are two scenarios
Was it because the kids like plain cookies more?
Was it because the older kids have bigger appetites and ate more cookies? That’s why the plain cookies finished first.
Both the cases are possible.You don’t know which one is true, and that’s the problem.
The Randomization
When we run an A/B test, we have to randomize the groups. Here one group had older kids, and the other had younger kids. Because the groups were different in age, we can’t trust our conclusion.
This morning at school, your classmates were all from the same class. Both groups had similar appetites and similar characteristics. So you can trust the result that the chocolate chip cookies are liked by kids more.
That’s why randomization is so important.
Randomization ensures that both groups you divide are truly random and share the same properties.
Example: When you randomly split users or traffic between Version A and Version B of call to action button, you remove any hidden bias like age, intent, past behavior, or preferences. Without randomization, you can’t tell if the results came from the change, you tested or from some other factor. If you only show your
Red button (Version B) to tech-savvy users
Blue button (Version A) to older users
You won’t know if higher clicks are because the red button is better or because tech-savvy users click more often anyway.
Randomization lets you see the real difference, free from the influence of external factors.
Some common types of Randomization
User ID based Randomization: The user sees the same experience across different sessions and devices
Cookie/Device based Randomization: When a user is anonymous (not logged in), randomization is tied to a browser cookie or a device ID.
Session-Based Randomization: A user is re-randomized every time they start a new session.
Cluster (Group) Randomization: Instead of randomizing individuals, you randomize entire groups or “clusters.” In a ride-sharing app, you might randomize by City
Quick Question:
What if you want to test more than 2 options or interactions of multiple features together? Stay tuned for the answer
If you’d like to dive deeper into experimentation, here are a few of our learning programs you might enjoy:
A/B Testing Course for Data Scientists and Product Managers
Learn how top product data scientists frame hypotheses, pick the right metrics, and turn A/B test results into product decisions. This course combines product thinking, experimentation design, and storytelling—skills that set apart analysts who influence roadmaps.
Advanced A/B Testing for Data Scientists
Master the experimentation frameworks used by leading tech teams. Learn to design powerful tests, analyze results with statistical rigor, and translate insights into product growth. A hands-on program for data scientists ready to influence strategy through experimentation.
Master Product Sense and AB Testing, and learn to use statistical methods to drive product growth. I focus on inculcating a problem-solving mindset, and application of data-driven strategies, including A/B Testing, ML, and Causal Inference, to drive product growth.
Not sure which course aligns with your goals? Send me a message on LinkedIn with your background and aspirations, and I’ll help you find the best fit for your journey.