Explain Like I am 5 | The Final Chapter: 31 Days of A/B Testing

Explain Like I’m Five Series | Series Conclusion | The complete index, a thank-you, and everything you need to bookmark

30 days ago, we started with a simple question: “What if we explained A/B testing the way you’d explain it to a five-year-old?”

No jargon. No intimidating formulas upfront. No assumed knowledge.

Just cookies, chocolates, toy mysteries, birthday parties, basketball shooters, and a lot of really good analogies.

We went from “what even is A/B testing?” all the way to “Closing the loop with A/B Testing” and covered everything in between. The series was intentionally slow and intentionally simple. Because the fundamentals are where most people skip ahead — and then get lost later.

---

👋 Hey! This is Manisha Arora from PrepVector. Welcome to the Tech Growth Series, a newsletter that aims to bridge the gap between academic knowledge and practical aspects of data science. My goal is to simplify complicated data concepts, share my perspectives on the latest trends, and share my learnings from building and leading data teams.

---

Where We Started

The series kicked off with the very basics, the why before the how. What is an experiment? Why do we need a control group? What is a hypothesis?

It was intentionally slow, intentionally simple. Because the fundamentals are where most people skip ahead and then get lost later.

How Far We’ve Come

Look at what you now understand:

From hypothesis to statistical design, from metrics to multiple testing and closing the loop. That’s not a beginner anymore. That’s someone who can sit in an experimentation review and actually create value from A/B Testing.

The Complete Series Index

Here’s every lesson, in order. Bookmark this. Share it. Come back to it whenever you need a refresher.

Foundation (Days 1–7)

• What is A/B Testing?

• Randomization

• What is A/B/n Testing and Multivariate Testing

• Null & Alternate Hypotheses

• Minimum Detectable Effect (MDE)

• P-Values

• Week 1 Summary

Going Deeper (Days 8–14)

• Level of Significance

• Type I & Type II Errors

• Statistical Power

• Confidence Intervals

• Sample Size Calculation

• Duration of Experiment

• Week 2 Summary

Metrics & Design (Days 15–21)

• Primary Metrics

• Secondary & Guardrail Metrics

• Metrics Framework

• Sample Ratio Mismatch (SRM)

• Chi-Square Test

• t-test

• 21-Day Summary

Advanced Traps & Pitfalls (Days 22–26)

• Peeking Problem

• Statistical vs. Practical Significance

• Simpson’s Paradox

• A/A Testing

• Multiple Testing Problem

• Handling Multiple Testing Problem

• Network Effects

• Novelty Effects

• Closing the loop with A/B Testing

What You Now Know

If you followed this series from the start, you can now:

• Design an A/B test from scratch — hypothesis, metrics, sample size, duration

• Identify what can go wrong before the test runs - SRM, randomization, novelty effects

• Read results clearly — by peeking, multiple testing, or Simpson’s Paradox

• Tell the difference between a win that’s statistically real and one that’s practically meaningful

• Know when a t-test applies vs. a chi-square test

• How to handle multiple testing problem, novelty effect and close the loop with A/B Testing

That’s not a five-year-old’s curriculum anymore. That’s the foundation of a working data scientist or PM.

A Note of Genuine Gratitude

This series was a commitment on our end, and yours.

You showed up. You asked questions, sent DMs. You shared posts with teammates who needed to finally get what p-values mean.

Every subscriber who replied. Every share. Every comment that said “this finally made it click.” You validated that the ELI5 format works that great explanations don’t need to be complicated to be rigorous.

Thank you for reading, for sticking around, and for trusting PrepVector with your learning time.

How to Use This Post

• Bookmark it. Come back when you need a refresher before an interview, a presentation, or when onboarding someone new to experimentation.

• Share it. If you know a PM, data scientist, or analyst who’s navigating A/B testing for the first time this is the series to send them.

• Save it. This is your cheat sheet for the full A/B Testing curriculum

What’s Next?

We’ll keep writing about experimentation, data science, AI, and everything in between. If you haven’t already, subscribe so you don’t miss what’s coming.

Thank you for joining us on this 30-day journey from A/B Testing basics to advanced experimentation concepts your support, questions, and engagement turned simple analogies into a complete roadmap for becoming truly confident in experimentation.

30 days, countless analogies, and one mission making A/B Testing finally click; thank you for learning, sharing, and growing with us through every experiment.

---

If you’d like to dive deeper into experimentation, here are a few of our learning programs you might enjoy:

1. A/B Testing Course for Data Scientists and Product Managers

   Learn how top product data scientists frame hypotheses, pick the right metrics, and turn A/B test results into product decisions. This course combines product thinking, experimentation design, and storytelling—skills that set apart analysts who influence roadmaps.

2. Advanced A/B Testing for Data Scientists

   Master the experimentation frameworks used by leading tech teams. Learn to design powerful tests, analyze results with statistical rigor, and translate insights into product growth. A hands-on program for data scientists ready to influence strategy through experimentation.

2. Product Data Science

   Master Product Sense and AB Testing and learn to use statistical methods to drive product growth. I focus on inculcating a problem-solving mindset, and application of data-driven strategies, including A/B Testing, ML, and Causal Inference, to drive product growth.

Not sure which course aligns with your goals? Send me a message on LinkedIn with your background and aspirations, and I’ll help you find the best fit for your journey.