The Experimentation Playbook - Holiday Edition

A comprehensive, practical guide to experimentation for product and data teams, available this holiday season at a discounted price of $100 (original price $300)!

The Holiday Sale Is Live!

Over the past few years, I’ve spoken with so many data scientists, product managers, and analysts who all share a similar experience:
Running experiments is powerful but not always straightforward.

Teams often ask the same questions:

• How do we pick the right design?

• What MDE makes sense?

• How do we interpret results with confidence?

• How do we communicate readouts clearly to stakeholders?

  

To make this easier, I’m launching ‘The Experimentation Playbook’. It is a 6-part series that will serve as a structured, end-to-end guide on running reliable, meaningful experiments to drive product growth.

And since we’re heading into the holiday season, I’m making it available at a special launch price of $100 (original price: $300) . Discount details at the end! 🎉🎉

About the Authors

🔹 Manisha Arora

A data science and analytics leader with 12+ years of experience building and scaling data science solutions across high-impact products. She currently leads Data Science at Google, driving measurable growth for top advertisers and shaping the vision and strategy for Ad Measurement and Experimentation. Passionate about democratizing data science, Manisha also leads PrepVector, helping professionals level up through coaching, community, and industry-focused learning.

🔹 Banani Mohapatra

A seasoned data science product leader with 13+ years of experience across e-commerce, payments, and real estate. She currently leads a data science team for Walmart’s subscription product, driving growth while supporting fraud prevention and payment optimization. Known for translating complex data into strategic solutions that accelerate business outcomes.

What’s Inside the Experimentation Playbook

This playbook is structured as a 6-edition master guide, giving you an end-to-end system for designing, running, and analyzing clean, high-signal experiments.

Here’s a high-level look at the six editions:

📘 Part 1: Introduction to Experimentation

A modern foundation for understanding experimentation and why it is so important and prevalent in tech. You will learn about the Experimentation Lifecycle and how it drives confidence in product decisions.

📘 Part 2: Designing High-Quality Experiments

This part walks through everything that should happen before a test goes live. We discuss forming a clear hypothesis, picking the right metrics,selecting the right experimental design and defining statistical parameters.

📘 Part 3: Running Experiments in Production

Once the experiment is live, the complexity shifts to operations. This part covers validating logging setups, ensuring assignment consistency, preventing data contamination, and monitoring metrics in-flight without introducing peeking bias. I

📘 Part 4: Analyzing Experiment Results

Here, we get into the heart of interpreting outcomes correctly. This includes understanding p-values, confidence intervals, effect sizes, variance, statistical vs. practical significance, and how to handle guardrail failures.

📘 Part 5: Challenges of Experimentation

This part is a realistic look at the messy parts of experimentation that don’t show up in textbooks. It covers understanding errors, noise, risk, and bias in experiments. It highlights how these challenges show up in practice and offers strategies for reducing risk.

📘 Part 6: Industry Case Studies

In this part, we breakdown some of the iconic experiments from the industry.

• Google: 41 shades of blue

• Bing: Color palette redesign

• Meta: Explore Feed test

You’ll learn what these companies tested, why they tested it, what happened, and what the industry learned from their wins and failures. The goal is to show how theory meets reality: experiments that contradicted intuition and tests that “failed” but led to better decisions.

Who This Playbook Is For?

This guide is ideal for anyone shaping product outcomes:

• Product Managers who want clearer, faster decisions

• Data Scientists who want rigorous, interpretable experiments

• Analysts building causal intuition

• Engineers involved in feature rollouts and experiment platforms

Holiday Discount! 🎊

The end of the year is when many people take time to reflect, reset, and invest in their skills. This playbook is built for those who want to enter the next year with a stronger, clearer understanding of experimentation

🎁 For December 2025, get the full 6-part Experimentation Playbook for $100 (normally $300) using the code: PREPVECTORHOLIDAYSALE

🎁 A Special Note for PrepVector Alumni

If you’re a PrepVector alum, this playbook is free for you!

You’ve already invested the time and effort to build strong experimentation skills through our courses, and this playbook is my way of saying thank you for being part of the community.

Send us a note at operations@prepvector.com and we will share a 100% discount code for you to get access to this playbook for free. Wishing you a season full of learning, clarity, and fun! Happy Holidays!

Want to go deeper? Join the Advanced AB Testing Course!

If you want to go beyond concepts and get hands-on with real-world experimentation problems, I also teach an Advanced A/B Testing Course.

It’s built for mid–senior professionals who want to:

• Advanced designs like switchbacks, geo-tests, CUPED and more

• Defining the right metrics

• Navigate ambiguity and conflicting signals

• Build strong experimental judgment and product intuition