Recap of PrepVector's A/B Testing Weekend Workshop
A deep dive into our A/B Testing Weekend Workshop—key learnings, mentor insights, and how experimentation transforms data scientists into strategic product partners.
Last week, we successfully wrapped up the first session of our AB Testing for Data Scientists & Product Managers — and it was an exciting milestone for us at PrepVector. Over two days, we brought together a vibrant group of learners eager to deepen their skills in experimentation and data-driven decision-making.
This was more than just a course. It was about building a community that values curiosity, experimentation, and rigor.
👋 Hey! This is Manisha Arora from PrepVector. Welcome to the Tech Growth Series, a newsletter that aims to bridge the gap between academic knowledge and practical aspects of data science. My goal is to simplify complicated data concepts, share my perspectives on the latest trends, and share my learnings from building and leading data teams.
About Workshop Mentors:
🔹 Banani Mohapatra — A seasoned data science product leader with 13+ years of experience across e-commerce, payments, and real estate. She currently leads a data science team for Walmart’s subscription product, driving growth while supporting fraud prevention and payment optimization. Known for translating complex data into strategic solutions that accelerate business outcomes.
🔹 Manisha Arora — A data science professional with 12+ years of experience leading teams and driving business growth through data-driven decision making. Passionate about democratizing data science and enabling others to level up in their careers.
We concluded the workshop with an engaging discussion among Manisha, Banani, and Siddarth Ranganathan (who joined as our guest speaker) on the skills required to excel as a Product Data Scientist and to nail data science interviews.
A big thank you once again to Banani and Siddarth for bringing in such a wealth of knowledge and for inspiring learners to think beyond frameworks and focus on impact.
📚 Course Recap: Key Learnings
Across two days, participants progressed through the end-to-end experimentation lifecycle, building both conceptual understanding and practical skills:
🧪 The Role of Experimentation
We began by establishing why experimentation is critical in product and data contexts. A/B testing was introduced as the benchmark methodology for validating ideas, enabling organizations to separate assumptions from measurable impact.
📊 Designing Robust Experiments
The sessions then focused on how to design experiments that drive actionable insights. Participants learned to formulate clear hypotheses, select appropriate evaluation metrics, and structure tests that scale to real-world product environments. Case studies from leading organizations such as Netflix, Airbnb, and Amazon demonstrated the value of thoughtful design.
📈 Practical Statistics for Experimentation
Statistical concepts were made accessible and directly applicable. Learners gained hands-on experience with p-values, confidence intervals, and sample size estimation, equipping them to distinguish meaningful outcomes from noise with greater confidence.
⚠️ Navigating Common Pitfalls
We addressed recurring challenges in experimentation practice, including premature analysis, novelty effects, and the risks of misaligned success metrics. Frameworks were shared for identifying and mitigating these risks proactively.
🛠 Applied Case Studies
To consolidate learning, participants worked through practical scenarios such as optimizing signup funnels and testing pricing strategies. These exercises provided transferable frameworks that can be applied immediately to ongoing projects.
Mastering the experimentation lifecycle is fundamental to being an effective product data scientist. It moves you beyond simply reporting numbers to influencing product strategy with rigor and confidence.
You become a strategic partner, not just a data analyst. Instead of just answering what happened, you can help teams figure out why it happened and, most importantly, what to do next. You learn to design tests that directly answer product questions and drive business value.
You build credibility. By understanding statistical rigor, you can confidently explain the validity of your findings and defend against flawed interpretations. This allows you to distinguish between meaningful results and statistical noise, helping teams avoid making poor decisions based on unreliable data.
You proactively identify and solve problems. Knowing the common pitfalls of experimentation allows you to anticipate potential issues before they derail a project. You can design more robust tests from the start, saving time and resources.
You turn data into a powerful tool for innovation. By applying these skills, you enable a culture of evidence-based decision-making. You help your organization learn, iterate, and innovate faster, ensuring that product changes have a measurable and positive impact on user experience and key business metrics.
✨ As Manisha Arora summed it up beautifully:
“AB testing is the gold standard for decision-making in product and data. It’s not just about running tests—it’s about building a culture of experimentation.”
🌟 Learner Spotlight
Our cohort included professionals from diverse backgrounds — data science, product management, and marketing. Learners like Kshitija Gupte, Data Scientist, shared live examples from ongoing experiments. Mitali Deshpande, Manager - Product Analytics, explored how her team could integrate testing into their roadmap.
Their stories reminded us that experimentation isn’t limited to one function — it’s a universal skill for better decision-making.
Here’s what Kshitija had to say about the workshop:
“Even when you think you know A/B testing, it surprises you. This workshop reminded me of something important: there’s no finish line when it comes to learning experimentation. Every domain, every product, every audience brings a different story, and with it, new insights. That’s what makes this space so exciting.”
🚀 What’s Next: Product Data Science Course
We don’t want the learning to stop here. The Product Data Science course is an 8-week live coaching program designed to build on these foundations and take your data science skills to the next level. Here’s a snapshot of what it offers:
Live, instructor-led sessions covering Product Sense, Experimentation, Machine Learning, Statistics & Probability
Regular assignments and interview-style questions with feedback to sharpen your thinking and presentation skills.
Resume reviews, compensation negotiation guidance, and career planning support so you’re ready not just technically, but also professionally.
Flexible format with evening/weekend sessions, suited for working professionals.
🎁 Alumni Benefit
As part of our alumni community from the A/B Testing Workshop, you’ll receive an exclusive $400 discount for the upcoming cohort of the Product Data Science Course starting on Sept 27. It’s our way of saying thanks — and helping you keep the momentum going.
Together, let’s continue building a culture of experimentation.
-Team PrepVector