Evaluate how free shipping affects membership conversions using Propensity Score Matching — a step-by-step case study showing how causal inference can guide product decisions without A/B testing.
Thank you for sharing this knowledgable case study! One question I have here is as we deploy the feature to all clients using the free trial, so is the population should be all clients using the free trial, and the treatment is the clients who opted to use the free shipping?
Glad you found this useful, Mike! Yes, the entire target population consists of users on a free trial. The treatment group includes users who use free shipping, and instead of comparing them to the full baseline of users who don’t use free shipping, we select a matched control group of users - those who are similar but didn’t opt for free shipping.
I have couple of questions - You said that free shipping is rolled out to every user then why treatement is 1 and 0. Please clarify
During trail period if we are giving free shipping to trial users then how does it mitigate the issue with RCT. You have hinted towards ethical concerns. How do they vanish in this case.
Thanks for the write-up and sharing it across the board. I had a quick clarification around the treatment definition.
From what I understand, all users on the free trial were offered free shipping, but the analysis/comments defines treatment as actual usage of free shipping (vs. non-usage), and uses propensity score matching to construct a control group of non-users with similar observed characteristics.
Just wanted to confirm a couple of things:
1. Since everyone was exposed to the benefit, does defining treatment based on usage behavior introduce a risk of self-selection bias? (i.e., users who use free shipping may differ in unobserved ways from those who don’t).
2. It seems the estimate reflects the effect of using free shipping on conversion, not the effect of offering it. Is that the intended goal?
I am asking because the framing of "treatment" as usage feels closer to estimating a complier effect, and I want to make sure I am interpreting the setup correctly.
Thank you for sharing this knowledgable case study! One question I have here is as we deploy the feature to all clients using the free trial, so is the population should be all clients using the free trial, and the treatment is the clients who opted to use the free shipping?
Glad you found this useful, Mike! Yes, the entire target population consists of users on a free trial. The treatment group includes users who use free shipping, and instead of comparing them to the full baseline of users who don’t use free shipping, we select a matched control group of users - those who are similar but didn’t opt for free shipping.
I have couple of questions - You said that free shipping is rolled out to every user then why treatement is 1 and 0. Please clarify
During trail period if we are giving free shipping to trial users then how does it mitigate the issue with RCT. You have hinted towards ethical concerns. How do they vanish in this case.
Thanks for the write-up and sharing it across the board. I had a quick clarification around the treatment definition.
From what I understand, all users on the free trial were offered free shipping, but the analysis/comments defines treatment as actual usage of free shipping (vs. non-usage), and uses propensity score matching to construct a control group of non-users with similar observed characteristics.
Just wanted to confirm a couple of things:
1. Since everyone was exposed to the benefit, does defining treatment based on usage behavior introduce a risk of self-selection bias? (i.e., users who use free shipping may differ in unobserved ways from those who don’t).
2. It seems the estimate reflects the effect of using free shipping on conversion, not the effect of offering it. Is that the intended goal?
I am asking because the framing of "treatment" as usage feels closer to estimating a complier effect, and I want to make sure I am interpreting the setup correctly.