Randomized Experiments and Observational Comparisons

Randomized controlled trials (RCTs) have a rich history, dating back to the 1920s when researchers used them to compare crop yields under different conditions. In medicine, RCTs gained prominence in the mid-20th century, thanks to pioneers like Austin Bradford Hill, who demonstrated their power in evaluating the effectiveness of treatments like streptomycin for tuberculosis. Today, RCTs are considered the gold standard for establishing causality in many fields, from healthcare to social policy.

In the tech sector, a simplified version of the RCT – the A/B test – has become ubiquitous. Companies routinely run A/B tests to evaluate new features, website designs, and marketing campaigns. The allure of A/B testing lies in its simplicity: randomly assign users to different groups, expose them to different versions of a product or experience, and measure the outcomes. This allows for a clean comparison, isolating the effect of the change from other factors that might influence user behavior.

Yet in business, pure randomization is not always possible or sufficient. Budget constraints, customer experience considerations, and operational realities often force us to evaluate interventions using observational data. This part concludes with matching, which serves as a conceptual and methodological bridge from randomized experiments to observational identification. When treatment is not randomized, comparing treated and untreated units requires strong, explicit identifying assumptions: that all confounders influencing both assignment and potential outcomes are measured (unconfoundedness or conditional exchangeability), that every unit has a non-zero probability of receiving either treatment level (positivity and overlap), and that there is no interference or hidden treatment variation (SUTVA). Matching attempts to recreate balance on observed covariates, highlighting what observational designs can adjust for and what they must assume.