Adjusting Treatment Effect Estimates by Post-Stratification in Randomized Experiments
Experimenters often use post-stratification to adjust estimates. Post-stratification is akin to blocking, except that the number of treated units in each strata is a random variable be- cause stratification occurs after treatment assignment. We analyze both post-stratification and blocking under the Neyman model and compare the efficiency of these designs. We derive the variances for a post-stratified estimator and a simple difference-in-means estimator under different randomization schemes. Post-stratification is nearly as efficient as blocking: the difference in their variances is on the order of 1/n2, provided treatment proportion is not too close to 0 or 1. Post-stratification is therefore a reasonable alternative to blocking when the latter is not feasible. However, in finite samples, post-stratification can increase variance if the number of strata is large and the strata are poorly chosen. To examine why the estimators variances are different, we extend our results by conditioning on the observed number of treated units in each strata. Conditioning also provides more accurate variance estimates because it takes into account how close (or far) a realized random sample is from a comparable blocked experiment. We then show that the practical substance of our results remain under an infinite population sampling model. Finally, we provide an analysis of an actual experiment to illustrate our analytical results.