Experimental evidence on a wide range of interventions in developing countries is accumulating rapidly. Setting aside questions of internal validity, we ask whether it is possible to reach externally valid conclusions from experimental evidence for other contexts of interests, and which factors (geography, context variables, individual variables) determine the accuracy of such an extrapolation. We use the Angrist and Evans (1998) sex composition natural experiment and data from International IPUMS on 166 country-year censuses to develop and implement a framework for considering the external validity of an experimental evidence base. We find significant variation in the effect of sex composition on incremental fertility. We explore the external validity function from external comparisons, and find that unconditionally differences in geography, education, time, and labor force participation between the target and experimental countries are all important predictors of external validity in the expected direction (closer matches lead to lower prediction error). We show that as experimental evidence accumulates external validity improves in three ways. First, predictive accuracy of our prediction error model improves. Second, out-of-sample predictive accuracy systematically approaches zero prediction error (although absolute error remains meaningful relative to the magnitude of the treatment effect). And third, with sufficient data, even simple rules of thumb for picking which experimental context to use in predicting the treatment effect in a context of interest perform as well as a model-based approach. |
