Angus Deaton on the “project evaluation” craze:
Randomized controlled trials cannot automatically trump other evidence, they do not occupy any special place in some hierarchy of evidence, nor does it make sense to refer to them as “hard” while other methods are “soft”. These rhetorical devices are just that; a metaphor is not an argument… thirty years of project evaluation in sociology, education and criminology was largely unsuccessful because it focused on whether projects work instead of on why they work.
The wholesale abandonment in American graduate schools of price theory in favor of infinite horizon intertemporal optimization and game theory has not been a favorable development for young empiricists. Empiricists and theorists seem further apart now than at any period in the last quarter century. Yet reintegration is hardly an option because without it there is no chance of long term scientific progress.
and after listing a number of papers that he thinks have a good mix of theory and data, he says:
In all of this work, the project, when it exists at all, is the embodiment of the theory that is being tested and refined, not the object of evaluation in its own right, and the field experiments are a bridge between the laboratory and the analysis of “natural” data.
Science is about finding underlying mechanisms. Its not about testing hypotheses and
[H]eterogeneity is not a technical problem, but a symptom of something deeper, which is the failure to specify causal models of the processes we are examining. This is the methodological message of this lecture, that technique is never a substitute for the business of doing economics.
I use the “project evaluation” and “experiment” rhetoric in one of my papers. I might have to rethink the organization of that paper…