Johannes Haushofer and Jeremy Shapiro recently released a paper (HS18) estimating the three-year impacts of individually randomized cash transfers on GiveDirectly recipients. We wrote a short note on these results in February; recently there has been active discussion among researchers (e.g., e.g.), and we’ve since followed up with many of them.
Our failure in original post
These conversations have demonstrated that we did a bad job in our initial note in describing the data and the range of interpretations one might take from them, including more negative interpretations. We take responsibility for that. The post was criticized for lacking “nuance and detail,” and we agree with that assessment.
Results w/ confidence: (i) increase in assets (ii) inability to draw strong conclusions on spillover
To be clear, we agree with the authors’ summary as laid out in the most recent working paper (Jan 2018). Their abstract concludes: “thus, cash transfers result in sustained increases in assets. Long-term impacts on other dimensions, and potential spillover effects, remain to be substantiated by future work.”
- The first point refers to the fact that treated households’ (non-land) assets were higher by 40% (or 60% of the mean transfer size) and that this conclusion is robust to both estimation methodologies: within-village and between-village.
- The second clause refers to the inability to draw strong conclusions about the long-term impact on other outcomes and on non-recipients.
We think this is a fair summary; in particular it makes clear the interpretive nature of conclusions about the impacts on outcomes other than assets, which we did not make explicit enough in our first post.
Interpretation of within-village estimates is less certain: two potential interpretations to debate
At the heart of the discussion is the interpretation of the following results:
These results allow us to confidently conclude that recipient households in treated villages are better off over the long-term than their eligible non-recipient neighbors (i.e., the households in treatment villages that lost the randomization lottery) across most indicators. But they do not on their own say whether this is because:
The study does not allow us to conclude confidently which of these is correct
To decide between these interpretations, one must look to the between-village or spillover estimates; unfortunately, such estimates are potentially biased in this paper because households in untreated villages (a) were sampled using a different process and (b) attritted differentially from the study (i.e., control villages were 6 percentage points more likely to not be found between endline 1 and 2 than either group in treatment villages).
The authors handle this in a responsible way by bounding the estimates, and conclude that:
This statistical insignificance and more broadly, the potential bias of the estimates, is what prevents us from forming confident conclusions about spillover (or between-village treatment) estimates.
These limitations are one reason (among several) why along with Johannes and other investigators we launched a larger study in 2014. This study (“GE”) is randomized at the village level and used a consistent process to sample both eligible and ineligible households in both treated and untreated villages.
The second, more negative interpretation would obviously be a source of concern and have important implications for the way we design and test programs. We had similar concerns two and half years ago that losing a within-village lottery could be a negative emotional experience for the untreated eligible households (see footnote 86 in GiveWell post). Based on that feedback we chose to:
- Stop allowing researchers to randomize treatment within communities in future studies except in exceptional cases, and
- Send transfers to the untreated, eligible households from those studies that did involve within-village randomization.2
In aggregate, approximately 5.4% of GiveDirectly recipients have been part of an individual-level randomization since September 2015, and in 79.5% (and hopefully 100%) of these cases, the initial lottery-losers will also receive a cash transfer. Recently we have gone further, shifting towards “saturating” treated villages by making essentially all households within them eligible so that there almost no non-recipients of any sort.
… although we don’t think the most negative interpretation (B) is the most plausible
While we can only confidently conclude that we can’t confidently conclude much on the spillovers, we’re skeptical of the negative interpretation.
First, the magnitude of the effect is puzzling. The results would imply that the transfers, which were a relatively small change to villages as a whole, representing at most 3.4% of annual expenditure (with 9% of those in treatment villages receiving a transfer) lowered expenditure of their non-recipient neighbors by 20% and their (non-land) assets by 14%. This is surprising on its own, but even more so since (i) the authors did not find negative spillovers at 9 months, implying that the negative spillovers only appear with time (ii) the most common mechanism for negative spillovers, an increase in prices, was not observed by JS after 9 months. (tables 149-157 of the online appendix).
This combination of factors is what leads us to think that the program more likely than not had positive results across the main indicators, as opposed to just non-land assets. We recognize, however, that this is not the only interpretation and we should have made that clearer. We look forward to future research that will help clarify which of the interpretations is accurate.
1 Exception is expenditure which is negative and significant across the various specifications in Table 8.
2 The untreated HS households received theirs in 2016, shortly after the 3-yr survey was conducted.