«Moving to opportunity voluMe 14, nuMber 2 • 2012 U.S. Department of Housing and Urban Development | Office of Policy Development and Research ...»
A different way to think about how MTO changes people’s neighborhood “quality” within the larger neighborhood-quality distribution is to measure MTO’s impacts in standard deviation (sd) units. This metric essentially compares the size of the MTO impacts on census tract poverty rates with the amount of “spread” in the larger census tract poverty rate distribution. Exhibit 2 shows that, 1 year after random assignment, the experimental group ITT effect is about -1.4 sd within the national tract distribution as measured in the 2000 decennial census data, whereas the TOT effect is equal to fully -2.8 sd.8 The experimental group effects on duration-weighted average tract Exhibit 2 also shows results that standardize MTO’s impacts on tract poverty rates using the standard deviation of the control group’s tract poverty distribution, rather than the national tract poverty distribution found in the 2000 census.
poverty rates averaged over the entire study period equal about -0.7 sd (ITT) and -1.5 sd (TOT).
It is difficult to think of many social experiments that generate such large changes in important aspects of the living conditions of poor families.
MTO can also look like a weak treatment if analysts divide neighborhoods up into a small number of discrete and essentially arbitrary “types,” which has the effect of throwing away information and making it harder to see how neighborhood conditions differ across randomly assigned groups. For example, Clampet-Lundquist and Massey (2008) create four neighborhood categories by dividing them on two separate dimensions: “poor” versus “nonpoor” (whether the tract’s poverty rate is above or below 20 percent); and “segregated” versus “integrated” (whether the tract’s minority share is above or below 30 percent). Similarly, Turner et al. (2011) use threshold values of tract characteristics to define various categories of “high-opportunity” neighborhoods, such as those with “high work and income” (tract poverty rates below 15 percent and employment rates above 60 percent) or “high education” (20 percent or more of adults have a college degree). They conclude: “Although MTO enabled families to escape from the most severely distressed neighborhoods, very few actually gained and sustained access to high-opportunity neighborhoods” (Turner et al., 2011: 7).
Defining “low-poverty” or “high-opportunity” neighborhoods on the basis of whether tract characteristics are above some threshold value makes sense if and only if we believe that neighborhoods only influence behavior once they reach some “quality” threshold. Put differently, dividing neighborhoods into a small number of categories is sensible only if neighborhood effects on outcomes are nonlinear, so that (say) moving from a tract that has a 50-percent poverty rate to one with a 16-percent poverty rate has no effect on people’s outcomes (both of those neighborhood types would be “poor” under the Turner et al. definition), but moving from a neighborhood with a 16-percent poverty rate to a 15-percent poverty rate would have important impacts on outcomes (this would be a move from a “poor” to “nonpoor” area in a Turner et al.-type definition).
The evidence presented in Kling, Liebman, and Katz (2007), however, seems to suggest that a 1-percentage-point change in tract poverty rates has the same effect on people’s life outcomes regardless of whether we are going from 16 to 15 percent poor, or 26 to 25 percent, or 36 to 35 percent, and so on. If neighborhood effects on people’s outcomes are linear, as the findings by Kling, Liebman, and Katz seem to suggest, then dividing up neighborhoods into a small number of categories winds up masking some of MTO’s impacts on the neighborhood conditions of participating families, by treating all neighborhoods with poverty rates above some threshold value as indistinguishable members of the same type of place (in Turner et al.’s typology, going from 50 percent to 16 percent poor leaves one within the same neighborhood “type”). If neighborhood effects on outcomes are linear, then the most appropriate way to measure MTO impacts on neighborhoods is by reporting the impact on percentage point changes in the tract characteristics themselves—that is, looking at continuous measures.
So is MTO too much of a “weak treatment” to be useful for social science purposes? Is there enough difference in average neighborhood conditions between the two treatment groups and the control group to let us learn something about neighborhood effects? One benchmark we might use is to compare the amount of variation we see in neighborhood conditions in the MTO data with that captured by what is arguably the most important observational (nonexperimental) study of neighborhood effects ever carried out, the PHDCN. Sampson, Sharkey, and Raudenbush (2008) 16 Moving to Opportunity
Guest Editor’s Introductionused the PHDCN to examine effects on verbal ability of African-American children from living in a census tract in the bottom quartile of Chicago’s distribution with respect to concentrated tract disadvantage (defined previously), or the “treatment group” in their study, compared with all other African-Americans in their study, the “controls.” The treatment group in their study lived in tracts that were 38 percent poor compared with control tracts that were 20 percent poor on average, for a difference of 18 percentage points—almost identical to what we see in MTO.9 What Do the MTO Results Mean for Social Science?
Twenty-five years ago, Wilson (1987) argued that a key reason why people living in high-poverty central-city neighborhoods tended to drop out of school or be out of the labor market was because of the harmful effects of the neighborhood environments in which they were living. The MTO data do not seem to support that hypothesis, at least for the sort of low-income, disadvantaged family that signed up for MTO.
This raises the question of whether families as disadvantaged as those enrolling in MTO could have been expected to experience improved schooling and labor market outcomes from moving to less distressed areas. Presumably, the U.S. Congress and HUD thought so, because schooling and earnings were key outcomes mentioned as a focus of the demonstration. Previous observational studies like PHDCN have reported finding neighborhood effects on schooling outcomes for people about as disadvantaged as those in MTO.10 And the sorts of very disadvantaged families who live in our nation’s most distressed public housing projects have, for understandable reasons, commanded a disproportionate share of the media and policy attention. Although the MTO results might not generalize to families with higher levels of socioeconomic status, knowing whether neighborhoods exert causal effects on key outcomes like schooling and work for very disadvantaged families is important in its own right for social science and public policy.
Some people have concluded that MTO could have had bigger impacts on schooling outcomes if only the experimental group moves generated larger changes in the characteristics of the schools that children attended (see also the articles in this symposium by Turner and Oreopoulos). Maybe.
Previous studies suggest that attending a higher quality urban school (public or charter) tends to If we look instead at Sampson, Sharkey, and Raudenbush’s (2008) concentrated disadvantage index, defined without share African American included in the index, the treatment group in their study has an average value of 1.71 and controls have a value of 1.04, for a difference of 0.67. As shown in MTO, the control mean is 1.39 and the average value for those who move with an experimental group voucher is 0.90, for a difference of 0.49.
For example Sampson, Sharkey, and Raudenbush (2008) report statistically significant neighborhood effects on verbal test scores among African-American children in Chicago who were in the PHDCN study. As reported in the previous footnote, the average value of the concentrated disadvantage index for their high-poverty (“treatment”) group was 1.71 compared with an average value for the MTO high-poverty group (which we happen to call our “control group,” instead) was 1.31— or, put differently, their study sample is living in neighborhoods that are, on average, even more distressed than those of the average MTO family. Supplemental Table 6 for their paper reports on the mean values of their baseline covariates among all African Americans in their study sample. Their study children are living in overwhelmingly (92 percent) female-headed households, just as in MTO. A lower share of their PHDCN study sample is receiving welfare at baseline than in MTO (49 versus about 75 percent), but it is important to note that the baseline covariates they present are averaged across the entire set of African-American families in the PHDCN. If they reported baseline covariate values just for the families living in highly distressed neighborhoods, their baseline covariates would surely be even closer to what we see in MTO (http://www.
have beneficial impacts on behavioral outcomes like schooling persistence or delinquency. This is not as consistently true with respect to achievement test scores, which have, for better or worse, been an outcome of particular interest in policy discussions and for which previous studies tend to find more mixed impacts (Abdulkadiroglu et al., 2011; Angrist et al., 2010; Angrist, Pathak, and Walters, 2011; Cullen, Jacob, and Levitt, 2006; Deming, forthcoming; Hastings, Kane, and Staiger, 2006). How do we make sense of the fact that gaining access to a better school does not always lead to higher achievement test scores for all students?
One candidate explanation is that not all children experience a given school environment the same way. As my University of Chicago colleague Stephen Raudenbush once said to me: “Dealing with heterogeneity across students in their academic needs is the challenge of education.” What a child gets out of attending a given school might all too often depend on where he or she falls within the school’s test-score distribution. Anyone who has ever taught will be familiar with the idea that teachers tend to target instruction towards the middle of a classroom’s achievement distribution. Some previous studies suggest teachers might even devote disproportionate attention to those students at the top of the distribution (B. Bloom, 1984). Children who are already far behind in school might not benefit much from attending a better school if that means that they experience a lot of instruction pitched above their heads. Common components to many successful educational interventions include frequent assessments to gauge what students are learning, targeted instruction through tutoring or small-group settings, and extra time for this sort of instruction—something that regrettably few disadvantaged children seem to receive regardless of where they live and go to school.11 Just as MTO lets us rule out the strong claim that neighborhoods always matter, I spend a lot of my time talking to economists who tell me that, to them, the lesson from MTO is that neighborhood environments are just not that important for poor families. The fact that MTO moves generated changes in some important outcome domains, particularly mental and physical health, means that we can reject that view as well.
What is particularly remarkable about the MTO health impacts is how massive they are. As Sanbonmatsu et al. note in their article, moving with an MTO experimental group voucher reduced rates of extreme obesity (Body Mass Index ≥ 40) and diabetes (HbA1c ≥ 6.5 percent) by around 40 percent expressed as a share of the control group’s prevalence rate. Although clinical trials in medicine rarely enroll study samples quite as economically disadvantaged as that in MTO, it is still quite striking that the MTO impact on diabetes is about as large as what we see from best-practice pharmaceutical treatment and public health lifestyle interventions. Similarly, Kling, Liebman, and Katz (2007) noted that MTO’s impacts on mental health outcomes in the interim (4- to 7-year) followup were about the same size as what we see from best-practice drug treatment.12 For example, Success for All, a comprehensive reading intervention that involves extra time for reading, ability grouping, frequent assessment, and remediation (including tutoring), has been found to improve reading scores for elementary school children and perhaps middle schoolers as well (Borman et al., 2007; Chamberlain et al., 2007). Angrist, Pathak, and Walters (2011) noted that the more effective urban charter schools they studied in the Boston area tended to be those adopting the “No Excuses” approach of the Knowledge is Power Program (KIPP) schools, which emphasize extra math and reading instruction time. Angrist et al. (2010) showed that those students who benefit the most from attending a KIPP school are those with low baseline test scores, with limited English proficiency, or in special education programs.
Note that, although we might have expected improved mental health among MTO adults to translate into improved children’s schooling and other outcomes, the size of the impact on children that we would expect from improved adult mental health would not be detectable in the MTO data.
A more difficult question to answer is why MTO had such pronounced impacts on health. Experiments in general tend not to be so well suited to answering why questions. In MTO, the problem is further compounded by the fact that the treatment (MTO moves) wound up changing a very large number of housing and neighborhood characteristics for participating families, as exhibit 2 makes clear, which complicates any attempt to figure out what is responsible for the observed differences in average health outcomes between the two treatment groups and the control group (or the lack of observed differences in other outcome domains). Therefore, trying to figure out why MTO affected health more than other outcomes will necessarily involve some speculation.