«Moving to opportunity voluMe 14, nuMber 2 • 2012 U.S. Department of Housing and Urban Development | Office of Policy Development and Research ...»
In practice, however, the outcomes in this article’s exhibits generally are limited to smaller samples, because they include age-based subsets of all youth (for example, we submitted only youth ages 15 to 20 for the postsecondary schooling data match). We also have proxy reports on 3,217 grown children (from 3,273 adult survey interviews), and we submitted all 4,643 grown children from the 4,604 families to administrative data agencies.
144 Moving to Opportunity The Long-Term Effects of Moving to Opportunity on Youth Outcomes where Yi is some outcome for MTO program participant i; Expi and S8i are binary indicator variables equal to 1 if participant i was in the experimental or Section 8 group (and the control group is the omitted reference group); and Xi represents a series of individual- and family-level baseline covariates that Sanbonmatsu et al. (2011) described and similar to the covariates Orr et al. (2003) described.
The coefficients on Expi and S8i capture the ITT estimates for the experimental and Section 8 groups, respectively. The ITT effect represents the estimated effect of MTO on the assigned group as a whole, including both families who leased up and families who never used an MTO voucher. The ITT estimate eliminates the problem of self-selection bias, because it compares the average outcomes of the entire treatment group (regardless of whether the family moved through MTO) with the average outcomes of the control group. Because of random assignment, the treatment and control groups should, on average, be identical regarding their baseline characteristics, so we can confidently attribute any subsequent differences in outcomes to the fact that the treatment groups were offered the opportunity to relocate through the MTO demonstration.
The TOT estimate represents the effect of MTO on the program movers; that is, the experimental and Section 8 group members who actually moved with the program vouchers. Under certain assumptions (for example, that the program did not affect families who did not use their MTO voucher), we can estimate TOT by dividing the ITT effect by the share of the experimental or Section 8 group that relocated with an MTO voucher (Angrist, Imbens, and Rubin, 1996; Bloom, 1984). The TOT estimate does not remove the self-selection bias, because it compares the members of the treatment group who leased up, a self-selected group, with would-be movers in the control group.4 Because 47 percent of the experimental group and 62 percent of the Section 8 group relocated with an MTO voucher (Ludwig, 2012), TOT estimates are substantially larger than ITT estimates. For example, if the ITT for an outcome was 8 percentage points for the experimental group, the TOT estimate would be [.08/.47] =.17, or 17 percentage points.
Measures The MTO long-term survey for the final impacts evaluation included an innovative combination of survey and administrative data collection. Within the survey interview setting, we administered math and reading achievement assessments; measured height and weight; constructed a full history of schools attended over the followup period; and used audio-enhanced, computer-assisted selfinterviewing (audio-CASI) to ask about sensitive items related to mental health and risky behavior.
We also collected a variety of administrative data, including postsecondary schooling data, criminal justice records, UI data, and government assistance data (food stamps and Temporary Assistance for Needy Families records).
The TOT approach assumes that those who did not use an MTO voucher experienced no average effect of being offered a voucher, which we believe is reasonable. Although the TOT estimates do not remove self-selection bias, the estimates are policy relevant because they focus on the effects that a new neighborhood environment would have on the individuals who would be most likely to participate in a housing voucher program.
Cityscape 145 Gennetian, Sciandra, Sanbonmatsu, Ludwig, Katz, Duncan, Kling, and Kessler School Characteristics We used two types of information to describe school characteristics: (1) a variety of socioeconomic and demographic characteristics available from three national databases, and (2) students’ self-reports of school climate. We constructed a full history of schools attended for each youth by combining parent reports on the youth’s schooling through the time of the followup survey for the interim impacts evaluation (or kindergarten for youth who were not of school age when the family volunteered for the MTO program) with youth self-reports through the time of the long-term survey for the final impacts evaluation (or the highest grade attended for youth who were no longer in a primary or secondary school). We then matched the school histories to school characteristics from two National Center for Education Statistics databases (the Common Core of Data for public schools and the Private School Universe Survey for private schools) and a school-level test score database.
We also constructed a school climate index based on whether youth strongly agreed, agreed, disagreed, or strongly disagreed with five statements about their most recent school’s climate. We asked youth if teachers were interested in students, if they felt “put down” by their teachers, if discipline was fair, if students who studied hard were teased, and if they felt safe in school. We constructed the index as the fraction of positive responses on the five items; that is, strongly agree or agree responses on teacher interest in students, fair discipline, and feeling safe, and disagree or strongly disagree responses on feeling put down by teachers and teasing of students who study hard.
Math and Reading Achievement At the end of the survey interview, we administered a 45-minute achievement assessment, an adapted version of the assessment used for the ED’s Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K). Youth ages 13 to 20 as of December 2007 took a slightly modified version of the eighth grade ECLS-K assessment, administered in two stages: a first-stage routing test that the survey interviewers scored in real time, the score of which then determined which form of the second-stage test to administer.5 We contracted with the Educational Testing Service (ETS) to score the assessments via estimates (known in the testing literature as theta scores) of each youth’s underlying academic ability from a statistical model based on item response theory (IRT). IRT scoring allows for the reliable prediction of a student’s ability on a full set of testing items based on only a subset of those items, which was important for the MTO study, given the limited time available in the survey interview setting.6 We converted the ETS achievement theta scores into z-scores by subtracting the control group’s average test score from each youth’s individual test score, then dividing by the standard deviation of the control group’s test score distribution. By construction, the control group’s average test score in this z-score metric will equal 0.
We selected the ECLS-K assessments for several reasons. They are designed to measure what children learn in school (as opposed to measuring aptitude only) and are sensitive to capturing whether MTO moved children into improved schooling and learning environments. They also Youth ages 10 to 12 also took an assessment, based on the ECLS-K fifth grade test, but the results for that age group do not qualitatively differ from those for youth ages 13 to 20, and we focus here on the eighth grade test.
Further details are available upon request. Also, see Reardon (2008).
include appropriate coverage of material that is relevant for the wide dispersion of ages of youth in the long-term survey for the final impacts evaluation, have good discriminating power across a wide range of ability levels, and have been extensively pretested and piloted (for example, to ensure that the test items work equally well for racial and ethnic subgroups).
The possibility that older youth in our survey sample would find the items on the tests too easy (known in the testing literature as a ceiling effect) was one concern about the ECLS-K assessments.
To address this concern, we supplemented the ECLS-K eighth grade test with a small set of math and reading items from ED’s National Educational Longitudinal Survey-1988 (NELS) assessment for high school students. Only about 8 percent of MTO youth ages 13 to 20 performed well enough to take these additional NELS items in math or reading. The possibility of a floor effect, in which the assessment is too difficult for some children and so loses its ability to distinguish the achievement of students at the bottom of the distribution, was another concern. About 14 percent of youth ages 13 to 20 performed at less than the level of chance on the reading test—that is, more poorly than we would have expected if they had simply guessed at every test question—and about 7 percent did so on the math test.
Educational Completion and Idleness During the survey interview, we asked older MTO youth (ages 15 to 20 as of December 2007) about their schooling, completed education, and participation in employment or training. From these measures, we constructed a measure of youth or young adult idleness defined as “not currently employed” and “not currently in school” at the time of the survey interview. We also obtained UI records and postsecondary enrollment data from the National Student Clearinghouse (NSC).7 Physical Health Self-reported physical health measures included overall health, asthma, and accidents and injuries.
To measure obesity, interviewers measured youth height and weight using the same protocols they used for adults then converted the results to the standard Body Mass Index (BMI) formula of weight in kilograms divided by height in meters squared. Because BMI tends to increase naturally through adolescence, instead of using the standard definition of obesity used for adults (BMI ≥ 30), we defined obesity using criteria developed by the International Obesity Task Force (Cole et al., 2000). Those criteria use growth curves based on age and gender that align with the standard adult BMI standards. The criteria further break down curves by gender because, whereas BMI tends to follow a linear trend for males, it tends to follow a more concave trend line for females, and because puberty generally affects female bodies at different ages than it does male bodies.
Mental Health We administered two short questionnaires to measure psychological distress and behavioral and emotional problems. The first was the Kessler 6 (K6), a six-item questionnaire used to determine NSC data were available back through 2001, but it took until the end of 2006 for NSC to be near complete (96 percent of schools had joined NSC by then), so we have limited our analysis to the 3-year period from January 2007 to January 2010.
general psychological distress. Youth reported how often in the past 30 days they felt so sad that nothing could cheer them up, nervous, restless or fidgety, hopeless, that everything is an effort, and worthless. The K6 raw score can range from 0 (no distress) to 24 (highest distress), and our K6 measure is a z-score based on the control group mean and standard deviation, with standardization separated by gender and flipped such that a lower score indicates less psychological distress. The second questionnaire was a brief version of the Strengths and Difficulties Questionnaire (SDQ), which is used to identify behavioral and emotional problems. Interviewers read five statements to youth, who reported how true (very, somewhat, or not) each statement was about their general behavior. The five statements concerned general obedience, worry and anxiety, feeling unhappy or depressed, getting along better with adults than with peers, and task completion and attention span. Raw SDQ scores can range from 0 (no behavioral or emotional problems) to 10 (severe behavioral or emotional problems). A score of 6 or higher is a commonly used indicator of serious behavioral or emotional problems.
Risky and Criminal Behavior Similar to the approach used in the followup survey for the interim impacts evaluation, our approach measured risky and criminal behavior through both youth reports and data matches with criminal justice records. To reduce the likelihood of youth underreporting sensitive or illegal behaviors, we administered many of the sensitive items in the survey about risky behaviors via
audio-CASI. We constructed three indices: risky, problem, and delinquent behavior. The risky behavior index is the fraction of 4 risky behaviors in which the youth reported ever having engaged:
smoking, alcohol use, marijuana use, and sex. The delinquency index is, similarly, the fraction of 8 delinquent behaviors: drug selling, gang involvement, gun possession, attack on another person, property destruction, theft of an item worth less than $50, theft of an item worth more than $50, and other property crime. Finally, the behavior problems index is the fraction of 11 behaviors that the youth reported were true or sometimes true (as opposed to not true) of their behavior in the 6 months before the survey interview: trouble paying attention, lying or cheating, teasing others, disobeying parents, trouble sitting still, hot temper, would rather be alone, hanging out with kids who get in trouble, disobeying at school, not getting along with other kids, and trouble getting along with teachers.
Results MTO had few detectable long-term effects on achievement and educational outcomes, physical health, and several aspects of risky behavior. Children assigned to the experimental and Section 8 groups had similar scores on reading and math achievement tests compared with those in the control group. This finding held true for children who had not yet enrolled in school at baseline and who would have experienced particularly large MTO-induced changes in neighborhood environments very early in their development of cognitive, social-emotional, and behavioral skills. A pattern of generally beneficial effects on female youth and some detrimental effects on male youth echoes, but is more muted than, the pattern the followup survey for the interim impacts evaluation found (Kling, Liebman, and Katz, 2007; Kling, Ludwig, and Katz, 2005). Male youth who moved through MTO engaged in relatively more of some risky behaviors (smoking) than male youth in