«Moving to opportunity voluMe 14, nuMber 2 • 2012 U.S. Department of Housing and Urban Development | Office of Policy Development and Research ...»
Abstract The Institute for Social Research (ISR) at the University of Michigan successfully led an intensive, long-term, in-person survey for the Moving to Opportunity (MTO) for Fair Housing demonstration final impacts evaluation (Sanbonmatsu et al., 2011), achieving final effective response rates (ERRs) of 89.6 percent among MTO adults and 88.7 percent among youth, well above what response rates of surveys with comparable low-income populations have accomplished. A variety of survey field strategies ISR employed— careful staff selection, strategic use of financial incentives, and close collaboration between ISR and the National Bureau of Economic Research—all contributed to these high ERRs.
The high costs associated with achieving high ERRs for in-person surveys like that employed in MTO raises questions about added value. Costs per survey interview nearly quadrupled during the last 4 fielding months. This extra investment increased the MTO adult survey ERR by only about 3.2 percentage points. A reanalysis of intention-to-treat estimates on selected outcomes suggests the merits of such an investment. If survey fielding had stopped at an 81-percent ERR for adults, we would have falsely concluded that MTO had no effect on two of four key health outcomes, that MTO had no effect on female youth mental health, and that MTO increased female youth idleness.
Introduction As early as the 17th century, scientists observed that individuals who live in economically disadvantaged neighborhoods fare worse on a range of outcomes—from physical and mental health to employment and earnings, schooling, crime, and consumer bankruptcy filings—than individuals who live in economically well-off neighborhoods (Macintyre and Ellaway, 2003). Untangling whether neighborhoods per se, or the variety of characteristics of the individuals residing in particular neighborhoods, drive this observed association has been difficult. The question of whether neighborhoods matter is further complicated by the fact that we cannot always observe or measure the reasons why individuals decide to live in particular types of neighborhoods, and these very same reasons might be highly related to their outcomes. The Moving to Opportunity (MTO) for Fair Housing demonstration is a study uniquely positioned to contribute to our understanding of whether neighborhoods have causal effects on individuals’ well-being. The MTO experiment produced changes in housing mobility and subsequent experiences in low-poverty neighborhoods that can help isolate the effects of neighborhood circumstances on outcomes from a host of other individual, household, or local community characteristics.
To maximize MTO’s contribution to science and policy, the long-term survey for the final impacts evaluation1 (Sanbonmatsu et al., 2011) had an ambitious data collection strategy that included a broad set of outcomes measured from administrative records sources and an intensive in-person survey, occurring up to 15 years after study entry, led by the Institute for Social Research (ISR) at the University of Michigan. The National Bureau of Economic Research (NBER) research team set very high response rate goals to ensure that survey data collection adequately represented the eligible MTO sample and captured a breadth of outcomes across the domains of housing, neighborhood safety, physical and mental health, employment, education, financial security, and youth risky behavior. ISR successfully reached a final effective response rate (ERR)2 of 89.6 percent among MTO adults and 88.7 percent among MTO youth (ages 10 to 20 as of December 2007) in the long-term survey for the final impacts evaluation. These ERRs are much greater than what some studies of low-income populations have accomplished (Weiss and Bailar, 2002) and on par with several longestablished and well-resourced survey initiatives such as the Panel Study of Income Dynamics3 (American Association of Public Opinion Research, 2012; Gouskova, 2008; Groves et al., 2004).
Reaching such high response rates not only required substantial time investment from the NBER research team (to fundraise and design the survey) and financial commitment from the U.S.
Department of Housing and Urban Development (HUD) and various other funders, but it also required creativity and flexibility among ISR staff to navigate and strategize in real time while Research on MTO originally launched separately for each site, with a series of academic research investigators leading each site. The followup survey for the MTO interim impacts evaluation (Orr et al., 2003) that Abt Associates Inc. conducted was the first effort to administer a comparable data collection for MTO families overall. HUD also funded Abt Associates to canvass MTO families through 2007 to maintain an updated contact list.
Ludwig (2012) describes the calculation of the effective response rate, which reflects the weighted proportion of interviews completed for the eligible adult and youth samples.
The Panel Study of Income Dynamics (PSID) obtains response rates of 93 to 94 percent, and recent studies of PSID youth have response rates of between 87 and 91 percent.
interviewers worked in the field. Following and finding thousands of economically disadvantaged families who lived or currently live near resource-poor, potentially unsafe neighborhoods is a complex task. This complexity was compounded by the amount of time that had passed since the last in-person or phone contact with MTO study members and changes in MTO households—many of the youngest cohort at MTO study entry have since split off to create their own households.
The previous in-person interview with an MTO study household member had been a minimum of 5 years before the start of the long-term survey data collection for the final impacts evaluation, and some of the sample (37 percent, or 3,830) had not been interviewed at the followup survey for the interim impacts evaluation (Orr et al., 2003), meaning the most recent contact may have occurred more than 10 years before the start of this data collection. In addition to facing the pure locational challenges of finding the eligible MTO survey sample in light of the high overall ERR aims, MTO researchers wanted to maintain balance in the temporal flow of completed interviews by site and by treatment status. Maintaining sample balance in this way required particular monitoring, nimbleness, and flexibility among ISR’s data collection staff to target eligible survey sample members strategically, on a week-by-week basis, for extra attention from interviewers.
Some challenges that the ISR data collection staff faced for the MTO long-term survey effort are common to survey data collection efforts in general, but many were relatively unique or new to the experiences of ISR. The first sections of this article describe the various data collection design strategies ISR employed to maximize the probability of achieving the high ERR and strategies ISR implemented to address unanticipated challenges that protected, as much as possible, the quality of data and research design after survey data collection was in the field. These sections address factors that contributed to (and worked against) achieving a high response rate for both adults and youth.
Overall, the MTO in-person long-term survey for the final impacts evaluation was a reliable, efficient, and essential resource for capturing multiple aspects of life circumstances and individual outcomes that otherwise would have been difficult to capture at scale compared with lower cost alternatives. As one very poignant example, researchers would not have discovered MTO’s surprising effects on mental health outcomes if not for a survey instrument with diagnostic questionnaires used to measure mental health disorders. The intensive efforts required to achieve the very high MTO response rates do raise questions, however, about the relative worth of extra resources necessary to complete interviews among those last, difficult-to-find respondents. If NBER researchers and the MTO study funders had spent fewer resources and stopped data collection at a lower response rate, what would the estimated effects on survey-based outcomes have looked like? In an attempt to evaluate the ex post scientific value of expending additional resources on increasing the survey response rate in a study such as MTO, the last section of this article describes the cost of MTO long-term survey data collection and a few back-of-the-envelope calculations of MTO’s effects under varying response rate assumptions.
Background The MTO demonstration began in the mid-1990s at five sites (Baltimore, Boston, Chicago, Los Angeles, and New York City). Low-income families with children living in public housing in highly disadvantaged areas who volunteered for the MTO program were randomly assigned to one of
Cityscape 59Gebler, Gennetian, Hudson, Ward, and Sciandra
three groups: an experimental group that was offered housing vouchers that had to be used in a low-poverty area along with mobility counseling from nonprofit agencies, a Section 8 group that was offered a traditional housing voucher with no locational restrictions, and a control group that was not offered a housing voucher but remained eligible for any public assistance to which they were otherwise entitled.
MTO long-term survey fielding launched in June 2008 and continued through April 2010, with a carefully staged release of sample by each of the initial five MTO sites across three waves,4 with second-stage subsampling of the hardest-to-locate cases triggered at a predetermined initial response rate threshold of 75 percent. The ISR data collection staff conducted interviews in person, using a laptop computer and averaging 108.3 minutes for adult interviews and 116.7 minutes for youth interviews. The staff interviewed one adult and up to three youth ages 10 to 20 in each MTO family. In families with more than three eligible youth, ISR randomly selected three for inclusion in the sample. In addition to employing computer-assisted interviews, the data collection protocol included taking physical measurements, collecting dried blood spot samples from adults, and facilitating achievement assessments for youth.
Conducting an extensive and complex data collection effort with a highly disadvantaged and mobile population posed many challenges. First locating the family; then convincing respondents to participate; and finally completing a survey, physical measurements, and achievement assessments on multiple individuals in a single household combined to make this data collection operation extremely difficult. Some MTO families included foster children or youth who left home several years before the interview and had not kept in contact with other family members. Address information was often outdated or incorrect, and many families were living under the radar, without credit cards, mortgages, driver’s licenses, or other identification that could help locate respondents. As a result, interviewers often conducted tracking with a labor-intensive, door-to-door search, checking address information and asking neighbors if they had information about where the respondent or family may have moved.
In addition to tracking challenges, the ISR data collection staff faced considerable challenges working in and around economically disadvantaged neighborhoods. The interviewing protocol required interviewers to carry a laptop computer and a large bag of supplies and equipment.5 Many areas had no public parking, requiring interviewers to carry the equipment long distances in inclement weather and up multiple flights of stairs in highrise buildings. Interviewers often had to work in unsafe neighborhoods and conducted many interviews in suboptimal locations, including small and crowded living rooms, sometimes with no heat or electricity. Family members and friends coming and going, loud televisions and radios, and other distractions often made it difficult to maintain respondents’ focus and confidentiality. Finally, ISR experienced staffing shortages in some areas because of the inability to recruit and retain qualified interviewers willing and able to The first release of sample was in June 2008, the second in September 2008, and the third in February 2009, upon securing enough funding to survey a random two-thirds of Section 8 group adults.
The combined weight of the laptop computer, equipment, and supplies needed for completing an interview was approximately 30 pounds.
work successfully under the challenging conditions MTO required. As a result, production progress across the five MTO cities was at times uneven and required adjustments in field protocols to take into account differences in completion rates by site.
MTO Survey Data Collection: Strategies for Optimizing Response Rates ISR used a multifaceted set of approaches to address the previously listed challenges. This section discusses the ISR field structure and tracking efforts and the tools and strategies it used to maximize respondent participation.
Field Team Structure In any study, building an effective field team is an essential element of successful data collection.
ISR developed its field staffing model for MTO to take advantage of the clustered sample and to address the challenges of working with a sample that was highly mobile and lived in disadvantaged areas. The ISR data collection staff was made up of seven teams: a team of approximately 8 to 20 field interviewers in each of the five MTO cities, a team of Internet trackers supporting the field interviewers, and a travel team of field interviewers. The tracking team was composed of individuals who had Internet access to public records and were skilled at conducting Internet searches, networking, and piecing together information from many different sources. The travel team comprised experienced field interviewers with a demonstrated ability to work effectively and efficiently in the field and who lived in other parts of the country and were available to travel to interview respondents who had moved away from the main MTO cities. The travel team also supplemented data collection in cities that were understaffed.