«Moving to opportunity voluMe 14, nuMber 2 • 2012 U.S. Department of Housing and Urban Development | Office of Policy Development and Research ...»
A team leader, who provided coaching and monitored quality and production and efficiency statistics, supervised each team. A production manager in the central office worked with a production coordinator in the field to supervise the team leaders and provide guidance and direction for the field activities, lead training activities, and act as the main conduit for the flow of information between the central office and the dispersed MTO field team. Exhibit 1 depicts the field staffing structure.
Interviewer Hiring and Training ISR recruited and hired all data collection staff—field interviewers, travelers, tracking experts, and supervisory staff. When recruiting staff for MTO, making sure the field staff members could work safely in low-income neighborhoods was important. The production manager and human resources specialist worked together with local interviewers already on staff at ISR to identify local advertising resources, such as neighborhood newspapers, community centers, libraries, and churches. The job posting identified some areas of the city in which MTO interviewers would work, clearly outlined the challenges interviewers would face working on this project, and emphasized the importance of the study. This emphasis on hiring local interviewers who were familiar with MTO neighborhoods and comfortable working in disadvantaged areas proved challenging (recruitment goals were not met in all areas), but it also yielded an exceptionally dedicated data
ISR = Institute for Social Research (University of Michigan). NBER = National Bureau of Economic Research.
collection staff who remained with MTO until the end of the project. Approximately 75 percent of the ISR interviewers were newly hired (some had previous social science interviewing experience with other organizations but were new to ISR). All of the field supervisory staff (team leaders, production coordinator, and production manager) were experienced staff members who had worked on a variety of other social science data collection studies at ISR.
ISR conducted two separate training sessions for field interviewers in June and September 2008.
Although having two training sessions was not the initial plan, it turned out to be beneficial in many ways. The smaller groups enabled team leaders to work more closely with each interviewer, and the slower start to production in June provided a good shakedown period, during which ISR identified and resolved issues and made adjustments ahead of the later training. Having two recruitment and training periods also enabled ISR to capitalize on the enthusiasm for the project of the first group of interviewers to help recruit additional field interviewers. Finally, the ISR data collection staff trained during the first session shared valuable tips and tricks with the newer team members, helping to mentor new interviewers and bring them up to speed more quickly after training.
Retention of the field staff was higher for MTO than for similar ISR studies. Of the 91 interviewers and trackers trained and certified to work on MTO, 85 percent were still on staff 6 months after the start of data collection, and 74 percent remained on staff through the first year of data collection. Starting in June 2009, ISR began intentionally consolidating the interview sample and releasing some interviewers to work on other projects as the second-stage subsampling began and the available number of respondents was reduced. Even with this reduction in the second half of 2009, 50 percent of the field staff stayed with MTO through the end of 2009.
62 Moving to Opportunity Achieving MTO’s High Effective Response Rates: Strategies and Tradeoffs Communication The central office project managers and production manager, in consultation with the NBER research team, set data collection goals and priorities for the data collection staff in the field. Each week, the ISR project managers met with the NBER research team to discuss progress, issues, requests from the field, strategies for improvement, and priorities. The priorities and action steps that emerged from this meeting provided direction for the following week’s field activities. The close collaboration among the ISR project managers, the NBER research team, and the field supervisors on the data collection staff resulted in a common set of goals and priorities, which were relayed to the field teams through standardized weekly meeting agendas and reinforced in a weekly newsletter, continuous training sessions, and coaching for groups and individuals.
In addition, field supervisors on the data collection staff met weekly with the ISR project managers to review progress, learn of project updates and priorities from the NBER research team, and together develop a set of standardized agenda items for the weekly team meetings. Each of the seven data collection teams met weekly and reviewed the project agenda items and discussed team-specific issues and items. The ISR project managers produced a weekly newsletter that reinforced the training and discussions in the weekly meetings. The newsletter started as a tool to help connect the ISR project managers to the field, focusing on project updates and reminders from the ISR central office. It was adapted over time to include notes of thanks and congratulations, progress updates, training reminders, tips for effective interviewing, and more. Communication flowed from the data collection staff in the field to the ISR central office and NBER through weekly written field reports and discussions in weekly management meetings. This communication structure often led to adjustments or changes in procedures and materials based on recommendations from team leaders and interviewers. Communication was also very important from a safety perspective. ISR provided interviewers and team leaders with cell phones and asked them to let their team leader know when they were interviewing alone in dangerous neighborhoods, in unfamiliar areas, or after dark.
Team Interviewing The use of team interviewing was another feature of the MTO data collection. Because the MTO protocol included interviewing multiple individuals in a family, ISR encouraged team interviewing, whereby two (or occasionally three) interviews were conducted in a household at the same time.
This procedure had several benefits: it reduced the time burden on families when multiple interviews were conducted; improved interviewer efficiency; and reduced safety concerns for interviewers traveling into dangerous areas, especially for evening interviews. It also helped ensure confidentiality by keeping the parent occupied with her6 own interview while the youth was being interviewed.
Finally, it enabled experienced interviewers to mentor others, helped create strong local teams that supported each other, and fostered the sharing of strategies for effective data collection. In addition to participating in team interviewing, teams often worked together on tracking blitzes, in which all members of a local team would focus on tracking for a weekend, working closely with the tracking team to try to locate as many individuals as possible and set up interview appointments for the following week. Tracking was a difficult and often frustrating part of the process, and having the Most adult respondents were female (although some male adults were in the sample).
entire team track respondents together and share success stories proved to be very motivating and an effective way to complete interviews. Likewise, interviewers who were effective at tracking could pair with those who were not as strong to help teach valuable skills and techniques.
Tracking and Locating Respondents ISR integrated tracking into its data collection operations from the start of the project and developed sample management systems to facilitate both the interviewing process and tracking lost respondents.
To help limit the number of contacts needed to locate each family member, take advantage of family connections for making multiple interview appointments, and locate respondents who had moved, ISR loaded all addresses (including those at study entry, those at the time of the interim MTO study, and updates obtained through the National Change of Address system and from the HUD database) into the sample management system for each family. ISR assigned all members of a family to one interviewer, making that single interviewer responsible for locating the family. The goals were to reduce burden on the family by using one point of contact and to gather information for multiple family members in a single contact. Using this procedure served to contain costs by working first with a family-locating instrument, which enabled the interviewer to contact any family member (or informant who knew where a family or respondent may be located) and update contact information for all selected members of the family. The family-locating instrument also gave the interviewer a view of the entire work scope for that family: the number of interviews to be completed at a single address and the ages of the respondents (which determined, for example, which youth achievements assessments would be administered). After the family instrument was complete, individual sample lines were released to the interviewer with contact information updated from the family instrument, enabling each individual survey respondent to be contacted, tracked if necessary, and interviewed. Often, interviewers were able to set up the appointments for interview sessions (and arrange for team interviewing) when working with the family instrument.
In addition to taking advantage of family connections to help locate respondents who had moved since the last interview, the data collection staff included a tracking team that was integrated into the five local interviewing teams (one at each MTO site) and the travel team. Although all trackers could and did work sample lines from any location, having one primary tracker working with each local interviewing team enabled the tracker to establish strong working relationships with the field interviewers and to become very familiar with the geographic region in which they were working.
The tracking team used a combination of pay-for-service Internet search engines that pulled from public records and free tools, such as reverse phone searches and white page telephone directories, coupled with extensive telephone networking with family members and contact people to generate leads for the field interviewing team to follow in person when searching for lost respondents. Although telephone and Internet tracking was effective for many cases, a substantial amount of inperson tracking was required to reach the response rate goals. Often, the tracking team would identify a potential address, and the local interviewer would visit the address to find that the person no longer lived there. The local interviewer would then check with neighbors to gather information about where the respondent had moved.
This process sometimes generated additional leads that were then passed back to the tracking team for further Internet searches and telephone work. For example, one fieldworker shared a
tracking story about working with an interviewer, “Tim,” who was traveling in a particular city trying to locate an address where the respondent reportedly owned a house. The first attempt to find the address was unsuccessful because GPS, maps, and the initial in-person visit revealed only the 300-to-500 block of the street name, and the respondent’s address was in the 1400 block. Tim e-mailed the tracker to report the finding, and the tracker double-checked the property records and determined that the respondent’s residence was in the 1400 block of the street in question.
The tracker suggested that Tim check with the post office or a municipal office to see if the road continued elsewhere. Another tracker suggested checking with the fire department, which led the interviewer to a new subdivision where he found the address and the respondent. Teamwork and persistence paid off, and the team completed the interview.
Some of the more difficult respondents to locate included youth who had run away or left home without keeping contact with family, respondents intentionally living off the grid, and institutionalized respondents (in nursing homes, detention facilities, and so on). Trust was an issue for some MTO respondents; interviewers required a great deal of persistence to convince some family members that the family was not going to be reported to the authorities. The data collection staff was very successful in getting family members or friends to ask the respondent to call the centralized ISR toll-free line or agree to meet in a public place (such as a library or fast food restaurant) to explain the study and set up an appointment for an interview. The toll-free line forwarded calls directly to the interviewer’s MTO cell phone, which was very helpful because the difficult-to-reach respondents often did not have a phone or had a very limited number of cell-phone minutes. If a call was missed, the respondent may have run out of minutes or would no longer be near a telephone by the time the interviewer was able to return the call. Over the course of the project, the data collection staff tracked respondents who were in jail or detention facilities, noting their release dates and keeping in touch with family members to enlist their assistance in setting up an interview as soon as possible after their release. Some respondents did not speak English or Spanish (the two languages in which the survey instrument was available), requiring translators to help locate and interview family members.
Advantages to Real-Time Tracking The tracking team used a real-time, web-based sample management system that provided a history of all searches and telephone contacts and enabled trackers to work together to locate particularly difficult-to-find respondents.
At times, the interviewer in the field would visit an address, find it vacant, and telephone the tracker, who would identify another lead or the name of a neighbor while the interviewer was still in the neighborhood. This close collaboration between the tracking and interviewing teams was especially helpful in avoiding the expense of a return trip when travelers were working in nonMTO cities.
Interviewers offered small monetary incentives (finder’s fees) of $5 to $10 to contact people who provided the interviewer or tracker with information on the location of a difficult-to-find respondent.