«Moving to opportunity voluMe 14, nuMber 2 • 2012 U.S. Department of Housing and Urban Development | Office of Policy Development and Research ...»
ISR added the finder’s fees late in the study, with some limited success. Near the end of the project, finder’s fees rose to $50 as interviewers worked to find the last of the hardest-to-locate respondents.
Cityscape 65Gebler, Gennetian, Hudson, Ward, and Sciandra
Nearly one-half (46 percent) of MTO families required tracking by the tracking team. Even after someone in the family was located, tracking did not stop. A total of 9 percent of sampled individuals (adults and youth) were referred to the tracking team after the family had been located. ISR was able to find and interview 48 percent of respondents referred to the tracking team. The remaining 52 percent were divided among refusals (5 percent), final noninterviews—such as respondents in institutional settings like jails or on military duty abroad—who were not reasonably accessible to the data collection staff (23 percent), and respondents who were randomly excluded from the sample as part of the second-stage subsampling design (24 percent).
Strategies To Maximize Respondent Participation After locating the respondent, the interviewer’s next challenge was to convince each selected participant in the MTO family to agree to complete an interview (and convince parents to give consent for their children to participate). The financial incentive was by far the most effective tool for obtaining respondent participation. Interviewers offered each respondent a $50 cash payment at the time of the interview and offered adult respondents a $25 cash payment for agreeing to provide a dried blood spot sample.
ISR implemented additional monetary incentives throughout the data collection period to help meet specific project goals. Consent to audio record the interviews was especially important for MTO, which used audio recordings for quality control checks and for questionnaire items designed to measure the effect of MTO on language patterns. Early in the project period, audio recording consent was lower than projected. In addition to working with interviewers to improve their persuasion skills and increase the number of respondents selected for audio recording, in December 2008, ISR added a $10 cash incentive for respondents who consented to having their interview recorded. These efforts resulted in an increase in the overall audio consent rate by 10 percentage points, from 74 to 84 percent.
End-Game Strategies ISR used several strategies at the end of the data collection period in an attempt to obtain the last few interviews needed to raise the response rate to the desired level. We offered additional cash incentives to respondents and interviewers, implemented two-stage subsampling to enable the interviewers to focus limited resources on a smaller number of cases, and extended the study period to allow more time to locate lost respondents and convince reluctant individuals to participate.
As mentioned previously, hiring shortfalls and attrition led to some cities having a smaller data collection staff than planned, with the result being that completion rates (and response rates) varied by site. Rather than implementing end-game strategies at one point in time for the entire sample, ISR added the end-game strategies and incentives on a city-by-city basis.
The survey fielding design employed two-stage subsampling to obtain responses from a representative subsample of hard-to-locate respondents. In stage 1, the ISR data collection staff attempted to contact and interview all the adults and youth in the sample frame. When the response rate at each site reached approximately 75 percent, the team selected a random subset of 35 percent of the remaining cases for more intensive interviewing efforts during stage 2. When calculating the ERR
and analyzing the survey data, respondents interviewed as part of stage 2 received an additional weighting factor so that they represent the other hard-to-reach respondents who were not selected for stage 2 and that ISR did not attempt to interview.
ISR implemented end-game cash incentives in each MTO city based on the completion rate for that city’s sample and coupled subsampling with an increase in respondent incentives. ISR increased the interview incentive by $25 once and then a second time, bringing the incentive offer up to $100 per interview (plus $25 for adults for the dried blood spot sample and $10 for audio recording). When ISR first added end-game incentives in each city, the data collection staff had flexibility in offering the incentives. The interviewer and the team leader discussed each case and decided which cases they were most likely to complete if they offered the additional financial incentive. For the final 2 months of data collection, interview incentives increased to $200 as the ISR data collection staff attempted to convince the most reluctant and elusive respondents to participate. The final phase of the end-game effort also included two additional options: completing the interview by telephone or completing a shorter version of the survey for a smaller incentive ($100).
The response rate for respondents to whom interviewers offered an end-game incentive was nearly identical to the response rate for respondents whom interviewers contacted during the end-game period of the study but to whom the interviewers did not offer the end-game incentive. Among respondents who had initially refused to do the interview, 49 percent of those who were offered an end-game incentive and 45 percent of those who were not offered the incentive completed interviews. These findings may reflect the fact that the team leaders used the end-game incentive offers very judiciously, encouraging interviewers to do their best to complete the interview without offering the additional monetary incentive if possible. By the time the end-game incentives were in place, the data collection staff was very experienced and had built up a large toolkit of effective introductions, and their tracking and persuasion skills were well refined.
One aspect of the end-game incentives apparent during the field period was the motivation that they provided to the interviewers. Having something new to say when calling respondents made it easier for interviewers to make additional contact attempts with respondents who had been avoiding them for months. ISR also gave interviewers the flexibility to offer small nonmonetary incentives, such as a small plant or gift, as part of the end-game strategy. These alternative incentives were sometimes effective in helping the interviewer gain access to the respondent’s home, thereby enabling the interviewer to pitch the study to a family member or the respondent in an attempt to elicit participation from a reluctant respondent.
Strategies To Retain and Motivate Data Collection Staff In addition to making efforts to select interviewers who were well suited to the type of interviewing MTO required, ISR focused considerable effort on maintaining staff morale and motivation over the course of a nearly 2-year data collection period. Successful strategies included the continuous efforts of the field supervisory staff, frequent communication among teams and between the ISR central office and the field staff, and a variety of monetary and nonmonetary incentives offered to interviewers and team leaders throughout the data collection period. The direct involvement of the ISR project managers and the NBER research team, along with the importance of the study topic, provided a great deal of motivation to the field staff and helped reduce attrition despite the many
Cityscape 67Gebler, Gennetian, Hudson, Ward, and Sciandra
challenges the study presented. The field supervisors of the data collection staff and ISR project managers incorporated continuous interviewer training into all field activities, including standardized training topics in the weekly team meetings, one-on-one coaching with team leaders, working in pairs, and sharing tips and tricks with each other.
One especially motivating technique used during the MTO long-term survey was having the NBER research team meet with the data collection staff. The interviewing and tracking teams met in person in each of the five MTO cities, when possible, and also used telephone conference calls to help contain costs. During the meetings, teams discussed the purpose of the study and progress to date, and, in later months, the NBER research team shared some preliminary demographic information or other simple results with the interviewers. The meetings also included time for discussion: the interviewers provided observations and stories from the field, asked the research team questions, and gathered information that they used in their introductions to help convince reluctant respondents of the importance of the MTO project. The field teams greatly appreciated these meetings, which often resulted in changes to materials or procedures based on interviewers’ comments and suggestions. The fact that the research team took the time to meet with—and listen to—the interviewing team was a very positive factor in maintaining high morale and commitment in the field.
ISR offered incentive programs for interviewers at several points in the study. At the end of each calendar year, ISR distributed production bonuses based on productivity and efficiency. In March 2009, ISR held a March Madness competition. The interviewers did not receive any additional compensation, but ISR project managers made a donation to a local food bank in each city based on the number of interviews completed during the month of March. ISR included fun, sports-inspired progress updates in weekly newsletters. This challenge led to a big spike in production and engendered a great deal of team spirit, with the teams adopting names (for example, the New York “Hard Knocks” and the Chicago “Terminators”) and team leaders even engaging in some good-natured competitive trash talk on weekly conference calls. Later in the study period, ISR offered interviewers bonuses for completing high-priority interviews ($15 for each high-priority interview completed and $20 if the interview had previously been coded as reluctant). In 2010, as the study was winding down and a very small number of cases remained to work, ISR offered remaining staff a retention bonus for staying on the project and a weekly incentive if they met goals for number of hours worked, followed the work plan for tracking, and made a specified number of contact attempts during the week.
In addition to offering monetary and nonmonetary incentives, ISR provided training opportunities for interviewers and team leaders to help the data collection staff remain productive and effective as the study progressed and as completing interviews became increasingly difficult. Team meetings and newsletter entries provided techniques for convincing reluctant respondents to participate, tracking strategies, and tips for successful interviewing. Newsletters also highlighted success stories, including naming staff to the “Century Club” when they completed 100 interviews and a weekly “kudos” segment recognizing individuals and teams for their efforts. All incentive programs included team leaders. The positive attitude of the team leaders was contagious, and their leadership was a big factor in the success of the field effort. In addition to including team leaders in the incentive programs, ISR developed a professional development series for team leaders, offering seminars on topics such as communication, motivating and getting the best from a team, and work/life integration (recognizing that working from a home office can be very challenging).
68 Moving to Opportunity Achieving MTO’s High Effective Response Rates: Strategies and Tradeoffs Coordination Between NBER and ISR Throughout the preproduction and data collection periods, the ISR project managers and the field supervisors of the data collection staff worked very closely with the NBER research team, which helped ensure that all parties clearly understood goals, priorities, and the realities of fieldwork.
When issues arose, the ISR project managers, field supervisors of the data collection staff, and NBER research team members discussed options and made joint decisions that best met the project needs within the limits and constraints of time, available resources, cost, and quality.
In addition to participating in weekly meetings and countless telephone calls and exchanging e-mails, ISR and NBER developed a wide variety of reports to facilitate the close monitoring and coordination of the field data collection effort. Statistical reports displayed, in tabular and graphical form, information about project cost, data quality, consent rates for components (for example, audio recording or blood spot collection), and completion and response rates. ISR modified the reports over time to better meet the needs of the ISR project managers, field supervisors of the data collection staff, and NBER research team members. This information exchange enabled the data collection staff to identify areas needing additional training or encouragement in the field and to develop programs to meet those needs. The field supervisory staff wrote weekly qualitative reports, providing context and stories from the field that helped to complete the picture of how data collection was progressing.
The ISR project managers, field supervisors of the data collection staff, and NBER research team members carefully monitored completion and response rates across cities, treatment and control groups, and respondent type (adults and youth) throughout the project. Such rates differed for a variety of reasons. Differences in staffing levels led to differential completion rates across sites (for example, the Section 8 group adult sample was released midway through the data collection period [February 2009], when additional funding was obtained) and completion rates naturally differed across the three groups. When differences in completion rates began to appear, ISR added a priority flag to the data collection protocol. This flag was a designation given to selected respondents, with the goal of improving sample balance in completion and response rates. ISR project managers instructed the field team to arrange its work so that team members called and tracked high-priority cases more aggressively, with the goal of completing as many high-priority interviews as possible.
As the study progressed and all interviews became more difficult to complete, ISR added a monetary incentive for each high-priority case that was completed as an interview.