«Unresponsive and Unpersuaded: The Unintended Consequences of a Voter Persuasion Effort Michael A. Bailey1 • Daniel J. Hopkins2 • Todd Rogers3 Ó ...»
allows the ABB to more accurately reﬂect variability from the imputation. One can draw the donor observations with equal probability in each iteration, which effectively assumes that the missingness is ignorable conditional on the observed covariates. But importantly, researchers can also take weighted draws from the donor pool, which is the equivalent of placing an informative prior on the missing outcome data (Siddique and Belin 2008b). This allows researchers to relax the ignorability assumption, and to build in additional information about the direction and size of any bias.
Irrespective of the prior, we then build a model of the outcome using the covariates for the respondents with no missing outcome data, being sure to weight the donor observations by the number of times they were drawn in each iteration of ^ the bootstrap. The subsequent step is to predict Y for all observations—both donor and donee—by applying that model to the covariates X. For each observation with a missing outcome—there are 33,025 in this example—we next need to draw a ‘‘donor’’ observation that provides an outcome.
Following Siddique and Belin (2008b), we do so by estimating a distance metric for each observation i as follows:
Di ¼ ðj^0 À yi j þ dÞk, where d is a positive number which avoids distances of ^ y zero.24 For each missing observation, an outcome is imputed from a donor chosen with a probability inversely proportional to the distance Di. As k grows large, note that the algorithm chooses the most similar observation in the donor pool with high probability, while a k of zero is equivalent to drawing any observation with equal probability.25 Unlike a single-shot hot deck imputation, this approach does account for imputation uncertainty—and here, we ﬁt our standard logistic regression model to 5 separately imputed data sets and then combine the answers using the appropriate rules (Rubin and Schenker 1986; King et al. 2001). Yet there is an important potential limitation to this technique. While running the algorithm multiple times will address the uncertainty stemming from the imputation of missing observations, it will not address the uncertainty stemming from small donor pools—and the reweighting in the non-ignorable ABB has the potential to exacerbate this concern (Cranmer et al. 2013).26 We ﬁrst run the Approximate Bayesian Bootstrap assuming ignorablility (which means the prior is zero) and setting k ¼ 3. Table 11 shows that, as we reported in the manuscript, such a model estimates the average treatment effect of canvassing to be
-1.65 percentage points, with a corresponding 95% conﬁdence interval from -3.29 to -0.01. That estimate is similar to those recovered using listwise deletion. We also report additional results after adding an informative prior which reduces the share of respondents who back Obama from 57.5% in the observed group to 54.0% in the unobserved group. We chose the magnitude of the decline–3.5 percentage points–to approximate the largest decline in survey response observed across any of Here, d is set to 0.0001.
Siddique and Belin (2008a) report that a value of k ¼ 3 works well in their substantive application, while Siddique and Belin (2008b) recommend values between 1 and 2.
Still, even in light of this potential to under-estimate variance, Demirtas et al. (2007) demonstrate that the small-sample properties of the original ABB are superior when compared to would-be corrections.
This table reports the lower bounds and upper bounds for several Approximate Bayesian Bootstrap estimations. The lower and upper bounds are the 2.5th and 97.5th percentiles of the average treatment effect. The units are percentage points Note: ‘‘Phone score’’ refers to the 44,875 experimental subjects for whom a pre-treatment phone match score was available via Catalist. The prior indicates the level by which Obama support was adjusted in among unobserved respondents. As k increases, the preference for matching similar observations in the ABB increases the turnout groups. In other words, in light of the differential attrition identiﬁed above, 3.5 percentage points is a large but still plausible difference between the observed and unobserved populations conditional on observed covariates. Here, the estimated treatment effect becomes -1.73 percentage points, with a 95% conﬁdence interval from -3.34 to -0.05. This result is essentially unchanged from the result with no prior. The table then presents various combinations of the prior and the k parameter, with little difference across the speciﬁcations except that reducing k below two (which means we are reducing the penalty for matching less similar observations) appears to increase the uncertainty regarding the estimated treatment effect. We also report results using all observations with, again, similar results.
Inverse Propensity Weighting
Inverse propensity weighting (IPW) is an alternative approach to dealing with attrition that uses some of the same building blocks as multiple imputation: it leverages information in the relationships among observed covariates to reweight the observed data such that they approximate the full data set (Glynn et al. 2010).
Speciﬁcally, we ﬁrst use logistic regression on the full sample27 to estimate a model of survey response. We employ the same model speciﬁcation as above, with the exception that we drop our measure of age because it has substantial missingness. From the model, we generate a predicted probability of survey IPW requires data that are fully observed with the exception of the missing outcome. We thus set aside 20 respondents who were missing data for covariates other than age or Obama support.
response for each respondent, estimates which vary from 0.13 to 0.35. For the 12,439 fully observed respondents, we then calculate the average treatment effect of canvassing, weighted by the inverse predicted probability of responding to the survey. Doing so, the estimated treatment effect of canvassing is -1.79 percentage points, with a 95% conﬁdence interval from -3.52 to -0.05 percentage points.
Heckman selection models assume that the errors in the selection equation and outcome equation are distributed bivariate normally. With this assumption, the expected value of the error in the outcome equation conditional on selection can be represented with an inverse Mills’ ratio. There is considerable disagreement in the
literature about the appropriateness of this assumption. Some ﬁnd it implausible, given that the key assumption is about the joint distribution of unobserved quantities. Others ﬁnd the approach more plausible than assuming away the correlation of errors across selection and outcome equations as is done in other selection models.
Table 12 shows results from several speciﬁcations of a Heckman selection model. In the ﬁrst column no additional controls are included. In the second column, the controls listed at the bottom of the table are included. In the third column, the sample is limited to those who voted in 2 or fewer previous elections in the dataset.
The results are qualitatively similar to the non-parametric selection model. The signiﬁcant (or nearly so) q parameter indicates that there is some modest correlation between errors in the two equations. A statistically signiﬁcant q parameter indicates that the errors are correlated, a necessary, but not sufﬁcient condition for selection bias. In this case, since the estimates are similar to methods that assume no correlation of errors, there does not appear to be selection bias.
References Adams, W. C., & Smith, D. J. (1980). Effects of telephone canvassing on turnout and preferences: A ﬁeld experiment. Public Opinion Quarterly, 44(3), 389–395.
Albertson, B., & Busby, J. W. (2015). Hearts or minds? Identifying persuasive messages on climate change. Research & Politics.
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identiﬁcation of causal effects using instrumental variables (with discussion). Journal of the American Statistical Association, 91, 444–455.
Ansolabehere, S., & Hersh, E. (2011). Who really votes? In P. M. Sniderman & B. Highton (Eds.), Facing the challenge of democracy: Explorations in the analysis of public opinion and political participation. Princeton University Press.
Ansolabehere, S., & Hersh, E. (2012). Validation: What big data reveal about survey misreporting and the real electorate. Political Analysis, 20(4):437–459.
Arceneaux, K. (2005). Using cluster randomized ﬁeld experiments to study voting behavior. The Annals of the American Academy of Political and Social Science, 601(1), 169–179.
Arceneaux, K. (2007). I’m asking for your support: The effects of personally delivered campaign messages on voting decisions and opinion formation. Quarterly Journal of Political Science, 2(1), 43–65.
Arceneaux, K., & Kolodny, R. (2009). Educating the least informed: Group endorsements in a grassroots campaign. American Journal of Political Science, 53(4), 755–770.
Arceneaux, K., & Nickerson, D. W. (2009). Who is mobilized to vote? A re-analysis of 11 ﬁeld experiments. American Journal of Political Science, 53(1), 1–16.
Bechtel, M. M., Hainmueller, J., Hangartner, D., & Helbling, M. (2014). Reality bites: The limits of framing effects for salient and contested policy issues. Political Science Research and Methods (forthcoming).
Broockman, D. E., & Green, D. P. (2014). Do online advertisements increase political candidates’ name recognition or favorability? evidence from randomized ﬁeld experiments. Political Behavior, 36, 263–289.
Cardy, E. A. (2005). An experimental ﬁeld study of the GOTV and persuasion effects of partisan direct mail and phone calls. The Annals of the American Academy of Political and Social Science, 601(1), 28–40.
Cranmer, S. J., & Gill, J. (2013). We have to be discrete about this: A non-parametric imputation technique for missing categorical data. British Journal of Political Science, 43(2), 425–449.
Das, M., Newey, W. K., & Vella, F. (2003). Nonparametric estimation of sample selection models. The Review of Economic Studies, 70(1), 33–58.
Demirtas, H., Arguelles, L. M., Chung, H., & Hedeker, D. (2007). On the performance of bias-reduction techniques for variance estimation in approximate Bayesian bootstrap imputation. Computational Statistics & Data Analysis, 51(8), 4064–4068.
Enos, R. D., Fowler, A., & Vavreck, L. (2014). Increasing inequality: The effect of GOTV mobilization on the composition of the electorate. The Journal of Politics, 76(1), 273–288.
Enos, R. D., & Hersh, E. D. (2015). Party activists as campaign advertisers: The ground campaign as a principal-agent problem. American Political Science Review, 109(02), 252–278.
Gerber, A., Karlan, D., & Bergan, D. (2009). Does the media matter? A ﬁeld experiment measuring the
effect of newspapers on voting behavior and political opinions. American Economic Journal:
Applied Economics, 1(2), 35–52.
Gerber, A., & Green, D. (2000). The effects of canvassing, telephone calls, and direct mail on voter turnout: A ﬁeld experiment. American Political Science Review, 94(3), 653–663.
Gerber, A. S., Kessler, D. P., & Meredith, M. (2011). The persuasive effects of direct mail: A regression discontinuity based approach. Journal of Politics, 73(1), 140–155.
Gerber, A. S., & Green, D. P. (2012). Field experiments: Design, analysis, and interpretation. New York, NY: W.W. Norton and Company.
Gerber, A. S., Huber, G. A., Doherty, D., Dowling, C. M., & Hill, S. J. (2013). Who wants to discuss vote choices with others? Polarization in preferences for deliberation. Public Opinion Quarterly, 77(2), 474–496.
Gerber, A. S., Huber, G. A., & Washington, E. (2010). Party afﬁliation, partisanship, and political beliefs:
A ﬁeld experiment. American Political Science Review, 104(04), 720–744.
Gerber, A. S., Gimpel, J. G., Green, D. P., & Shaw, D. R. (2011). How large and long-lasting are the persuasive effects of televised campaign ads? Results from a randomized ﬁeld experiment.
American Political Science Review, 105(01), 135–150.
Glynn, A. N., & Quinn, K. M. (2010). An introduction to the augmented inverse propensity weighted estimator. Political Analysis, 18(1), 36–56.
Green, D. P., & Gerber, A. S. (2008). Get out the vote: How to increase voter turnout. Washington, DC:
Brookings Institution Press.
Heckman, J. (1976). The common structure of statistical models of truncation, sample selectionand limited dependent variables, and simple estimator for such models. Annals of Economic and Social Measurement, 5, 475–492.
Hersh, E. D. (2015). Hacking the electorate: How campaigns perceive voters. New York, NY: Cambridge University Press.
Hersh, E. D., & Schaffner, B. F. (2013). Targeted campaign appeals and the value of ambiguity The Journal of Politics, 75(02), 520–534.
Hopkins, D. J. (2009). No more wilder effect, never a Whitman effect: When and why polls mislead about black and female candidates. The Journal of Politics, 71(3), 769–781.
Huber, G. A., & Arceneaux, K. (2007). Identifying the persuasive effects of presidential advertising.
American Journal of Political Science, 51(4), 957–977.
Imai, K., King, G., & Stuart, E. A. (2008). Misunderstandings between experimentalists and observation lists about causal inference. Journal of the Royal Statistical Society: Series A, 171(2), 481–502.
Issenberg, S. (2012). Obama Does It Better. Slate.
King, G., Honaker, J., Joseph, A., & Scheve, K. (2001). Analyzing incomplete political science data: An alternative algorithm for multiple imputation. American Political Science Review, 95(1), 49–69.
Ladd, J. M., & Lenz, G. S. (2009). Exploiting a rare communication shift to document the persuasive power of the news media. American Journal of Political Science, 53(2), 394–410.
Little, R. J., D’Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., et al. (2012). The prevention and treatment of missing data in clinical trials. New England Journal of Medicine, 367(14), 1355–1360.