«Stephanie Riegg Cellini George Washington University School of Public Policy and Public Administration 805 21st Street, NW Washington, DC 20052 (202) ...»
If half of the colleges in a state were randomly assigned to a new financial aid policy, while others continued with the status quo, one would in theory be able to judge differences in student enrollment and persistence under the two schemes. Though the exact number of observations needed to successfully carry out a random assignment experiment depends on many factors, such as the level of randomization (e.g. schools vs. students) and the outcome under study, in general the more observations one has, the more power one has in detecting significant differences between treatment and control groups. 10 Despite the simplicity of the approach and the accuracy of the estimates derived from random assignment experiments, the approach has some significant drawbacks. First, it is costly in terms of time and money. A research team must establish a close relationship with policymakers and program administrators to be allowed to carryout the experiment, they must also identify and solicit potential participants, design a lottery to assign participants to treatment and control groups, implement the program for the treatment group, and track both groups’ outcomes over time. Despite these substantial short-term costs, Cook (2002) argues that the costs of undertaking random assignment experiments are likely to be lower than other research designs in the long-run.
same level of confidence in causal estimates, random assignment experiments are more costeffective in answering certain research questions. Moreover, the increased likelihood of obtaining accurate estimates of program effectiveness under random assignment will save money and time overall since policymakers will be more likely to implement the most appropriate and effective policies.
A second important practical challenge in conducting random assignment experiments is in implementing the treatment and measuring outcomes. Manipulating and isolating the effect of the treatment can be particularly difficult in educational settings. 11 Incomplete compliance may occur if some members of the assigned treatment group refuse treatment (e.g. they don’t avail themselves of the financial aid that is offered to them). Similarly, some control group members may seek treatment outside of the experiment (e.g. they obtain a different source of financial aid). Similar difficulties arise in measuring outcomes under differential attrition, where participants drop out of the experiment for potentially important unobservable reasons. Cook (2002) suggests that these types of problems can be minimized with proper monitoring and incentives for participation, and he stresses the need to study implementation quality in its own right. As discussed in more detail below, new methods, such as instrumental variables techniques, also offer promise in addressing some of these problems.
Third, random assignment introduces issues of equity that can be difficult to overcome.
This is a particularly important consideration in the case of financial aid research. In the student aid example above, can random assignment be justified if it means that some students with high incomes receive grants as part of the treatment group while some students with low incomes are left without grants in the control group? From an ethical perspective, the situation is For more details on measurement and compliance see Angrist and Krueger (1991), Chatterji (2005), Cook (2002), Cronbach et al. (1980), Shadish, Cook, and Campbell (2002).
troublesome, but there are ways to ameliorate the situation. In particular, one can pick a treatment and control group from a more equal pool of students. If all students are equally eligible for the aid—for example, all are in households with incomes barely below the poverty line—then random assignment is a bit less troubling on this dimension. Along the same lines, programs that are oversubscribed are good candidates for random assignment. If for example, there are a finite number of grants available and a large number of equally-qualified and eligible students, one could enter these students into a lottery for the grants. In some cases, this process might be more justifiable than selecting recipients based on other arguably arbitrary measures— such as a students’ state of residence or gender.
One final drawback of random assignment experiments is their lack of external validity or generalizability. Every random assignment experiment is done in the context of a specific treatment in a specific location, making inferences to other contexts difficult. For example, a random assignment evaluation of a financial aid policy in San Francisco may show that every $1,000 spent on a program induces 20 more students to enroll. However, the same policy randomly evaluated in Milwaukee might show negligible effects, because the students of Milwaukee face a very different set of circumstances than those in San Francisco. In this case random assignment can control for individual differences in students—it is internally valid—but it cannot account for the context of the experiment. Still, with many small-scale context-specific experiments, patterns may emerge that lend themselves to generalization. Moreover, there has been renewed attention in recent years to the importance of exploring and understanding the specific context of each experiment, and employing multiple research methods such as ethnography and descriptive quantitative analysis to describe, understand, and evaluate the context of a particular random assignment experiment before implementation. 12 Despite its limitations, random assignment is becoming increasingly popular in education research and the Department of Education’s Institute for Education Sciences has made a concerted effort to fund studies of this type in the past few years (Myers and Dynarski 2003, U.S.
Department of Education 2003). As such, a substantial literature has grown in detailing and overcoming challenges of implementing random assignment experiments in education. 13 At the same time, a literature has grown debating its role, and the role of the federal government, in shaping the education research agenda. 14 Some of the most notable random assignment experiments in education are based on oversubscribed elementary school voucher programs in cities like Milwaukee (Rouse 1998) and New York (Meyer and Mayer 2003; Mayer et al. 2002; Krueger and Zhu 2003), among others.
Other large-scale random assignment projects include an evaluation of the effects of class size in the Tennessee Student/Teacher Achievement Ratio (STAR) Experiment (Krueger 1999), Upward Bound college readiness program (Meyers and Schirm 1999; Meyers et al. 2004), the Quantum Opportunities After-School Program (Hahn, Leavitt, and Aron 1994), and the Opening Doors Community College Program (Purnell and Blank 2004). The latter program randomly assigned low-income community college students to the “Opening Doors Program” which offered enhanced student services, curricular and instructional reform, and financial aid in form of See for example, Chatterji (2005), Cohen, Raudenbush, and Ball (2003), Dowd and Tong (2007), Erickson and Gutierrez (2002), Raudenbush (2005).
For more details on the implementation of random assignment experiments see Chatterji (2005), Cohen, Raudenbush, and Ball (2003), Cook (2002), Cronbach and Associates (1980), Raudenbush (1997), Raudenbush, Martinez, and Spybrook (2007), and Shadish, Cook, and Campbell (2002).
See Berliner (2002), Chatterji (2005), Dowd (forthcoming), Dowd and Tong (2007), Erickson and Gutierrez (2002), Feuer, Towne, and Shavelson (2002a, 200b), and St. Pierre (2002).
vouchers for transportation, child care, and books. This study is among the first to use random assignment to study financial aid in postsecondary education.
3.2. Multivariate Regression Multivariate regression—including linear and nonlinear estimation—is still the most often used method for addressing omitted variable bias in the financial aid literature (see for example Curs and Singell 2002; DesJardins et al. 2006; Dowd 2004, Dowd and Coury 2006, Fuller, Manski, and Wise 1982, Jackson 1978, 1990; Moore et al. 1991; Seneca and Taussig 1987; St. John et al. 2003; and St. John 1990, 1999). With this approach, one can control for all observed and measurable variables that were previously omitted from the univariate model.
Adding a vector, X s, that includes every exogenously-determined state-level variable that might be related to enrollment to equation (1), will remove these factors from the error term and reduce the bias of the univariate OLS estimator. Again, using the linear specification for simplicity, the
equation is now:
In our case, economic theory suggests that X s would include variables such as population, unemployment rate, poverty rate, percent minority—all of the measurable variables that might make enrollments in one state different from another. However, it is worth emphasizing that the control variables should be determined exogenously (outside the model) irrespective of the state’s enrollment or financial aid policies, or they will introduce further bias.
In the student-level analysis the equation looks similar:
In this case, X i represents student-level characteristics that might include parental income or education level, academic achievement in high school, the type of high school attended, and the student’s race, ethnicity, gender, and age. These variables are generally agreed to be important determinants of enrollment in both the economics and education literature, though perhaps for different reasons. In economics, these variables are typically derived from models of human capital investment and expected lifetime utility maximization developed by Mincer (1958), Becker (1964), Kohn, Manski and Mundel (1976), Fuller, Manski, and Wise (1982), and Manski and Wise (1983). In education, the choice of variables has been influenced by the work of Hossler, Braxton, and Coopermith (1989), Hossler and Gallagher (1987), Weiler (1990), Welki and Navratil (1987), among others. While both traditions emphasize the role of student and institutional characteristics—including financial aid—the education research has gone further in integrating the role of contextual and sociocultural factors such as parental support, peers’ college plans, and student perceptions.
While multivariate regression analysis is relatively easy to understand and carry out, the only way this method can fully eliminate omitted variable bias is if all possible omitted variables that might be correlated with enrollment are included in the model,
selection are known precisely, 15 most often economists argue that unobservable heterogeneity remains—for example, the preferences of voters, the public perception of community colleges, and other unobservable or unmeasurable characteristics that make Wisconsin different from California. In the student financial aid case, innate ability, a student’s knowledge about college and financial aid programs, and sociocultural factors such as a family’s expectations of college A study of the Vietnam-era draft by Angrist (1998) comes close.
enrollment, are unobservable, rendering straightforward multivariate regression unconvincing.
Moreover, even if some of these variables can be measured and included in multivariate models they can introduce additional bias if they are endogenously determined.
As Angrist and Krueger (2001) point out, it is rare that a theory specifies all of the variables that must be controlled for in a given relationship and even if it did, it would be nearly impossible to observe and measure them all. While rich data sets with extensive background characteristics are helpful in this regard, omitted variable bias remains problematic in multivariate regression analysis. In the following sections, I discuss several methods that can be used to ameliorate this problem.
3.3. Proxy Variables One of the simplest methods to mitigate bias from unobservable omitted variables is the use of proxy variables. This approach involves finding suitable observable variables that can “stand in” for each unobservable. Adding a proxy, denoted Ps, to the model in equation (5)
above, we have:
There are two key conditions that must be met by a potential proxy. First, the proxy variable must be ignorable, or redundant, in the equation. Mathematically, E ( Enrolls | Fs, X s, Ps ) = E ( Enrolls | Fs, X s ). Intuitively, the proxy must be irrelevant in explaining enrollment once Fs and X s have been controlled for.
Assume for a moment that there is only one omitted variable in the student-level analysis—a student’s unobservable innate ability (in fact, this is the classic case in the economics of education literature, see Angrist and Krueger 1999 and Card 1999). If we believe that a student’s performance on an IQ test is a good measure his or her innate ability, the IQ score could serve as a proxy for ability since the redundancy condition is met. That is, if we knew a student’s true innate ability we would not need to include IQ in the equation (Wooldridge 2002).
The second condition that must be met for a good proxy, is that it must be sufficiently closely related to the omitted variable, so that once Ps is included in the equation, there is no remaining variation in the unobservable variable that is correlated with Fs or X s. If this condition does not hold—and we can never know with certainty if it does—then omitted variable bias may remain. Still, as Wooldridge points out, if it does not hold, a reasonable “imperfect proxy” may still reduce omitted variable bias.