FREE ELECTRONIC LIBRARY - Theses, dissertations, documentation

Pages:     | 1 || 3 | 4 |   ...   | 5 |

«Stephanie Riegg Cellini George Washington University School of Public Policy and Public Administration 805 21st Street, NW Washington, DC 20052 (202) ...»

-- [ Page 2 ] --

If half of the colleges in a state were randomly assigned to a new financial aid policy, while others continued with the status quo, one would in theory be able to judge differences in student enrollment and persistence under the two schemes. Though the exact number of observations needed to successfully carry out a random assignment experiment depends on many factors, such as the level of randomization (e.g. schools vs. students) and the outcome under study, in general the more observations one has, the more power one has in detecting significant differences between treatment and control groups. 10 Despite the simplicity of the approach and the accuracy of the estimates derived from random assignment experiments, the approach has some significant drawbacks. First, it is costly in terms of time and money. A research team must establish a close relationship with policymakers and program administrators to be allowed to carryout the experiment, they must also identify and solicit potential participants, design a lottery to assign participants to treatment and control groups, implement the program for the treatment group, and track both groups’ outcomes over time. Despite these substantial short-term costs, Cook (2002) argues that the costs of undertaking random assignment experiments are likely to be lower than other research designs in the long-run. He reasons that because fewer experiments are needed to achieve the See Raudenbush (1997), Raudenbush, Martinez, and Spybrook (2007), and Shadish, Cook, and Campbell (2002) for more details on sample size requirements.

same level of confidence in causal estimates, random assignment experiments are more costeffective in answering certain research questions. Moreover, the increased likelihood of obtaining accurate estimates of program effectiveness under random assignment will save money and time overall since policymakers will be more likely to implement the most appropriate and effective policies.

A second important practical challenge in conducting random assignment experiments is in implementing the treatment and measuring outcomes. Manipulating and isolating the effect of the treatment can be particularly difficult in educational settings. 11 Incomplete compliance may occur if some members of the assigned treatment group refuse treatment (e.g. they don’t avail themselves of the financial aid that is offered to them). Similarly, some control group members may seek treatment outside of the experiment (e.g. they obtain a different source of financial aid). Similar difficulties arise in measuring outcomes under differential attrition, where participants drop out of the experiment for potentially important unobservable reasons. Cook (2002) suggests that these types of problems can be minimized with proper monitoring and incentives for participation, and he stresses the need to study implementation quality in its own right. As discussed in more detail below, new methods, such as instrumental variables techniques, also offer promise in addressing some of these problems.

Third, random assignment introduces issues of equity that can be difficult to overcome.

This is a particularly important consideration in the case of financial aid research. In the student aid example above, can random assignment be justified if it means that some students with high incomes receive grants as part of the treatment group while some students with low incomes are left without grants in the control group? From an ethical perspective, the situation is For more details on measurement and compliance see Angrist and Krueger (1991), Chatterji (2005), Cook (2002), Cronbach et al. (1980), Shadish, Cook, and Campbell (2002).

troublesome, but there are ways to ameliorate the situation. In particular, one can pick a treatment and control group from a more equal pool of students. If all students are equally eligible for the aid—for example, all are in households with incomes barely below the poverty line—then random assignment is a bit less troubling on this dimension. Along the same lines, programs that are oversubscribed are good candidates for random assignment. If for example, there are a finite number of grants available and a large number of equally-qualified and eligible students, one could enter these students into a lottery for the grants. In some cases, this process might be more justifiable than selecting recipients based on other arguably arbitrary measures— such as a students’ state of residence or gender.

One final drawback of random assignment experiments is their lack of external validity or generalizability. Every random assignment experiment is done in the context of a specific treatment in a specific location, making inferences to other contexts difficult. For example, a random assignment evaluation of a financial aid policy in San Francisco may show that every $1,000 spent on a program induces 20 more students to enroll. However, the same policy randomly evaluated in Milwaukee might show negligible effects, because the students of Milwaukee face a very different set of circumstances than those in San Francisco. In this case random assignment can control for individual differences in students—it is internally valid—but it cannot account for the context of the experiment. Still, with many small-scale context-specific experiments, patterns may emerge that lend themselves to generalization. Moreover, there has been renewed attention in recent years to the importance of exploring and understanding the specific context of each experiment, and employing multiple research methods such as ethnography and descriptive quantitative analysis to describe, understand, and evaluate the context of a particular random assignment experiment before implementation. 12 Despite its limitations, random assignment is becoming increasingly popular in education research and the Department of Education’s Institute for Education Sciences has made a concerted effort to fund studies of this type in the past few years (Myers and Dynarski 2003, U.S.

Department of Education 2003). As such, a substantial literature has grown in detailing and overcoming challenges of implementing random assignment experiments in education. 13 At the same time, a literature has grown debating its role, and the role of the federal government, in shaping the education research agenda. 14 Some of the most notable random assignment experiments in education are based on oversubscribed elementary school voucher programs in cities like Milwaukee (Rouse 1998) and New York (Meyer and Mayer 2003; Mayer et al. 2002; Krueger and Zhu 2003), among others.

Other large-scale random assignment projects include an evaluation of the effects of class size in the Tennessee Student/Teacher Achievement Ratio (STAR) Experiment (Krueger 1999), Upward Bound college readiness program (Meyers and Schirm 1999; Meyers et al. 2004), the Quantum Opportunities After-School Program (Hahn, Leavitt, and Aron 1994), and the Opening Doors Community College Program (Purnell and Blank 2004). The latter program randomly assigned low-income community college students to the “Opening Doors Program” which offered enhanced student services, curricular and instructional reform, and financial aid in form of See for example, Chatterji (2005), Cohen, Raudenbush, and Ball (2003), Dowd and Tong (2007), Erickson and Gutierrez (2002), Raudenbush (2005).

For more details on the implementation of random assignment experiments see Chatterji (2005), Cohen, Raudenbush, and Ball (2003), Cook (2002), Cronbach and Associates (1980), Raudenbush (1997), Raudenbush, Martinez, and Spybrook (2007), and Shadish, Cook, and Campbell (2002).

See Berliner (2002), Chatterji (2005), Dowd (forthcoming), Dowd and Tong (2007), Erickson and Gutierrez (2002), Feuer, Towne, and Shavelson (2002a, 200b), and St. Pierre (2002).

vouchers for transportation, child care, and books. This study is among the first to use random assignment to study financial aid in postsecondary education.

3.2. Multivariate Regression Multivariate regression—including linear and nonlinear estimation—is still the most often used method for addressing omitted variable bias in the financial aid literature (see for example Curs and Singell 2002; DesJardins et al. 2006; Dowd 2004, Dowd and Coury 2006, Fuller, Manski, and Wise 1982, Jackson 1978, 1990; Moore et al. 1991; Seneca and Taussig 1987; St. John et al. 2003; and St. John 1990, 1999). With this approach, one can control for all observed and measurable variables that were previously omitted from the univariate model.

Adding a vector, X s, that includes every exogenously-determined state-level variable that might be related to enrollment to equation (1), will remove these factors from the error term and reduce the bias of the univariate OLS estimator. Again, using the linear specification for simplicity, the

equation is now:

–  –  –

In our case, economic theory suggests that X s would include variables such as population, unemployment rate, poverty rate, percent minority—all of the measurable variables that might make enrollments in one state different from another. However, it is worth emphasizing that the control variables should be determined exogenously (outside the model) irrespective of the state’s enrollment or financial aid policies, or they will introduce further bias.

In the student-level analysis the equation looks similar:

–  –  –

In this case, X i represents student-level characteristics that might include parental income or education level, academic achievement in high school, the type of high school attended, and the student’s race, ethnicity, gender, and age. These variables are generally agreed to be important determinants of enrollment in both the economics and education literature, though perhaps for different reasons. In economics, these variables are typically derived from models of human capital investment and expected lifetime utility maximization developed by Mincer (1958), Becker (1964), Kohn, Manski and Mundel (1976), Fuller, Manski, and Wise (1982), and Manski and Wise (1983). In education, the choice of variables has been influenced by the work of Hossler, Braxton, and Coopermith (1989), Hossler and Gallagher (1987), Weiler (1990), Welki and Navratil (1987), among others. While both traditions emphasize the role of student and institutional characteristics—including financial aid—the education research has gone further in integrating the role of contextual and sociocultural factors such as parental support, peers’ college plans, and student perceptions.

While multivariate regression analysis is relatively easy to understand and carry out, the only way this method can fully eliminate omitted variable bias is if all possible omitted variables that might be correlated with enrollment are included in the model,

–  –  –

selection are known precisely, 15 most often economists argue that unobservable heterogeneity remains—for example, the preferences of voters, the public perception of community colleges, and other unobservable or unmeasurable characteristics that make Wisconsin different from California. In the student financial aid case, innate ability, a student’s knowledge about college and financial aid programs, and sociocultural factors such as a family’s expectations of college A study of the Vietnam-era draft by Angrist (1998) comes close.

enrollment, are unobservable, rendering straightforward multivariate regression unconvincing.

Moreover, even if some of these variables can be measured and included in multivariate models they can introduce additional bias if they are endogenously determined.

As Angrist and Krueger (2001) point out, it is rare that a theory specifies all of the variables that must be controlled for in a given relationship and even if it did, it would be nearly impossible to observe and measure them all. While rich data sets with extensive background characteristics are helpful in this regard, omitted variable bias remains problematic in multivariate regression analysis. In the following sections, I discuss several methods that can be used to ameliorate this problem.

3.3. Proxy Variables One of the simplest methods to mitigate bias from unobservable omitted variables is the use of proxy variables. This approach involves finding suitable observable variables that can “stand in” for each unobservable. Adding a proxy, denoted Ps, to the model in equation (5)

above, we have:

–  –  –

There are two key conditions that must be met by a potential proxy. First, the proxy variable must be ignorable, or redundant, in the equation. Mathematically, E ( Enrolls | Fs, X s, Ps ) = E ( Enrolls | Fs, X s ). Intuitively, the proxy must be irrelevant in explaining enrollment once Fs and X s have been controlled for.

Assume for a moment that there is only one omitted variable in the student-level analysis—a student’s unobservable innate ability (in fact, this is the classic case in the economics of education literature, see Angrist and Krueger 1999 and Card 1999). If we believe that a student’s performance on an IQ test is a good measure his or her innate ability, the IQ score could serve as a proxy for ability since the redundancy condition is met. That is, if we knew a student’s true innate ability we would not need to include IQ in the equation (Wooldridge 2002).

The second condition that must be met for a good proxy, is that it must be sufficiently closely related to the omitted variable, so that once Ps is included in the equation, there is no remaining variation in the unobservable variable that is correlated with Fs or X s. If this condition does not hold—and we can never know with certainty if it does—then omitted variable bias may remain. Still, as Wooldridge points out, if it does not hold, a reasonable “imperfect proxy” may still reduce omitted variable bias.

Pages:     | 1 || 3 | 4 |   ...   | 5 |

Similar works:

«8 POLICY ISSUES FACING THE MARKET FOR CREDIT DERIVATIVES Darrell Duffie THE FINANCIAL CRISIS has prompted calls for revamping the market for credit derivatives. For example, in a July 2008 speech, Fed Chairman Ben Bernanke noted that, “The Federal Reserve, together with other regulators and the private sector, is engaged in a broad effort to strengthen the financial infrastructure. In doing so, we aim not only to help make the financial system better able to withstand future shocks but...»

«The Local Labor Market Spillover Effects of Military Personnel Contractions Ben Zou University of Maryland Oct 2013 [Preliminary and incomplete, please do not circulate or cite] [Contains color graphs] Abstract This paper estimates the local spillover effects of military personnel on private sector employment in county level labor markets. I exploit a policy experiment in the United States when, between 1988 and 2000, the size of the military was cut by about 30 percent. The bulk of the cuts...»

«UNDERSTANDING POLICY FORMULATION: A TOOLS PERSPECTIVE 1 John Turnpenny Senior Lecturer in the School of Politics, Philosophy, Language and Communication Studies, University of East Anglia, Norwich, UK Andrew Jordan Professor of Environmental Policy, Tyndall Centre School of Environmental Sciences, University of East Anglia.ABSTRACT Policy formulation is a key element of policy-making, but in spite of increasing interest it is still greatly under-explored. This paper develops a research agenda...»

«2014 HANDBOOK OF IMF FACILITIES FOR LOW-INCOME February 2015 COUNTRIES IMF staff regularly produces papers proposing new IMF policies, exploring options for reform, or reviewing existing IMF policies and operations. The following document has been released and is included in this package: The Policy Paper on 2014 Handbook of IMF Facilities for Low-Income Countries, prepared  by IMF staff and completed in July 2014 and sent to the Executive Board for information on August 28, 2014. The policy...»

«Assessment & Inquiry-Based Science Education: Issues in Policy and Practice Assessment & Inquiry-Based Science Education: Issues in Policy and Practice In recent years there has been a rapid expansion of interest in inquiry-based science education (IBSE). Classroom and laboratory practices and materials which encourage students to take an active part in making sense of events and phenomena in the world around are being promoted and developed through pilot projects in countries across the globe....»

«TOWARDS A REEVALUATION OF THE TONINÁ POLITY Eric TaladoirE Unité Mixte de Recherche, “Archéologie des Amériques”, Université Paris 1 Panthéon-Sorbonne AbstrAct: Among the numerous polities of the Usumacinta region, Toniná stands as the worst defined. In spite of its reduced population, Toniná developed an aggressive policy, and won several victories upon close-by and distant cities as well. This article tries, from the available archaeological and epigraphic data, to draw a more...»

«The Centre for Globalisation, Education and Societies is based at the University of Bristol and is coordinated by Professor Susan L. Robertson. On-Line Papers – Copyright This online paper may be cited or briefly quoted in line with the usual academic conventions, and for personal use. However, this paper must not be published elsewhere (such as mailing lists, bulletin boards etc.) without the author’s explicit permission.If you copy this paper, you must: • include this copyright note....»

«Change of Ownership American General Life Insurance Company The United States Life Insurance Company in the City of New York A member of American International Group, Inc. (AIG) In this form, the Company refers to the insurance company whose name is checked above. The Company shown above is solely responsible for the obligation and payment of benefits under any policy that it may issue. No other Company is responsible for such obligations or payments.Mailing Instructions: Send form(s) to:...»

«CHAPTER 2 addition to an annual salary. THE SUPERVISOR AS POLICYMAKER The strength of the office of township supervisor as a policymaking position rests largely on the image the public accords the office and the political strength of the officeholder. How the public views the office undoubtedly varies from township to township, but it seems fair to say that most township citizens consider the supervisor the chief elected township officer. To the extent that this is so, the township supervisor...»

«The value of words: Narrative as evidence in policymaking Dmitry Epstein, Josiah B. Heidt, and Cynthia R. Farina Cornell eRulemaking Initiative, Cornell Law School Policymakers today rely primarily on statistical, financial, and other forms of technical data as their basis for decision-making. Yet, there is a potentially underestimated value in substantive reflections of the members of the public who will be affected by a particular piece of regulation. We discuss the value of narratives as...»

«Issue paper no. 140 Ambivalence and contradiction A review of the policy environment in Tanzania in relation to pastoralism A.Z. Mattee ERETO II NGORONGORO PASTORALIST PROJECT and M. Shem March 2006 NGORONGORO – TANZANIA Ambivalence and contradiction A review of the policy environment in Tanzania in relation to pastoralism A.Z. Mattee and M. Shem Issue Paper no. 140 About the authors Amon Z. Mattee was born in Tanzania and holds a B.Sc. in Agriculture (Hons) from the University of Dar es...»

«THE DEMOCRATIC QUALITIES OF COURTS: A CRITICAL ANALYSIS OF THREE ARGUMENTS Richard Bellamy The democratic critique of judicial review by constitutional courts has prompted its defenders to counter that courts have democratic qualities as good as, and in certain respects even stronger than, conventional democratic politics. This article offers a critical analysis of three arguments favouring this approach. The first argues that constitutional courts operate as exemplars of democratic...»

<<  HOME   |    CONTACTS
2016 www.theses.xlibx.info - Theses, dissertations, documentation

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.