«Policy Research And The Policy Process: Do The Twain Ever Meet? JAMES L. GARRETT YASSIR ISLAM This Gatekeeper Series is produced by the International ...»
GATEKEEPER SERIES No. 74
and Rural Livelihoods
Policy Research And The
Do The Twain Ever Meet?
JAMES L. GARRETT
This Gatekeeper Series is produced by the International Institute for Environment
and Development to highlight key topics in the field of sustainable agriculture. Each paper reviews a selected issue of contemporary importance and draws preliminary conclusions of relevance to development activities. References are provided to important sources and background material.
The Swedish International Development Authority (SIDA) funds the series, which is aimed especially at the field staff, researchers and decision makers of such agencies.
James L. Garrett is a Research Fellow in the Food Consumption and Nutrition Division of the International Food Policy Research Institute (IFPRI). In addition to his work on political economy, he is currently co-leading a research programme on urban food and nutrition security in developing countries. He can be contacted at: IFPRI, 1200 17th St.,
NW, Washington, DC 20036, USA. Tel. (202) 862-4598; Fax (202) 467-4439; E-mail:
email@example.com Yassir Islam was formerly a Research Analyst in the Outreach Division of IFPRI where he worked on rural infrastructure, food security, and impact assessment. He continues to be interested in research and policy analysis, and how to increase its relevance to developing countries. He can be reached by E-mail at firstname.lastname@example.org or at 1920 S Street, NW, No.
302, Washington DC 20009, USA. He welcomes any comments and enquiries related to this work.
GATEKEEPER SERIES NO. SA74
EXECUTIVE SUMMARYIn a time of tight government budgets, donors and clients want to know whether the research they fund makes any difference to public policy choices and, ultimately, to people’s lives. Yet the methodologies for evaluating the impact of social science research on policy choices and policy outcomes are not well-developed. The complexity of the policy process and the general nature of much policy research makes it difficult to attribute policy decisions or policy outcomes to specific research findings. This paper contributes to the development of these methodologies by first summarising what we know about the policy process and the use of information in policymaking, and then relying on that review to suggest ways to assess the impact of research on policymaking.
Most importantly, the paper suggests that policy research can have significant impact on policymaking, just not necessarily on discrete choices nor in the linear sequence that researchers and donors would like to see. Research is only one of many competing sources of information, which, as suggested by a descriptive review of the policy process, is itself only one of many factors that affect the final policy decision. In this milieu, policymakers frequently use research less to dictate specific solutions than to help them think about issues and define the scope of problems and possible responses. Thus, research information provides a diffuse ‘enlightenment’ function, providing an understanding and interpretation of the data and the situation that is critical to the policy decision.
Faced with the near impossibility of tracing a precise pathway from specific research effort to policy choice and outcome, this paper recommends that evaluations of the impact of
social science research institutes should:
• Evaluate the quality and timeliness of research output, the contribution of research to the policy debate, including the effectiveness of a proactive communications strategy, and the potential impact of the research (rather than its actual impact) on policy outcomes.
• Evaluate contributions of the research to ‘enlightenment’, and not only to policy change.
• Take into account the diverse ways in which research findings enter and influence the policy process.
• Perform evaluations over time to capture the different ways and different points in time at which research influences policy actors and policy processes.
In sum, the paper recommends a mixed-method approach to evaluation that looks at output, processes, and potential outcomes, rather than focusing on actual policy outcomes. This would better reflect how researchers produce their findings and policymakers actually use research and would help to identify how the organisation could improve its effectiveness.
GATEKEEPER SERIES NO. SA74
POLICY RESEARCH AND THE POLICY
PROCESS: DO THE TWAIN EVER MEET?James L. Garrett & Yassir Islam Policy research organisations fulfil their mandates primarily through the provision of information to policymakers. Yet, especially in a time of tight government budgets, donors and clients want to know whether the research they fund makes any difference to public policy choices. They also want to know if the research has an impact not only on a government’s decisions but, ultimately, on people’s lives. As donors look at the allocation for foreign aid, for instance, they ask why they should commission and fund research on and in developing countries if it does not have a demonstrable impact on poverty, food insecurity, malnutrition, or environmental sustainability.
An impact assessment exercise would seem the most logical course to help a research organisation gauge its effectiveness and to identify strategies to increase its impact on policy decisions. The assessment could also respond to concerns of clients and donors. But the methodologies for the evaluation of the impact of social science research on policy choices and policy outcomes are not yet well developed.
This paper contributes to the development of these methodologies by first summarising what we know about the policy process and the use of information in policymaking, and then suggesting what these insights imply for ways to assess the impact of research on policy choices.
What is ‘impact’?
Even before asking how to measure impact, we must determine what we mean by ‘impact’.
For example, do we evaluate the format and quality of information that the research organisation produces (output), or how the organisation provides information to policymakers and whether that actually influences policy choices (process), or whether the policies pursued by a government to which the organisation provides in-form-ation actually affects final outcomes, by reducing poverty for instance (outcomes)?
The very phrase ‘impact assessment’ can be problematic, as it conveys an impression that we intend to evaluate the effect of a clearly identifiable action on a clearly defined target.
This leads many to think that research only has impact if there is a clear, direct link between research and policy outcomes. The ‘problem-solving’ model of policymaking and research use that underlies this perspective implies that if a report is not read and the policy not immediately changed, the research was not useful and had no impact. It assumes that each instance of information use is a discrete event for which there is a well-defined problem and solution and that the government acts as if it were one person, a “unitary actor” (Feldman, 1989).
GATEKEEPER SERIES NO. SA74 This paper argues that an approach to assessing the impact of policy research that relies on the problem-solving model is inappropriate because it ignores what we know about the policy process and how research information is produced. Although certainly the most striking examples of impact result when a policymaker takes up research recommendations wholeheartedly, this does not happen frequently; the impact of research on policy choice is more likely not to be direct or linear. Rather, a research organisation’s greatest influence probably occurs by contributing high-quality policy-relevant information to a pool of knowledge that policymakers can access when they need it and use as they see fit. Focusing only on cases of direct impact because these are the easiest cases to identify and because they fit with preconceived notions of ‘impact’ would fail to account fully for the many ways in which research influences policymakers.
The making of public policy A research organisation’s influence on policymakers comes about primarily through the provision of information. It usually has control over the process of collecting data and transforming it into research information, but turning this information into policy is the prerogative of government decisionmakers. Indeed, the policy process is complex and subject to political, economic, and social pressures over which the research organisation has little control. Some people even feel that, although it may be possible to identify the indirect impact of research on the policy, the idea that policy research can have direct impact should be dropped, not least because the responsibility of government officials for their own actions should be recognised.
Can, then, a donor or client realistically hold a research organisation responsible for government actions in an evaluation of that organisation’s impact? Perhaps not directly, and examining the process more closely illustrates the difficulty of trying to do so. Still, at the same time such an examination offers some insights into what elements can form the basis of an evaluation of the impact of a social science research organisation.
The fundamental observation about policymaking is that decisions are not made by a single person. Even in a dictatorship, decisions are not made and implemented in isolation.
Although politicians and bureaucrats as a group make up the state, functionally they need not, and often do not, act as a unit, a single rational actor. A state does not make decisions;
people do. Policies and programmes are the cumulative result of conflict and co-operation among many government actors, principally politicians and bureaucrats, as well as members of external interest groups. Of course, these inter-actions can result in policy stasis as well as policy change.
In this process, policymakers do not simply react to pressure from interest groups inside or outside the government. Through judicious use of available resources, including those provided by outside interest groups, policymakers can advance their own agendas and shift discussion to a setting where they have greater control over resources that influence the GATEKEEPER SERIES NO. SA74 decision making process, widening the range of feasible policy options – their ‘policy space’ – from that which is first apparent (Grindle and Thomas, 1991).
This process does not have to follow precise analytical stages or otherwise be linear.
Policymakers at different levels can be making decisions about the same issue at the same time, and also be interacting with others inside and outside the policy space. The process is dynamic, but it is not haphazard. Formal and informal rules and procedures determine which individuals participate in the policy process. Formal arrange-ments may be embodied in a document like a constitution, and are often supplemented (and even supplanted) by informal ones, including well-known but unwritten ‘rules of the game’ (Allison, 1971).
Even the characteristics of policy reform can change over time, as the issue moves through the hands of different policymakers and as the policy environment changes. For instance, in many countries the discussion of ‘safety nets’ as a means to alleviate the negative effects of structural adjustment programmes has become a basis for a broader discussion of the role of social assistance programmes in alleviating poverty.
Different policymakers see different faces of an issue. These ‘faces’ determine how a policymaker analyses an issue and shapes a response. Different faces of the same issue can enhance an issue’s appeal to a policymaker or make it appear as a threat. The face policymakers see depends on their ideological inclination, professional expertise, even on their personal affinity or antipathy for other players, as well as how the issue affects the agency’s clientele or its influence within the bureaucracy (Allison, 1971; Grindle and Thomas, 1991). For example, in designing a national nutrition plan, the physician who heads the Ministry of Health may see it as a health intervention and focus on improving health facilities. The agricultural economist in charge of the Ministry of Food may think only about increasing agricultural production.
Interest groups outside the state can also exert significant influence on policy choices. The impact that a group has depends on how powerful the group is. As with government actors, the power of an external interest group is determined by its control over resources and access to information, its skill in using these advantages, including generation of media coverage, and other players’ perceptions of these characteristics, including perceptions of their credibility (Allison, 1971).
Powerful interest groups need not be rich or large. A decision maker’s interest in advancing a group’s cause may be sufficient to give the group access to the policy process. The decision by policymakers to take up, champion, or oppose an issue depends to a large extent on whether they believe furthering a particular group’s concerns will advance their own interests (See Box 1).
GATEKEEPER SERIES NO. SA74 Box 1. Agricultural policy in Zimbabwe The observation that policymakers have their own agendas cautions against assuming “the powerful get what they want, so those who get what they want must be powerful.” Herbst (1988) describes how the government of independent Zimbabwe acted on behalf of small producers, who were poor and poorly organised and who, by cruder measures of power, should have had little influence over government decisions. At times, the government even purposely discriminated against supposedly politically powerful white growers.