This article investigates how certainty and timing of evidence introduction impact the uptake of evidence by policy-makers in collective deliberations. Little is known about how experts or... Show moreThis article investigates how certainty and timing of evidence introduction impact the uptake of evidence by policy-makers in collective deliberations. Little is known about how experts or researchers should time the introduction of uncertain evidence for policy-makers. With a computational model based on the Hegselmann–Krause opinion dynamics model, we simulate how policy-makers update their opinions in light of new evidence. We illustrate the use of our model with two examples in which timing and certainty matter for policy-making: intelligence analysts scouting potential terrorist activity and food safety inspections of chicken meat. Our computations indicate that evidence should come early to convince policy-makers, regardless of how certain it is. Even if the evidence is quite certain, it will not convince all policy-makers. Next to its substantive contribution, the article also showcases the methodological innovation that agent-based models can bring for a better understanding of the science–policy nexus. The model can be endlessly adapted to generate hypotheses and simulate interactions that cannot be empirically tested. Show less
Krupnik, S.; Szczucka, A.; Wozniak, M.; Pattyn, V.E. 2023
Qualitative comparative analysis is gradually becoming more established in the evaluation field. The purpose of this article is to highlight the potential for evaluation research of engaging in... Show moreQualitative comparative analysis is gradually becoming more established in the evaluation field. The purpose of this article is to highlight the potential for evaluation research of engaging in consecutive rounds of this analysis. This is possible when approaching qualitative comparative analysis as a systematic strategy for configurational theorizing. To substantiate this potential, we present two evaluation studies on Research and Development subsidies for companies in Poland. Compared with the results of the first study, the findings of the subsequent consecutive qualitative comparative analysis studies were much more nuanced and helped in developing a full-fledged configurational program theory. In addition to elaborating on the strengths of a consecutive qualitative comparative analysis approach and the relevance of configurational program theories for evaluators, this article shares the main lessons learned in overcoming challenges common to such designs. Thus, concrete guidance is offered to researchers and evaluators who are willing to take configurational theorizing seriously. Show less
While discourse about the Sustainable Development Goals (SDGs) has primarily focused on 'whether the goals' are achieved, there remains limited understanding of how developed countries organize... Show moreWhile discourse about the Sustainable Development Goals (SDGs) has primarily focused on 'whether the goals' are achieved, there remains limited understanding of how developed countries organize their monitoring and evaluation (M & E) systems, which play a crucial role in tracking progress towards the SDGs. In this contribution, we unpack the M & E frameworks of Belgium, the Netherlands and the United Kingdom. To do so, we have devised an analytical heuristic that combines insights from the literature on policy performance measurement and measurement infrastructures with the more specific literature on SDG governance. Through document analysis and elite interviews conducted in 2021, our findings highlight similarity in underdeveloped M & E frameworks, rather than significant variation across the three case studies. The results do not suggest a linkage between SDG performance and the development of M & E frameworks. Show less
Jungblut, J.; Gouglas, A.; Katz, G.; Bandola-Gill, J.; Brans, M.; Timmermans, A.; ... ; Xhindi, N. 2023
The relevance and impact of political scientists' professional activities outside of universities has become the focus of public attention, partly due to growing expectations that research should... Show moreThe relevance and impact of political scientists' professional activities outside of universities has become the focus of public attention, partly due to growing expectations that research should help address society's grand challenges. One type of such activity is policy advising. However, little attention has been devoted to understanding the extent and type of policy advising activities political scientists engage in. This paper addresses this gap by adopting a classification that distinguishes four ideal types of policy advisors representing differing degrees of engagement. We test this classification by calculating a multi-level latent class model to estimate key factors explaining the prevalence of each type based on an original dataset obtained from a survey of political scientists across 39 European countries. Our results challenge the wisdom that political scientists are sitting in an "ivory tower": the vast majority (80%) of political scientists in Europe are active policy advisers, with most of them providing not only expert guidance but also normative assessments. Show less
Why do some types of evaluation use prevail in certain contexts and not in others? The aim of this article is to advance knowledge about organisational factors of evaluation use, that is,... Show moreWhy do some types of evaluation use prevail in certain contexts and not in others? The aim of this article is to advance knowledge about organisational factors of evaluation use, that is, determinants of evaluation use grounded in organisational theories. We critically review existing frameworks of organisational factors of evaluation use, highlighting key differences between them and pointing out discrepancies with empirical insights. We discuss the merits of two potential areas for future research that can help concretise theoretical stances: considering organisational legitimacy as a potential direct determinant of evaluation use and incorporating a dynamic perspective in organisational frameworks of evaluation use. Show less
Understandings of different policy learning types have matured over recent decades. However, relatively little is known about their nonlinear and interactive nature, particularly within crisis... Show moreUnderstandings of different policy learning types have matured over recent decades. However, relatively little is known about their nonlinear and interactive nature, particularly within crisis contexts. In this article, we explore how two of the most prominent learning types (instrumental and social) shifted and interacted during the COVID‐19 crisis. To do so, we created a policy learning storyboard of the Belgian COVID‐19 policy response over 2 years (from early 2020 to late 2021). Our analysis highlights the relationships between different epochs of instrumental and social learning throughout the crisis and their implications for policy change. Furthermore, while extant policy learning literature often posits that social learning unfolds over relatively long periods (spanning a decade or more), our empirical account shows that within certain conditions, creeping crises can lead to the creation of long‐term crisis policy‐making paradigms and goals. At this level, accelerated social learning can take place and lead to paradigmatic shifts within relatively shorter periods than in noncrisis conditions. Theoretically, our findings enhance our understanding of policy learning types and their relationships with policy change, particularly within crisis contexts. Show less
In this article, we inquire to what extent different manifestations of trust are associated with public support for evidence informed policy making (EIPM). We present the results of a cross... Show moreIn this article, we inquire to what extent different manifestations of trust are associated with public support for evidence informed policy making (EIPM). We present the results of a cross-sectional survey conducted in the peak of the second COVID-19 wave in six Western democracies: Australia, Belgium, Canada, France, Switzerland, and the United States (N = 8749). Our findings show that public trust in scientific experts is generally related to positive attitudes toward evidence-informed policy making, while the opposite is the case for trust in governments and fellow citizens. Interestingly, citizens' assessment of government responses to COVID-19 moderates the relationship between trust and attitudes toward EIPM. Respondents who do rather not trust their governments or their fellow citizens are more in favor of EIPM if they evaluate government responses negatively. These findings suggest that attitudes toward EIPM are not only related to trust, but also strongly depend on perceived government performance. Show less
Political science in the Netherlands has a long tradition and a history of institutionalization. It developed as a broad discipline, together with public administration, and grew into separate... Show morePolitical science in the Netherlands has a long tradition and a history of institutionalization. It developed as a broad discipline, together with public administration, and grew into separate research and education programmes. Does this segmented nature of political science appear in the external activities of those scholars concerned? Or is the overall consensus-style and neo-corporatist (‘polder’) advisory system in which political scientists are placed a more important determinant? What about developments in the policy advisory system itself, pressures on institutions and trends in the environment and their consequences for the supply and demand of scholarly political science advice? These are the central questions we examine in this chapter. We first present the development of the discipline in the country. Then we look at the main features and trends of the advisory system and the niche occupied by political scientists. As we will show on the basis of the survey results, political scientists in the Netherlands gauge their visibility and their social and political impact as relatively high, and a large majority of them engage in advisory activities. Our findings also highlight that the effects of the segmented structure of political science on the type of advisory role are relatively limited. Show less
Hecke, S. van; Walle, B. van de; Bouckaert, G.; Pattyn, V.E. 2022
This chapter focuses on the evaluation of public policy and administration. Institutionalized for decades in many places worldwide, policy evaluation has become a routinized and professionalized... Show moreThis chapter focuses on the evaluation of public policy and administration. Institutionalized for decades in many places worldwide, policy evaluation has become a routinized and professionalized activity. However, assessing the performance of public agencies and the impact of the policies they implement is a highly political endeavor. As part of the executive branch and directly subordinate to governments, public agencies can be scrutinized and evaluated as an extension of political struggles. On the other hand, public agencies also use evaluation to advance their own objectives. This chapter examines the issues attached to the evaluation of public policies and administrative activity in relation to power games within and across the branches of government. It reveals how politicians and public servants can make strategic use of policy evaluations, as well as how this instrument serves not only reflexive and oversight purposes, but also agenda-setting ambitions. The chapter then presents an overview of the controlling, defensive and proactive functions of evaluation in policy struggles. Drawing upon these developments, the chapter underlines just how far from neutral evaluations can be in the politico-administrative game. Show less
Background: While current public policy scholarship can take advantage of a decades-long accumulated knowledge base on the relationship between evidence and policy, it is hard to keep the overview... Show moreBackground: While current public policy scholarship can take advantage of a decades-long accumulated knowledge base on the relationship between evidence and policy, it is hard to keep the overview across different literatures. Over time, the ever more differentiated branches of public policy research have developed their own perspectives, languages, and conceptualisations of ‘evidence’ and ‘policy’, as well as their connections.Aims and objectives: Existing reviews have stressed that studies often do not provide clear definitions of ‘policy’ or ‘evidence’, and have outlined the importance of investigating underlying conceptualisations in the literature. Against this backdrop, this article investigates how present-day public policy scholarship approaches the concepts of ‘evidence’, ‘policy’, and their connections.Methods: We conducted a qualitative systematic review following the PRISMA method. Using a keyword search, we identified relevant articles (n=85) in eleven Q1 and Q2 policy journals included in Web of Science in the period 2015 to 2019.Findings: The synthesis confirms that ‘evidence’ and ‘policy’ are often not clearly defined, yet different trends regarding understandings can be identified. There are two approaches taken on the evidence and policy connection: a ‘use of evidence’ or a ‘use for policy’ perspective.Discussion and conclusions: Research on evidence and policy would benefit from more explicit conceptual discussions. This review may provide a heuristic for explicating conceptual choices when working with the notions of ‘evidence’, ‘policy’, and their connections. It also suggests several avenues that are worth exploring in future research. Show less
This chapter reviews the agenda-setting potential and influence of a particular group of evaluation-related tools governments often resort to in the agenda-setting stage. Particular evaluation... Show moreThis chapter reviews the agenda-setting potential and influence of a particular group of evaluation-related tools governments often resort to in the agenda-setting stage. Particular evaluation tools play an important analytical role in setting up and managing the agendas of governments. The chapter discusses the agenda-setting role of three in-house tools: budget reviews, policy reviews, and sunset reviews. It also discusses their major attributes, their underpinning rational assumptions, and the way they have been institutionalized and routinized in countries' evaluation systems and reflect on the conditions that foster or jeopardize their agenda-setting impact. The chapter describes different empirical examples from various countries. A pragmatic approach to evaluation requirements, varying from case to case, seems to prevail with recent attempts, particularly at the sub-state level, to establish the level of administrative burdens as a decisive criterion for the intensity of evaluation activities. Show less
Starting from the paradox that successive waves of reforms have jeopardized the functional coherence of better regulation as a horizontal policy, this chapter highlights the challenge of meta... Show moreStarting from the paradox that successive waves of reforms have jeopardized the functional coherence of better regulation as a horizontal policy, this chapter highlights the challenge of meta-design of different appraisal procedures and institutions. Indeed, among countries with a long tradition of regulatory management systems, there is an evident proliferation of diverse better regulation objectives pursued by oversight bodies with different mandates. In order to capture the institutional coordination design in specific countries, we identified different theoretical scenarios, ranging from hierarchical coordination to ad hoc, extemporaneous coordination. By way of example, this typology has been applied to analyze how in the UK and the Netherlands better regulation policy tools, procedures and institutions are coordinated. Our analysis revealed some major differences. We show that the UK government lies closer to the hierarchical coordination extreme than the Dutch government. The latter traditionally tends to rely on positive modes of coordination, although informally the neoliberal goal of business deregulation still has a central position in the practices of the Dutch government. Besides these differences, the two governments display similarities in the use of performance metrics to reduce business compliance costs. Yet, whereas business impact targets are established through legislation in the UK, the pivotal role of the Standard Cost Model in the Netherlands is rather informal. Neither of the countries seem to have developed a strategy of coordination in terms of problem solving, which would entail the deployment of particular appraisal tools and institutions depending on a specific issue at stake. Show less
Pattyn, V.E.; Noordink, M.; Modderman, P.; Cohn, R.; Groot, M. de; Braun, C.; Steen, M. van der 2021
Policy evaluations can be set up for multiple purposes including accountability, policy learning and policy planning. The question is, however, how these purposes square with politics itself. To... Show morePolicy evaluations can be set up for multiple purposes including accountability, policy learning and policy planning. The question is, however, how these purposes square with politics itself. To date, there is little knowledge on how government ministers present the rationale of evaluations. This article is the first to provide a diachronic study of discourse about evaluation purposes and encompass a wide range of policy fields. We present an analysis of evaluation announcements in so-called ministerial policy notes issued between 1999 and 2019 by the Flemish government in Belgium. The research fine-tunes available evidence on catalysts for conducting evaluations. The Flemish public sector turns out to be a strong case where New Public Management brought policy evaluation onto the agenda, but this has not resulted in a prominent focus on accountability-oriented evaluations. We further show that policy fields display different evaluation cultures, albeit more in terms of the volume of evaluation demand than in terms of preferences for particular evaluation purposes. Show less
Deze inleidende bijdrage wil de contouren van het thema schetsen vanuit een aantal basisvragen: Wat is effectiviteit van beleid? Wanneer in de beleidscyclus kan effectevaluatie aan bod komen?... Show moreDeze inleidende bijdrage wil de contouren van het thema schetsen vanuit een aantal basisvragen: Wat is effectiviteit van beleid? Wanneer in de beleidscyclus kan effectevaluatie aan bod komen? Kunnen beleidseffecten vlot gemeten worden? En wat zijn de belangrijkste methodologische benaderingen om causale linken tussen beleid en effecten na te gaan? We koppelen dit aan de tendensen en ervaringen in het praktijkveld met het accent op de Vlaamse overheid en belichten ook enkele uitdagingen en lessen voor de toekomst. Show less
