Society is facing multiple sustainability challenges including climate change and biodiversity loss and urgently seeks solutions to these complex problems. Science has played a prominent role in describing these challenges as well as identifying effective policy responses. Despite the pivotal role of science in informing policy, we also see that science is failing to influence public policy, as manifested by the ever-continuing degradation of natural resources and accelerating climate change.
In order to understand and respond to urgent and ever-worsening sustainability challenges, we need scientific evidence and we need that it is shared effectively. This was the starting point of my PhD thesis entitled “How To Generate And Share Evidence Effectively? Comparing Evidence Syntheses And Exploring Scientists’ Challenges In Environmental Science-Policy Interfaces” My research was also motivated by the need to understand how I and other researchers, who engage in science-policy interactions, can do our work more effectively. Furthermore, I think questions about the origin of evidence and the politics related to evidence “picking” are too rarely addressed within science and policy despite that they are critical questions. We know very well that knowledge and evidence is generated in various ways and by very different actors. We also know that policy-makers cherry-pick evidence and also dismiss and ignore conflicting evidence as irrelevant or uncertain. How can we thus generate evidence that is credible and policy relevant and how to deal with the tensions that emerge in science-policy interfaces?
Generating evidence through evidence syntheses
There are multiple types and forms of evidence. In my work, I focus on scientific knowledge and evidence synthesis. The first part of the thesis addresses effective evidence generation and compares the strengths and limitations of two popular evidence synthesis methods. These methods are systematic review (Article I and Article II) and realist synthesis (Article III). The syntheses topic related to a contested and long-standing debate within natural resource governance, namely: what are the environmental impacts of different property rights regimes in forests, fisheries and rangelands and how are the impacts influenced by contextual factors. Particularly in areas of the Global South, property regime related policy interventions (such as policy reforms that strengthen rights of local communities to local forests) have been identified as an important policy tool to address biodiversity loss, unsustainable resource management and poverty. However, the effectiveness of these interventions is subject to many open questions, including how does context influence environmental outcomes of the property regime interventions.
The evidence synthesis I examined (systematic review and realist synthesis) represented different research traditions and ways to generate evidence. The systematic review method, grounded in positivism, emphasized the rigor of evidence which is generated by following pre-defined data procedures, including data coding standards. The realist synthesis method, grounded in realism, emphasized the complexity and messiness of social reality and policy processes. Thus, what is evidence is influenced by reviewer choices.
Going beyond the challenges in evidence sharing
The second part of the thesis addressed challenges of effective evidence sharing. Many researchers are increasingly interested and expected to engage in science-policy interactions. Currently, the scholarship is focused on describing the overall challenges of evidence sharing, such as facilitators and barriers of effective science- policy interactions. However, the tensions, risks and trade-offs related to these interactions have remained understudied and this was the focus of my research. By taking a case study approach and using semi-structured interviews, I examined the perceptions of experienced researchers working in diverse forest related science-policy interfaces (Article IV). The science-policy interfaces are organizations and processes that connect scientists with policy-makers. Thus, they are a key for effective evidence sharing.
My analysis was guided by the framework of Cash et al. (2003) that highlights the importance of three characteristics for effective evidence generation and evidence sharing: These are credibility, relevance and legitimacy. Credibility is related to scientific adequacy and quality; relevance is about usefulness and practical applicability and legitimacy is related to fairness and inclusion of different values. The framework also highlights that actors have different perceptions, standards and priorities for these attributes. It may be possible to enhance all three characteristics, credibility, relevance and legitimacy simultaneously, however, there are also trade-offs between these characteristics.
Dr. Maria Ojanen is a recently graduated doctoral researcher from the University of Helsinki.
What about the results?
As noted earlier, the aim of the PhD was to assess the strengths and limitations of these evidence synthesis. In terms of credibility of the evidence, both evidence synthesis methods delivered comprehensive, transparent and rigorous evidence. However, their ability to make sense of the underlying evidence-base, that is the property regime cases identified from the literature, varied. Overall, the evidence-base was very heterogenous. This kind of data was very hard to fit into the pre-defined data categories of the systematic review and led to an identification of multiple “unknowns” (i.e. judgement can not be made based on the data given in the article). Also important linkages between the data categories were missed out. The systematic review resulted into rigorous but a very narrow evidence-base which offered very limited policy recommendations.
The realist synthesis, on the other hand, was better able to use the heterogenous data because the underlying research philosophy is more constructivist. Realist synthesis applied broad data categories and did not seek to quantify largely qualitative material. Furthermore, the realist synthesis acknowledges that in addition to contextual factors, mechanisms have a role in explaining the outcome. The context-mechanism-outcome framework worked well and it was useful in explaining policy outcomes. However, the relevance of these mechanisms for policy makers is limited because they are very abstract (e.g. one of the mechanisms was “perceptions of legitimacy”) and context-specific. In both cases, the ability of the syntheses to make policy recommendations was also affected by the very broad research question. The breadth of the question also made it difficult to communicate and engage deeply with national level policy-makers.
In terms of legitimacy of the evidence, I assessed the transparency and completeness of the knowledge coverage of the reviews. Transparency meant that both reviews reported conflicting interests and their funding sources. For a complete knowledge coverage, both synthesis methods looked for data sources beyond academic literature, such as NGO reports. Admittedly, legitimacy has many other aspects as well, such as how different values, voices or knowledges are heard. The evidence-base identified by the evidence syntheses were largely based on natural scientists’ measurements of the environment and local perceptions on environmental quality were typically missing. Given the increased attention to the value of local knowledge in contributing to scientific knowledge, there is much room for improvement when it comes to understanding environmental outcomes.
In terms of evidence sharing, the interviewed researchers discussed particularly tensions related to credibility and relevance. In terms of credibility, stakeholders could contest the reliability of the evidence and research results. In some cases, a powerful stakeholder even demanded changes. A refusal to change or omit certain parts could cause conflict and damage relationships with the stakeholder. In terms of relevance, the interviewees noted there was a strong push to provide useful evidence (answers) to specific policy problems. As a result, science-policy interfaces and evidence that related to policy topics that were dismissed by the main policy discourse (such as indigenous land rights) struggled to become and stay policy relevant. In terms of legitimacy, in some science-policy interfaces the researchers were perceived to be “outsiders” and their right to participate was a source of conflict.
In order to deal with the tensions, the interviewed researchers identified several responses, including strategic planning, knowledge co-production and collaboration with a wider set of stakeholders in society. Also, sensitivity and self-reflection were highlighted as response strategies. However, not all of the tensions could be solved and thus accepting conflict was an additional response. Article IV notes that a major risk for scientists is that they need to compromise scientific independence and credibility in order to ensure and maintain good working relationships with policy-makers. This was the case particularly in science-policy interfaces that were researcher-led or otherwise less official. I also note that the long- term consequences of this kind of compromise are not well-known.
Theoretical and empirical contribution of the PhD
As a theoretical contribution, my thesis emphasizes the need to pay critical attention to evidence generation methods and to the tensions that emerge in science-policy interfaces. Power and politics should be an integral part of any framework discussing and assessing science-policy interactions. It is crucial to more transparently to show the choices related to what and whose evidence matters. Furthermore, this work advances our understanding of the strengths and limitations of evidence synthesis methods. Particularly the discussion section of the PhD provides practical lessons and advice for future reviews and reviewers regarding methods and procedures. My work also highlights that researchers who work with contested policy questions and whose results challenge existing policy structures, will face tensions when sharing evidence. The scientific community needs to become more open and vocal about these tensions and conflicts of power. As a society, we also need to recognize that scientific freedom can not be taken for granted in science-policy work.
In addition to analyzing and writing about power relations and politics, power dynamics can also be addressed directly in science-policy interface structures and processes. Science-policy interfaces can thus enhance inclusive practices and promote discussion and debate. Naturally, this kind of collaborative, deliberative and open approach will not appeal to all actors and conflicts are likely to emerge. Thus, science-policy interface needs to be able to accommodate contestation and conflict. The civil society should become a more important ally, when we seek ways to strengthen the openness of science-policy interfaces. Opening up science-policy interfaces in different ways also ultimately supports and contributes to more open and participatory public governance.
---
The articles composing the thesis can be found here:
Article I: What are the environmental impacts of property rights regimes in forests, fisheries and rangelands? a systematic review protocol
Article II: What are the environmental impacts of property rights regimes in forests, fisheries and rangelands?
Article III: Fisheries’ Property Regimes and Environmental Outcomes: A Realist Synthesis Review
Article IV: Navigating the science-policy interface: Forest researcher perspectives