… the process of collecting, analysing and interpreting data, information and knowledge to increase the understanding of a topic.
Petru, 2018.
Evidence-based policymaking brings the field of research closer to the field of public policy. This is often done internally, within the governmental bodies that seek evidence to inform their policies. Policymakers may also commission the research that they need. Whether internal or external, research and analysis are essential to inform policymaking.
In order to effectively use research and analysis, it is necessary to understand their basic components. The way research is conducted and designed, and the questions it seeks to answer, affect the way it can be used in different policy cycle stages. The two basic types of research (academic and applied) have different but complementary uses in the policymaking cycle, as laid out in Table 6.
ACADEMIC RESEARCH | APPLIED RESEARCH | |
---|---|---|
CHARACTERISTICS |
Analytical perspective, regardless of whether research is theoretical or empirical (i.e. based on verifiable observation and experience). Aims to improve knowledge on a specific field in a generalized manner |
Practical perspective that aims to find solutions or improve a condition |
OUTPUTS |
Useful for the issue identification and definition stage, which requires deep understanding of a specific issue |
Useful for stages that focus on specific plans and activities, and that are solution oriented:
|
Any type of research seeks to answer a question. Identifying and understanding the research question helps to determine the relevance of the research and the policy cycle stage at which it is most useful. The research question sets a specific goal to be achieved through the research and determines the research design and the data collection and analysis methods that will be used (Bryman, 2008). Different research designs can be more or less relevant depending on the information that is necessary to support the policymaking process (see Assessing the relevance and quality of research and analysis).
Types of inquiry within qualitative, quantitative, and mixed methods approaches that provide specific direction for procedures in a research study. … A research design reflects the way research or analysis will be implemented, providing a framework for collecting and analysing data, information and knowledge.
Creswell, 2014.
To provide some examples, both academic and applied research usually start with desk research to refine the research question and review existing literature on the topic. Desk research is a type of secondary research that reviews and synthesizes relevant available research and analysis. In contrast, primary research entails the generation of data by directly doing interviews, taking polls, applying questionnaires, and the like. Desk research is particularly useful during the first stages of the policy cycle: issue identification; data, research and analysis; and policy formulation. As with data (see Traditional migration data sources), quantitative research focuses on quantifiable evidence to identify trends and patterns while qualitative research focuses on understanding why a phenomenon might be occurring and the different perceptions and factors that influence it.
SECONDARY RESEARCH
METHODS | COLLECTION METHODS | ANALYSIS METHODS | UTILITY |
---|---|---|---|
DESK RESEARCH |
|
|
Useful in policy cycle stages:
|
PRIMARY RESEARCH
METHODS | COLLECTION METHODS | ANALYSIS METHODS | UTILITY |
---|---|---|---|
QUALITATIVE |
|
|
Useful in policy cycle stages:
|
QUANTITATIVE |
|
|
Useful in policy cycle stages:
|
EITHER PRIMARY OR SECONDARY RESEARCH
METHODS | COLLECTION METHODS | ANALYSIS METHODS | UTILITY |
---|---|---|---|
MIXED METHODS |
|
|
|
The type, quality and quantity of research and analysis, including on migration, have increased dramatically over the years and so has access to such material. This results from such factors as the increase in the number and type of actors involved in the production of research and analysis, and the increasing variety of sources, websites and databases making research and analysis available (IOM, 2017). Some research and analysis are not freely accessible: they may be published in journals that require a subscription or, if produced by governments, are often kept internal.
Different types of research present advantages and disadvantages for the policymaker. Before delving into how to assess the quality of research and analysis and how to ensure its credibility and relevance (see Assessing the credibility of research and analysis), this topic outlines different types of research, their advantages and limitations, their producers and the sources where they can be found.
PRODUCERS | TYPE | WHERE TO FIND | ADVANTAGES AND DISADVANTAGES |
---|---|---|---|
ACADEMICS (professors, lecturers, researchers, university students) Sometimes authored by practitioners outside of academia, such as independent applied researchers or experts from international or non-governmental organizations |
|
Journals related to migration (examples):
To be found through search engines such as Google Scholar, academic databases or librarie |
+ peer reviewed + abstract/executive summary + policy recommendations
- extensive length (especially monographs, books, textbooks, handbooks) - academic jargon - long publishing process - usually behind paywalls with costly access |
One particularly useful type of academic publication for policymakers is the systematic literature review. These establish what is known from research, but also what remains to be researched. Their focus is “on ensuring a comprehensive review of all existing literature on the issue”; “they also appraise the evidence” (Bryman, 2008). They can therefore also be used very effectively when one seeks to confirm the credibility and relevance of a source.
This systematic review looks into studies on how migration outcomes are associated with and affected by environmental factors. The authors not only map studies that have been conducted on this matter, they also identify and categorize important elements of the studies such as the environmental and non-environmental factors that the studies identify as crucial, the context, but also the research methods used. The authors go a step further, in identifying strengths and challenges, and highlight gaps and opportunities in future research. Policymakers can take great advantage of such systematic reviews, as they constitute a great source of further research material and most importantly they acknowledge and reflect on the use of the reviewed material for policymaking and future needs.
Source: Obokata, Veronis and McLeman, 2014.
The majority of academic research material starts with an abstract, which can already help policymakers identify the relevance of a particular article for the issue they want to tackle. Some journals include policy recommendations and executive summaries, in an attempt to become more policy oriented.
The process of peer review, which is obligatory for publication in well-established and acknowledged scientific journals or publishing houses, ensures that the research meets scientific standards, and is thus credible. Given the pressure for academics to “publish or perish”, not all research is of good quality, which is why peer review is so important. Peer review is, however, a time-consuming process that delays publications often to almost a year after submission. This is often incompatible with the fast-paced process of policymaking.
Academic research is not specifically targeted at policymakers and the texts are not necessarily solution oriented or meant to inform policy. The long format of academic articles and especially monographs and books and the extensive use of academic language is often considered a barrier to using these sources in policymaking. That said, a number of academics do work towards informing policy and notable efforts are being made both to invite academics to make contributions and to increase their capacity to produce research that is policy relevant, concise and written in non-technical language.
The project South-South Migration, Inequality and Development Hub engages a consortium of 12 universities and research centres to explore the ways in which south-south migration can contribute to development and reduce inequalities. One of the goals of the project is to build the capacity of researchers to inform policy in countries involved in south-south migration. To this end, the project provides extended opportunities for knowledge exchange and for training to researchers.
Accessing academic research material can be done through academic databases or directly through journals and libraries. This, however, requires knowledge of search tools and subscriptions, which can be very costly. Open source directories are aiming at improving access and are increasingly widespread. Other solutions for accessing subscription material is through local, national or university libraries that may have access to several online databases and journals, often in addition to print copies.
- Directory of Open Access Repositories (DOAR)
- Directory of Open Access Journals (DOAJ)
- Registry of Open Access Repositories (ROAR)
- CORE, an aggregator of open access research papers
- BASE, a search engine especially for academic web resources
PRODUCERS | WHERE TO FIND | ADVANTAGES & DISADVANTAGES |
---|---|---|
|
Websites of organizations |
+ short length + policy recommendations included + generally accessible + less or no academic jargon + NGOs and international organizations may be more in touch with the realities faced by migrants
- quality and credibility varies - NGOs may be more focused on raising awareness and on advocacy - Government policy papers and briefs are often inaccessible outside of government |
Policy papers and briefs mainly target policymakers, focusing on policy analysis and policy recommendations. They may be produced in-house, be commissioned by governments or intergovernmental organizations, or may be funded by enterprises or NGOs, among others. The increasing focus of many organizations on migration issues, producing policy papers and briefs, reflects the increasing political, social and economic significance of migration.
The quality of policy papers and policy briefs varies significantly, depending on who publishes them and under what circumstances. In contrast with academic material, policy papers do not usually go through the process of peer review. Moreover, the funding, political affiliation or ideologies of the publisher may affect the quality of the research and induce bias. However, an increasing number of think tanks have been producing high-quality contributions.
- Afghanistan Research and Evaluation Unit
- Brookings Institution
- Bruegel
- Center for China and Globalization
- Center for Comparative Immigration Studies (University of California, San Diego)
- Center for Immigration Studies
- Centro de Divulgacion del Conocimiento Económico para la Libertad
- Danish Institute for International Studies
- Friedrich Ebert Foundation
- Inter-American Dialogue
- Institute for Immigration Research (George Mason University)
- Institute for the Study of International Migration (Georgetown University)
- Institute of Southeast Asian Studies (Yusof Ishak Institute)
- IZA Institute of Labour Economics (University of Bonn)
- Migration Policy Institute (MPI)
- Pew Research Center
- Overseas Development Institute (ODI)
PRODUCERS | WHERE TO FIND | ADVANTAGES & DISADVANTAGES |
---|---|---|
|
Websites of organizations |
+ analysis of best practices and impact + include policy recommendations and lessons learned + focus on current topics + generally accessible
- studies and handbooks are quite long and analytic - studies and handbooks may not be easily accessible - as commissioned work, evaluation reports may not be publicly accessible |
The main goal of studies and reports is to provide an analysis of a specific topic, often in a specific region. While academic publications also include handbooks and studies, the ones mentioned here are more policy oriented. Work in the field by NGOs and intergovernmental organizations places them in a privileged position regarding the gathering of data, which often evolves into reports and studies.
Virtually all United Nations agencies publish annual or biennial reports as their flagship publication, while a wide range of them publish reports related to specific aspects of migration and migrants in light of their respective mandate (IOM, 2019). Their quality may vary depending on whether they have been subject to peer review. Agencies also produce technical reports stemming from the mandate of working groups or special procedures of the United Nations Human Rights Council, such as the Special Rapporteur on the human rights of migrants. Their aim is to inform governments, parliaments, committees and other interested parties about the implementation of action plans and activities, about current trends and important updates or about the outcomes of conferences and efforts to create common policies.
- UNDESA: International Migration Reports (biennial)
- IDMC: Global Report on Internal Displacement (annual)
- IOM: World Migration Reports (biennial)
- UNHCR: Global Trends: Forced Displacement reports (annual)
- UNODC: Global Report on Trafficking in Persons (biennial)
Studies and reports can also be the outcome of work conducted in-house or commissioned by government departments. Studies that are commissioned research are sometimes not available to the general public. Their length also varies significantly depending on the scope of the study and the goal of the producer. Quality and credibility varies with the producers and their quality assurance processes.
A specific type of report is the evaluation report. Produced or commissioned by governments, regional organizations, NGOs and intergovernmental organizations, evaluation reports aim to:
- assess the success of a policy or project
- provide recommendations
- provide information on the methodology, the project planning and project implementation
This type of research and analysis can be very useful for policymakers as it focuses on the outcomes of projects or policies. In particular, this type of information can be valuable to policymakers seeking best practices. However, again, depending on the regulations of each organization, this type of material may not be publicly available.
- Global Forum for Migration and Development
- Regional Consultative Process: Prague Process
- Regional Consultative Process: Bali Process
- Medecins sans Frontieres
- European Commission
- OXFAM
- International Federation of Red Cross and Red Crescent Societies
- OECD
- KNOMAD
United Nations System
- International Labour Organization (ILO)
- International Organization for Migration (IOM)
- Office of the High Commissioner for Human Rights (OHCHR)
- United Nations Development Programme (UNDP)
- United Nations High Commissioner for Refugees (UNHCR)
- United Nations Children’s Fund (UNICEF)
- UN Women
PRODUCERS | WHERE TO FIND | ADVANTAGES & DISADVANTAGES |
---|---|---|
|
Websites of organizations |
+ analysis of best practices and impact + include policy recommendations and lessons learned + focus on current topics + generally accessible
- studies and handbooks are quite long and analytic - studies and handbooks may not be easily accessible - as commissioned work, evaluation reports may not be publicly accessible |
The main goal of studies and reports is to provide an analysis of a specific topic, often in a specific region. While academic publications also include handbooks and studies, the ones mentioned here are more policy oriented. Work in the field by NGOs and intergovernmental organizations places them in a privileged position regarding the gathering of data, which often evolves into reports and studies.
Virtually all United Nations agencies publish annual or biennial reports as their flagship publication, while a wide range of them publish reports related to specific aspects of migration and migrants in light of their respective mandate (IOM, 2019). Their quality may vary depending on whether they have been subject to peer review. Agencies also produce technical reports stemming from the mandate of working groups or special procedures of the United Nations Human Rights Council, such as the Special Rapporteur on the human rights of migrants. Their aim is to inform governments, parliaments, committees and other interested parties about the implementation of action plans and activities, about current trends and important updates or about the outcomes of conferences and efforts to create common policies.
-
UNDESA: International Migration Reports (biennial)
-
IDMC: Global Report on Internal Displacement (annual)
-
IOM: World Migration Reports (biennial)
-
UNHCR: Global Trends: Forced Displacement reports (annual)
-
UNODC: Global Report on Trafficking in Persons (biennial)
Studies and reports can also be the outcome of work conducted in-house or commissioned by government departments. Studies that are commissioned research are sometimes not available to the general public. Their length also varies significantly depending on the scope of the study and the goal of the producer. Quality and credibility varies with the producers and their quality assurance processes.
A specific type of report is the evaluation report. Produced or commissioned by governments, regional organizations, NGOs and intergovernmental organizations, evaluation reports aim to:
-
assess the success of a policy or project
-
provide recommendations
-
provide information on the methodology, the project planning and project implementation
This type of research and analysis can be very useful for policymakers as it focuses on the outcomes of projects or policies. In particular, this type of information can be valuable to policymakers seeking best practices. However, again, depending on the regulations of each organization, this type of material may not be publicly available.
-
Regional Consultative Process: Prague Process
-
Regional Consultative Process: Bali Process
PRODUCERS | WHERE TO FIND | ADVANTAGES & DISADVANTAGES |
---|---|---|
|
Blogs of the producer or well-established blogs |
+ freely accessible + short + non-technical and appealing language
- credibility and quality varies |
The increasing use of blogs as a way to disseminate information has led to the inclusion of blogs as a type of research and analysis useful for everyone with an interest in migration. Blog posts are usually concise articles posted on host websites and meant to reach new and wider audiences. Advantages include the open access nature of this material and its shorter, non-technical language. However, the credibility of the blog post mainly depends on the independence and expertise of the authors and the host website (IOM, 2019).
This topic focuses on how to obtain research and how to assess whether the research and analysis informing policy thinking are sound.
A common way for governments to obtain the necessary research, analysis and evidence for policymaking is to commission research (IOM, 2017, p. 98). Commissioning research is demanding and requires the administrative capacity, the trained personnel and a culture for evidence-based policymaking. The “classic” situation is for a policy unit to commission a researcher or a research group to better understand a problem and/or potential solutions to it. Research can define the policy issue, trigger a policy response or ideas for a policy response, or discuss the implications of the suggested policy responses, together with potential ways forward and their consequences. Another common situation is for policymakers to commission an evaluation or more in-depth research, which is useful for the implementation of a policy (further details on Evaluation in Stage 7: Policy Monitoring and Evaluation)
For the commission to be successful, the administration should be able to describe the policy issue and the research needed in as much detail as possible. If possible, a preliminary research design should be included in the tender although there should also be space for flexibility and deliberation with the experts. The research design will also affect the cost of the research which should also be considered.
- Preparing the ground: leadership commitment to the process, adequate resources (finance and skills), stakeholder engagement and a steering group to guide the process.
- Deciding on the type and scope of the research: what will the research be about? From there, the resources needed and the data that should be gathered can be determined, as well as the time frame established for the research to be concluded.
- Choosing the research team: considerations for an independent team of researchers include the exact expertise needed, the resources and time needed, and a due diligence selection process.
- Research design: decide on the research questions and methods together with the experts. For a research contract that is useful for the administration, ensure research is (a) described in as much detail as possible, (b) sufficiently specific and (c) targets the issue at stake.
- During the research: facilitate access to stakeholders and data sources, monitor progress and provide input or a reality check to recommendations.
- Using the research results: take action and make changes, draw lessons for the next research, develop an evidence-based policymaking culture.
Adapted from European Monitoring Centre for Drugs and Drug Addiction, 2017.
Capacity is necessary not only for commissioning the research but also for using the research. A common complaint by policymakers, government officials and experts alike is that studies commissioned and paid for by public funds often stay on the shelves and are not properly used. Governments may commission research because it is good for their public image or because of unused funds that need to be spent or just because they are looking for an argument in order to promote a policy that they already have in mind. It is very likely that in such circumstances research will be underused or even not used at all (Boswell, 2009). Building an evidence-based policymaking culture is important and takes time. It can be encouraged by successful cases of research informing policy.
- Boswell, C., Cultures of knowledge use in policymaking. The functions of research in German and UK immigration policy, 2015 (Chapter 2).
- Scholten, P. Research-Policy Relations and Migration Studies, 2018.
- Ruhs, M., K. Tamas and J. Palme (eds.), Bridging the Gaps: Linking Research to Public Debates and Policy-making on Migration and Integration, 2019.
To ensure the evidence used is of high quality, it is necessary to establish how credible the research and analysis is.
There are several techniques to determine whether a piece of research is credible. Reputation, ranking, reviews, and number of citations are some basic indicators. Academic research is deemed credible if it is subject to peer review, published in credible academic journals and cites relevant literature. Peer review cannot guarantee credibility but can give an indication. For other types of research and analysis, credibility is judged on a case-by-case basis. Some simple questions to help determine the credibility of a source can be asked.
Considerations include:
- How were the data about migrants collected? Did the researcher have access to reliable information?
- Was the publication in which the research and analysis appeared peer reviewed? Are the data on which it relies properly cited?
- Who is the author and the publisher? (i.e. is the author a published author and is the publisher well established?)
- Is the website where the research and analysis is made available credible?
- When was the source published or revised?
- Could funding have biased the research?
- Are there potential perceptions of migration and political orientations underpinning the research at stake?
Given that migration is a politicized field, research and policy are becoming increasingly intertwined. While researchers seeking to support the policymaking process should adjust to the fast pace of policymaking, they also need to keep their distance, rigour and objectivity. Governments can encourage policy-oriented research by exploring ways to engage with researchers at various stages of the policy cycle while always respecting the academic freedom of researchers. When research becomes institutionally connected to policymaking, it can become dependent on the policy context and to that extent can become biased (Scholten, 2018), which can negatively affect the credibility of the research. In addition to privileging topics that are related to policymakers’ agendas, the research can be used to support policy choices made previously. An in-house research unit can be an important asset for evidence-based policymaking if arrangements are made to preserve its independence.
- IOM, Migration research and analysis: Growth, reach and recent contributions (World Migration Report 2020), 2019.
- IOM, Migration research and analysis: Growth, reach and recent contributions (World Migration Report 2018), 2017.
- Boswell, C., Cultures of knowledge use in policymaking. The functions of research in German and UK immigration policy, 2015 (Chapter 2).
- Scholten, P., Research-Policy Relations and Migration Studies, 2018.
In addition to establishing the credibility of research and analysis, it is also important to assess their relevance and quality. These depend on two factors: the validity of the research design and the strength of the available evidence.
Validity of research design
The choice of the appropriate research design is critical. It ensures that the data collected are relevant to the question, and can be used to answer it in a convincing way. A good research design should:
- Be able to answer the research question posed;
- Lead to (policy) conclusions and not only to general findings;
- Be realistic and mindful of the available resources and time.
It should be noted that some research designs are better suited than others to specific research objectives. For example, if we know that a problem exists but we do not have information about its causes and characteristics (for instance, in the issue identification stage of the policy cycle), an exploratory study is most useful. The type of questions the research poses could include:
- what is the type/magnitude of the problem?
- Who is affected?
- What do the people affected think about the causes of the problem?
Exploratory and/or descriptive studies can be conducted via in-depth case studies and surveys.
To test the effectiveness of a potential solution (during the policy formulation stage), the research might pose a question such as: Will solution A work under the given circumstances? In this case, what is needed is a targeted pilot study which can rely on experimental or quasi-experimental methods (see examples in Types of evaluation in Chapter 1.4.4 Policy implementation, monitoring and evaluation).
Strength of evidence available
Assessing the strength of the evidence provided in a study requires appraising the quality of the data and of the information and knowledge produced. Some initial questions that can be posed include (Government of Malawi, 2016):
- Uniqueness: is the research original?
- Completeness: is any information missing?
- Coverage: what depth does the research go into?
- Timeliness: is the research up to date?
There are universal principles of research quality that can be applied in the appraisal of research and analysis.
PRINCIPLES OF QUALITY | ASSOCIATED QUESTIONS |
---|---|
CONCEPTUAL FRAMING |
|
TRANSPARENCY |
|
APPROPRIATENESS |
|
CULTURAL & GENDER SENSITIVITY |
|
PARTICIPATORY RESEARCH |
|
VALIDITY |
|
RELIABILITY |
|
CLARITY & ARGUMENTATION |
|
Adapted from Department for International Development (DFID) (2014) and Government of Malawi (2016).
To inform policy thinking, research that is credible, relevant and of good quality must also be applicable to the policy context and replicable.
Applicability refers to the feasibility of the findings of a research in a specific context. The question to ask is “Is it actually possible to implement the recommendations?” Related questions concern:
- Political acceptability of the research findings: for a reform to be politically acceptable there must be support in the political environment and tangible benefits for the government.
- Social acceptability of the research findings: the target population should be in support of the change and no ethical issues should arise.
- Resources: Even when a change is acceptable, the human and financial resources for implementing it need to be considered. This may involve training costs, equipment, office space, new expertise and additional funding.
Transferability or replicability refers to the extent to which the findings or policy propositions of a research can be generalized and used in a different context. A key question to ask is “Will the same policy intervention produce the same results in a different context?” Some issues to reflect upon include:
- whether the need or problem that the policy intervention is targeting is similar and what are the key differences;
- whether the policy intervention can reach all members of the targeted population in a cost-effective way by taking into account the different characteristics of this population (such as gender, ethnicity and other sociodemographic characteristics and numbers).
The co-production of evidence refers to a process where researchers and stakeholders in society and government cooperate to produce policy-relevant knowledge. This process is more effective than simply seeking evidence from experts since it allows for policy learning to take place while the knowledge is being produced. When co-production takes place, researchers have the opportunity to learn about the issues that are relevant and important to policy and stakeholders learn about the new findings as well as the limitations of scientific research. The group of participants is encouraged to see the world through a different lens. Although such a process is time consuming, it can have lasting effects in areas where scientific ambiguity exists and is not possible to simply select one best policy proposition (Dunlop and Radaelli, 2018).
One example of an effort to bring researchers and stakeholders together to produce evidence is the Irregular Migration Research Programme within the Australian Department of Immigration and Border Protection (DIBP). This research programme is designed “to maximize the policy relevance and sustainability of research conducted within a government context, particularly by operating within a framework of partnership and collaboration between a range of migration experts and policymakers.” It is structured to recognize that both policy-relevant and policy “irrelevant” research have value and ensures its sustainability. One part of this programme is a project of the DIBP with a private sector research company on large-scale collation and comparative analysis of media representations of migration and migrants in selected countries. These types of initiatives create forums “for policymakers to test ideas and approaches and an opportunity for migration experts and academics to provide input into policy thinking, gain insights into the challenges and constraints faced by policymakers, and discuss the difficulties involved in developing policy solutions to complex, multifaceted issues.”
McAuliffe and Parrinder, 2015.
By bringing together researchers and policymakers, this conference addressed the relation between research and policy in relation to migration, resulting to recommendations on research needs and topics as well as ways to improve the dialogue between the two fields. The conference was conducted in February 2016 and coordinated by the European Commission.
Recommendations for the improvement of the science-policy dialogue |
|
---|---|
Relevant research needs |
|
Suggested Research Topics |
|
Boswell, 2016.
Boswell, C., Understanding and Tackling the Migration Challenge – the Role of Research (Conference report), 2016.
- Policymakers often commission the research they need, while there is also a large body of research that is readily and freely available to use.
- Understanding the research question that guides a body of research helps determine its relevance and the stage of the policy cycle at which this evidence can be best used.
- Given that migration is a politicized field there is a danger that research produced to target specific policy problems may be biased.
- An evidence-based policymaking culture, along with sufficient administrative capacity, are necessary for research and analysis to be effectively used.
- The co-production of evidence is an effective and rich process where researchers and stakeholders in society and government cooperate for the production of policy-relevant knowledge.