By Eric M. Cooper

– Executive Summary –

Both the executive and legislative branches of government have mandated that evidence-based policymaking be implemented in federal governmental decisional processes.^[1] Evidence-based policymaking is the theory that governments should develop “public policies, programs, and practices that are grounded in empirical evidence.”^[2] In alignment with this direction, the 2021 National Strategy for Countering Domestic Terrorism cites that it “will ensure that its counter–domestic terrorism prevention efforts are driven by data [and will fund] evidence-based programs”^[3] in support of those efforts.

Although evidence-based policymaking has been in use since the 1990s, it has primarily been used in healthcare and scientific research^[4] and the use of evaluation and evidence in counterterrorism has neither been researched nor implemented extensively.^[5] The bipartisan passage of the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) recognized that there was not enough data or evaluation being used in the government’s decisions and the effectiveness of policies and programs.^[6] Following the Evidence Act, the Office of Management and Budget (OMB) released several memorandums^[7] to provide guidance to Departments in its implementation. The Department of Homeland Security released Directives 069-03 and 069-03-001 stating that “DHS is committed to ensuring a strong culture of evaluation, evidence building and organization learning”^[8] but they did not provide specific guidance on the steps needed to conduct an effective evaluation. Moreover, neither the Department of Homeland Security nor the Department of Justice has identified countering domestic terrorism as a priority in its Learning Agenda despite the direction provided in the White House strategy. Currently, there is no federal guidance that specifies how to use evidence and data to combat domestic terrorism.

Despite the lack of specific guidance, using evidence in support of domestic counterterrorism is challenging due to several factors. There is a lack of literature that demonstrates repeatable and effective programs combating any type of terrorism.^[9] Also, data collection is exacerbated by the relative rarity of domestic terrorism cases within western-based societies and that “domestic [terrorism] incident datasets have been virtually unavailable.”^[10] Additionally, policymaking “is an inherently political process”^[11] with many variables that impact the final decision. Moreover, “political leaders often insist that measurable results be available in a short time-frame, leading to greater focus on visible activities rather building the foundations for sustainable benefits”^[12] – resulting in reduced effectiveness of the evidence-based process. This can lead to ‘policy-based evidence’ when “governments tend to use and promote those research findings which fit policies already decided upon [and] the role of research is more often one of legitimizing policy rather than ‘driving’ it.”^[13] Finally, if evaluation methods are not conducted properly and/or bias is not accounted for, researchers can inadvertently include data that can potentially lead to discriminatory policy decisions.

This thesis first investigates the challenges of using evidence in value-based decisions and the social sciences. The second part of this thesis conducts three case study analyses of domestic and foreign policy initiatives—other than counterterrorism—with respect to the foundations of evidence-based policymaking. This analysis identifies common themes that guide recommendations on how to implement evaluation as one part of an integrated approach in the implementation of the National Strategy for Countering Domestic Terrorism in the United States.

This thesis supplements OMB’s guidance that “depicts and describes four interdependent components of evidence: foundational fact-finding, policy analysis, program evaluation, and performance measurement.”[14] OMB recommended that agencies consider these types of evidence in their activities but did not explicitly define or provide guidance on how to incorporate these four components of evidence-based policy. This thesis reviews different approaches by several organizations committed to incorporating evaluation in the use of decision-making and identifies more explicit steps to support OMB’s guidance. A synopsis of the recommendations from the case studies is organized under OMB’s four components and summarized in Table 1.

Table 1.           Recommendations Developed from the Case Studies

Components of Evidence	Recommendation
Foundational Fact Finding	The federal government should standardize data collection across all agencies, including federal, state, and local partners associated with countering domestic terrorism. Standardization will assist in transferable data and measuring the performance of policy initiatives.
Policy Analysis	Evidence-based policymaking agencies must establish a common methodology before beginning an intervention. The development of a common framework and guidelines may be useful in establishing consistency across agencies.
Agencies should conduct an inventory and classify programs for feasibility and other criteria before implementing evidence-based approaches. Agencies should also review and leverage existing evaluation-based social science interventions from other Departments.
Performance Measurement	Program facilitators should ensure that efforts remain focused on the programmatic goals and not inadvertently shift to other metrics.
All agencies should strive to avoid bias through effective evaluation design and robust data sets.
Program Evaluation	The Federal government can improve evidence-based policymaking efforts by hiring or training a cadre of professional evaluators who can review, compile and communicate the data effectively. The fusion of the data into a useful format is critical in implementing effective prevention and response techniques.
Establish a bipartisan Congressionally mandated office within the Federal government that minimizes the impacts of changing Administrations and politically-driven decisions. The relatively short tenure of executive-level Administrations within the United States creates competing priorities that impact data-driven decisions. To avoid policy-based evidence, a bipartisan or politically independent organization may provide stability when leadership or the environment changes.
The federal government should consider the creation of clearinghouses where existing evidence-based policy programs can be easily located and shared across other agencies with similar goals. This will also provide opportunities for agencies to mimic successful interventions while avoiding poorly developed methodologies.
All of the Dimensions	Develop a common framework and clear guidelines that provide detailed direction to federal agencies on how to properly apply evaluation in social issues. The United States should consider developing a comprehensive guide like the Magenta Guide[15] used in the United Kingdom.
Fund private centers of excellence that can support the development and execution of agency-driven initiatives in an evidence-based approach. Until the Federal government can develop the expertise and talent within its agencies, private institutions should be considered for support, including health-based agencies that may have more experience with evaluation practices.

^[1] Foundations for Evidence-Based Policymaking Act of 2018, Public Law 115-435 (2019): 5529-5557, https://www.govinfo.gov/content/pkg/PLAW-115publ435/pdf/PLAW-115publ435.pdf; Russell T. Vought, “Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Leaming Agendas, Personnel, and Planning Guidance (M-19-23)” (official memorandum, Washington, DC: Office of Management and Budget, 2019), https://www.whitehouse.gov/wp-content/uploads/2019/07/M-19-23.pdf.

^[2] Dvora Yanow, “Evidence-Based Policy,” Britannica, social sciences, accessed January 17, 2022, https://www.britannica.com/topic/evidence-based-policy.

^[3] White House, National Strategy for Countering Domestic Terrorism (Washington, DC: White House, 2021), 20, https://www.whitehouse.gov/wp-content/uploads/2021/06/National-Strategy-for-Countering-Domestic-Terrorism.pdf.

^[4] Dvora Yanow, “Evidence-Based Policy,” Britannica, social sciences, accessed January 17, 2022, https://www.britannica.com/topic/evidence-based-policy.

^[5] Cynthia Lum and Leslie W. Kennedy, Evidence-Based Counterterrorism Policy, Springer Series on Evidence-Based Crime Policy (New York: Springer, 2012).

^[6] “Foundations for Evidence-Based Policymaking Act of 2018,” 5 U.S.C. § 101 (2019), https://www.congress.gov/bill/115th-congress/house-bill/4174.

^[7] Russell T. Vought, Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Learning Agendas, Personnel, and Planning Guidance (M-19-23) (Washington, DC: Office of Management and Budget, 2019), https://www.whitehouse.gov/wp-content/uploads/2019/07/M-19-23.pdf.

^[8] Randolph D. Alles, Program, Policy, and Organizational Evaluations, DHS Directive 069-03 (Washington, DC: Department of Homeland Security, 2021), 1.

^[9] Cynthia Lum and Leslie W. Kennedy, Evidence-Based Counterterrorism Policy, Springer Series on Evidence-Based Crime Policy (New York: Springer, 2012), 4.

^[10] Berkebile, 1.

^[11] Sophie Sutcliffe et al., A Toolkit for Progressive Policymakers in Developing Countries, 2006, 3.

^[12] Brian W. Head, “Reconsidering Evidence-Based Policy: Key Issues and Challenges,” Policy and Society 29, no. 2 (May 1, 2010): 84, https://doi.org/10.1016/j.polsoc.2010.03.001.

^[13] Mike Maguire, “The Crime Reduction Programme in England and Wales: Reflections on the Vision and the Reality,” Criminal Justice 4, no. 3 (August 1, 2004): 227, https://doi.org/10.1177/1466802504048463.

[14] Vought, M-19-23, 13.

[15] Great Britain and Treasury, Prudential Standards in the Financial Services Bill: Policy Statement., 2020, https://www.webarchive.org.uk/access/resolve/20200311151411/https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/871680/Final_Budget_policy_statement_-_Basel_and_prudential_measures.pdf.