Shoot the Horse and Build a Better Barn Door: Exploring the Potential for a Superforecasting Methodology to Strengthen the DHS Leadership Selection Process

Ronald Dorman


…Not All Those Who Wander Are Lost[1]

One of the most enduring criticisms leveled at the Department of Homeland Security (DHS) over the years has been the performance of its leaders. DHS has responded to those criticisms by initiating or expanding programs intended to improve leadership at all levels by focusing on the development of existing leaders. Although these measures have resulted in marginal improvements in some sub-components, DHS remains entrenched in the bottom tier of every government scale of organizational health, a position it has maintained since 2010. While the issues associated with existing leaders have received ample attention, the selection process that precipitated them has not. This gap represents an opportunity to explore a nascent space and suggest new solutions to target the problem at its source.

The DHS leadership selection process relies on candidate storytelling, and the ability of experts to intuit the best candidate for a leadership position when they see one. The data to support the belief that experts can intuitively identify leaders is, quite simply, not good. The DHS selection process also fails to measurably correlate predictions with outcomes. In other words, after a promotion decision is made, no one can answer a simple yet vital question: Was that a good decision? Without an objectively accurate answer, officials cannot assess the performance of the selection process to identify errors, make corrections, and produce better subsequent outcomes. Systems that do not objectively compare expected outcomes against actual outcomes are open data loops. They do not produce good outcomes.

In 2011, in response to a string of catastrophic intelligence failures, the U.S. intelligence community (IC) launched a multi-year public forecasting tournament designed to discover better methods for predicting complex future geopolitical events. The winning team was led by social researcher Philip Tetlock, and the revolutionary method Tetlock created to beat the competition was superforecasting.

So, DHS has a leadership problem and the IC has an analysis problem. At its core, leadership selection is really just a prediction, or a forecast. It is an educated but nonetheless imperfect best guess about how a candidate observed today will perform tomorrow. That shared characteristic, as well as the causal overlaps for the fundamental flaws observed in both domains, suggest a novel possibility. Could DHS use a superforecasting methodology to improve its leadership selection predictions in the same manner that Tetlock used it to improve geopolitical predictions?

The answer to that question resides within the following five chapters. The first chapter contains the problem statement, scope, methodology and assumptions, as well as a comprehensive literature review covering cognitive bias, leadership, and superforecasting. In Chapter II, the literatures are used to disassemble Tetlock’s superforecasting process into its constituent parts, and place each piece under a microscope to blueprint foundations, form, and function in detail. What is it, where did it originate, how does it work, why does it work, and what is its role in relation to the other parts in the cycle of operation? The parts are then virtually re-assembled and animated to help the reader visualize the entire process, understand where the data originates, how it is processed and measured, and how feedback is used to refine performance. Chapter III uses the knowledge gained to envision a new Superforecasting process, purpose-built for promotion, and installed in the most advantageous environment imaginable to test the writer’s belief that it might work, even if only under optimal conditions. Chapter IV assumes a Red Team role and searches for logic gaps or other vulnerabilities in the hypothesis to invalidate the concept. Chapter V recounts the journey to an unexpected conclusion, the insights gained along the way, and opportunities for the next expedition. Ultimately, this thesis identifies systemic flaws in the DHS leadership selection process, and offers several concrete recommendations that can be implemented to produce better organizational outcomes. It also lays a crumb trail for others to follow, and potentially build upon.

Within the confines of a conventional thesis, the purpose of an Executive Summary is to provide the reader with a standalone version that condenses the larger document into a convenient travel-sized package. This is not a conventional thesis. This is a thought experiment. It is designed to allow the reader to experience the progression of the question through the writer’s eyes, stumbles and blunders included. A more complete synopsis would spoil the expedition.

To begin, turn the page.

[1] J.R.R. Tolkien, The Fellowship of the Ring (New York: Ballantine Books, 1977), 231.

No Comments

Post a Comment