The essential nature of the homeland security enterprise involves making consequential and complex policy decisions under uncertainty. The inputs policy makers use in making these decisions are facts, analyses, and predictions (which can fit a definition of intelligence), all of which are subject to significant uncertainty. Reduction in the uncertainty associated with these inputs may improve the soundness of decision making by policy makers. This thesis seeks to improve analysis by developing a crowd-based analytic methodology to address the problem of intelligence analysis while accounting for and taking advantage of the unique characteristics of the intelligence analysis process and the Intelligence Community culture itself.
The methodology developed in this thesis utilizes prediction markets-based techniques and crowdsourcing techniques that have significantly improved forecast accuracy in other contexts found in the literature. The thesis’s particular contribution focuses on understanding the unique characteristics of the Intelligence Community culture and work processes, and it uses this understanding to inform the design of the proposed crowd-based intelligence forecasting methodology. It can be argued that any analytic methodology hoping to improve the predictive accuracy of the Intelligence Community analysts must both reflect and adapt to the underlying Intelligence Community culture. If it does not, it is likely that any new or modified methodology either may be limited in its adoption, or more likely, be ignored by the intelligence analytic community at large.
The thesis’s proposed methodology applies learning regarding crowdsourcing and prediction markets-based forecasting in a new context, that of intelligence analysis and the Intelligence Community. This research excludes quantitative probabilistic assessments, quantitative and qualitative models, and polls-based techniques from consideration because others have already done extensive work on utilizing these techniques in an intelligence context.
This thesis discusses the characteristics of the proposed crowd, the proposed structure of the forecasting effort, the proposed incentive structure, the proposed task design, and the proposed prediction market design and associated structural parameters underlying the forecasting effort, as well as the key characteristics of the proposed platform used to implement the prediction market. Additionally, the thesis uses all of these critical concepts to design a methodology—a crowd-sourced forecasting tournament—that the U.S. Intelligence Community can use to improve its forecast accuracy. If implemented, the proposed methodology should improve Intelligence Community predictions of real-world events, based on results achieved in other contexts.
The thesis proposes that the utility of the methodology be demonstrated to the analytic branches of intelligence using a pilot program to help get buy-in to the methodology as a whole, as well as to engender participation in the methodology’s prediction market from individuals and teams drawn from the analytic community. If positive, the results of the pilot program may also be used to justify the Intelligence Community spending the financial, analytic time based, administrative time based, and other resources to implement the methodology. Finally, the proposed pilot should allow practitioners to test and tweak various aspects of the methodology from outreach to task design to ensure that the implemented methodology does indeed result in the analytic improvements as it seeks to do.
This thesis is just a starting point; the methodology should be subject to several rounds of peer review and revision before implementation even in pilot form takes place. Once this review and revision occurs, practitioners can implement the pilot, and ascertain if the methodology creates consistently more accurate forecasts than traditional methods. If the pilot is successful, the methodology becomes one more tool in the intelligence analysts’ quiver.