By William V. Pelfrey Sr. and William D. Kelley Jr.

Abstract

While there is nothing particularly wrong with proceeding forward into the uncertain future of homeland security education, much of the movement has been without directional evidence and debates as to direction have generated more heat than light. We conducted research to help us determine trajectory based on evidence. This research produced findings informed by three groups of homeland security professionals. One group, consisting of 382 respondents, represented homeland security leaders and administrators graduating from the master of arts program at the Naval Postgraduate School. The second group consisted of faculty teaching in that graduate program. The third group was a subject matter expert panel of national leaders in homeland security. Surveys were conducted across these groups, asking that they score the importance of objectives and capabilities associated with the multitude of disciplines comprising homeland security. We found that strategic collaboration, critical thinking and decision-making, foundations of homeland security, and analytical capabilities are the most important attributes of a graduate program dedicated to homeland security. Cognate or specific knowledge, the category frequently argued about in the literature, was scored as the least important category for graduate education. These capabilities and attributes represent a “way forward” that is research- and evidence-based, but questions remain.

INTRODUCTION: A Way Forward

In a remarkable occurrence, the American people over the past decade have come to value the set of activities that comprise homeland security and the related tasks of emergency and crisis management. In the wake of terrorist attacks, hurricanes, and earthquakes, a more genuine appreciation has developed for prevention, preparedness, response, mitigation, recovery, and consequence management. More to the point, most people seem to realize these activities significantly contribute to the quality of life or lack thereof in our communities, today and in the future. Accordingly, an unusual importance has attached to these tasks. Their performance is less and less seen as an aspirational goal and has moved toward becoming at least an expectation if not a mandatory requirement. It is a worthwhile exercise to identify and nurture the catalysts that are capable of enhancing our government’s abilities to successfully execute these tasks.

There is significant evidence that education is a potent and durable contributor to changing and enhancing performance in a wide range of endeavors in which excellence is sought.¹ This fact, coupled with what has been a significant investment in homeland security education for the past several years, suggests two lines of inquiry.

• First, what is the value of homeland security education?
• Second, and perhaps more importantly, what is it that homeland security education ought to be doing – and ought not to be doing – to ensure better solutions or performance in the face of more challenging threats and incidents?

Absent this inquiry, the potential for being “prepared” is not high and the opposite is possible.

To conduct this exploration of what homeland security education ought to be in order to best address the exigencies of a better-prepared nation, ongoing research was synthesized and new research conducted in 2010 and 2011. It is important to clarify that the exploration focused on education, not on training. Education intends to enhance the performance of strategic, complex cognitive tasks, such as planning, coordination, and achievement of consensus. Training is best suited to improving the performance of more tactical, simpler tasks such as using weapons or equipment, entering dangerous “hot” zones, or negotiating physical barriers, all in conformity with existing standards.

To focus and guide the discovery of a plausible way ahead for homeland security education, the research addressed five fundamental questions, with each question asked in the context of an overarching goal of national preparedness.

1. Who should be the consumers of homeland security education? Or, asked differently: Who are the most appropriate students for homeland security education?
2. What is the effect of homeland security education? Or: What does homeland security education best prepare students to do?
3. What learning objectives and capabilities should be the foundation of homeland security education? Or: What should courses and curricula for homeland security education teach?
4. What courses and curricula best serve as vehicles for educating the appropriate students on the appropriate objectives and capabilities?
5. Other than homeland security programs, are there established, more mature fields/disciplines/programs that provide education to appropriate students on the appropriate capabilities for homemade security?

Research Method

These fundamental questions were posed adhering to a methodology significantly impacted by what is best characterized as “research informed curriculum design.” The key feature of this methodology is the use of expert judges. The curricular elements are derived through research processes using subject matter experts to judge the worth and importance of the elements to them and other homeland security professionals like them.

Three distinct groups were surveyed for this research: (1) graduates of the master of arts degree program in National Security Affairs, Homeland Security and Defense, at the Naval Postgraduate School (NPS); (2) faculty teaching in this program at NPS; and (3) subject matter experts outside of the NPS graduate degree program.

The largest group was the graduates of the master’s degree program at the Naval Postgraduate School. These graduating cohorts of homeland security leaders represent a proxy for iterative subject matter expert panels. The first cohort completed the program in 2004 and the most recent cohort (for the purposes of this study) graduated in 2011. A total of 427 homeland security leaders and administrators have completed the graduate program (as of June 15, 2011) and a total of 382 completed surveys were used in the analysis. The survey completion rate was 89.5 percent, high enough to allow the researchers to generalize to the entire population of graduates. The importance of all objectives was assessed on a nine-point scale from low to high. (These data are idiosyncratic to the graduate academic program offered at only this one institution, but the longitudinal nature of the research, the professional heterogeneity of the respondents, and inclusion of competencies and objectives not part of the program, make these data compelling.)

A second group surveyed was faculty teaching in the graduate program offered by the Naval Postgraduate School. Based on the items from the subject matter expert panel described above, this survey was conducted in May 2010. Twenty-four faculty members were surveyed with the results compared to those of the graduating student surveys. While there might be contamination of the importance of some items taught by these faculty and experienced by the graduates, each group is independent enough to assess each item on its own merits. The results of the survey were presented to the faculty during a one-day session held in June 2010, allowing debate and discussion regarding the implications of the results. The fact that the results included strong ratings for capabilities not included in the instruction offered by these faculty increases the face validity of the results.²

The third group of subject matter experts surveyed to assess the importance of knowledge, skills, tasks, and capabilities was larger and more diverse in terms of professional disciplines represented. Surveys were received from selected homeland security leaders representing the following professional disciplines:

• EMA Leaders
• Law Enforcement Leaders
• Fire Leaders
• Public Health and Health Care Leaders

The survey items consisted of 124 core tasks and objectives. These were identified through a merging process that included screening 575 tasks from the original and revised ODP Training Strategy,³ and the ODP Prevention Guidelines,⁴ and identifying leader’s tasks appropriate for education, not training. These were merged with eighty-one learning objectives from graduate homeland security courses offered at the Naval Postgraduate School and forty-one Department of Homeland Security capabilities and related homeland security objectives. Duplicates were then consolidated to produce 124 Core Tasks, Objectives, and Capabilities. This survey was conducted in 2009 and two Subject Matter Expert Panels were convened to assist in interpreting the results and identifying the professional disciplines likely benefiting from the education.

These three sets of surveys represent both longitudinal expert panels and cross-sectional focus group research approaches. Individually each of these approaches has its limitations. However, when used in tandem the limitations are reduced and the benefits of comparisons over time are joined with the advantage of review of the comparisons by heterogeneous groups seeking to crosscheck and validate data.

Results

A formative, rather than summative, assessment perspective underlies this research. In a rapidly changing and evolving field like homeland security, summative evaluation may not be feasible, may be inappropriate, and may lead to a misdirection of preparedness, if not contraindicative approaches. Formative evaluation, however, is evolving and must be perpetual to be truly successful. Conclusions or certainties arrived at through summative evaluation seem to misunderstand this.

Application of this study’s methodology to the five fundamental research questions revealed the following answers.

1. Who should be the consumers of homeland security education?

The most appropriate students for homeland security education are homeland security practitioners in leadership positions. Individuals aspiring to be homeland security professionals are not the most appropriate students.

The most critical, and perhaps the exclusive, consumers for homeland security education today are practitioners with homeland security administrative or leadership responsibilities, working in the fifty-one professional disciplines or groups identified in the research. Additionally, the most appropriate tier of education is at the first graduate level (master’s degree). Committees sponsored by the US Department of Homeland Security, meeting in 2004 and 2005, identified some core elements of a homeland security curriculum. However, the report stated clearly and unambiguously, “Not a single workshop participant, or any of the committee members, voiced support for an undergraduate degree program focused specifically on homeland security.”⁵ Additionally, if a later recommendation for homeland security education to foster “post formal thought” is credible, that education is best provided at the graduate level.⁶ Training is appropriate for many others in the professional disciplines but the objectives and capabilities described below are most appropriate for graduate education.

2. What is the efficacy of such education?

Homeland security education best prepares appropriate students to perform complex, cognitive tasks. It is not appropriate for simple, tactical tasks.

The research suggests that graduate education could prepare professionals in homeland security leadership positions to be much more effective in their capability to operate in an ambiguous environment (37.7 percent more effective after the graduate education), engage in strategic collaboration (54.39 percent more effective after the graduate education), and engage in critical thinking (53.72 percent more effective after the graduate education). These data were collected from end-of-program assessments of the graduate program in homeland security offered at the Naval Postgraduate School, with participants who were selected because they were already in leadership positions in their local, state, tribal, or federal agencies.

It would appear that undergraduate vocational education in homeland security, as an employment opportunity, is not central to the largest potential employment – law enforcement – even though the professional discipline is engaged in homeland security preparedness activities. Of the 463,000 sworn officers employed at the local law enforcement level, about 4,000 (less than 1 percent) were serving in intelligence positions related to combating terrorism in 2007, the most recent year for data. “Overall, 11 percent of departments had sworn personnel serving in this capacity with a total of about 4,000 nationwide.” Only one percent of local law enforcement agencies employed non-sworn intelligence analysts. “Nationwide, an estimated 238 civilian personnel from local police departments served in this type of position.”⁷

The Department of Homeland Security, Transportation Safety Administration (TSA), another potential vocational outlet for undergraduates, employs about 55,000 Transportation Security Officers (TSO).⁸ As stated by the TSA, such an employee must:

• Be a US Citizen or US National at time of application submission;
• Be at least 18 years of age at time of application submission;
• Be proficient in the English language (i.e., able to read, write, speak, and listen);
• Have a high school diploma or General Educational Development (GED) credential OR at least one year of full-time work experience in the security industry, aviation screening, or as an X-ray technician.⁹

It would appear that homeland security vocational education at an undergraduate level would not be effective in enhancing employment as a TSO. Arguably, X-ray technician curricula would be a better vocational preparation for the TSO jobs in the Department of Homeland Security.

3. What learning objectives and capabilities should be the foundation of the education?

Objectives and capabilities that should be emphasized in homeland security education steer away from specific knowledge, centering instead on more complex, cognitive tasks.

Based on data gathered since 2004 from nineteen independent survey groups, across all major professional disciplines in homeland security, the most important objectives and capabilities for homeland security leaders and administrators are:

(1) Strategic collaboration,¹⁰ which involves the following capabilities:

• Coordinate, collaborate, and communicate across agencies;
• Identify and build strategic relationships within the individual’s homeland security organization and across the homeland security community;
• Demonstrate ability to build, sustain, and operate within interagency teams/task forces;
• Improve efforts for collaboration, information-sharing, threat recognition, and target hardening between various disciplines;
• Communicate appropriately with other agencies and organizations to insure the sharing of critical information during and following a homeland security threat or incident;
• Explicitly develop “social capital” through collaboration between the private sector, law enforcement and other partners so that data, information, assistance, and “best practices” regarding prevention and vulnerabilities may be shared and collaborative processes developed;
• Assess the opportunities and impediments to collaboration between and among the various disciplines;
• Participate in intelligence sharing with other appropriate agencies as part of a response to a homeland security threat or incident;
• Understand interagency coordination and the flow of intelligence for Homeland Security;
• Understand the confluence of law enforcement and the intelligence community;
• Foster and reward communication and collaboration across agency boundaries at all levels;
• Coordinate local, state, and federal assets in preparation for a homeland security threat or incident.

(2) Critical thinking and decision-making, which includes the following:

• Ability to think about complex issues using scientific/critical thinking approach to solving problems and make sound judgments;
• Capability to take action that is consistent with available facts, constraints, and probable consequences;
• Ability to operate in extreme ambiguity;
• Ability to respond quickly, effectively, and proactively to ambiguous and emerging homeland security conditions, opportunities, and risks;
• Willingness to use creative problem solving techniques to respond to homeland security issues in the most effective manner.

The entire list of categories of capabilities, from most important to least important, was:

1. Strategic collaboration
2. Critical thinking and decision-making
3. Foundations of Homeland Security
4. Analytical Capabilities
5. Leadership
6. Legal Issues
7. Strategic Planning
8. Cognate or Specific Knowledge

Arguably, these top two categories – strategic collaboration and critical thinking and decision-making – could be imbedded in every course in a graduate curriculum and the results would enhance practitioners’ capabilities regardless of their professional discipline.

4. Is there sufficient agreement as to the homeland security courses serving as the vehicles for educating appropriate students in the appropriate capabilities?

There is little agreement on what courses/curricula best serve the needs of homeland security professionals. A “cook book” of core courses is at best aspirational and at worst misleading or misdirected.

Based on available literature, it appears that there is no more agreement on homeland security core curricula today than in 2007, when Rollins and Rowan found “The homeland security academic discipline is currently an evolving ungoverned environment of numerous programs purporting to prepare students for various positions of responsibility.”¹¹ Fundamental debates over vocational education versus civic education, graduate only education versus undergraduate and graduate are still being conducted with little resolution of the issues, while some are calling for accreditation standards to mitigate the uncertainties (although accreditation prior to resolving the issues seems to be anachronistic).

5. Are established, more mature, parallel disciplines better capable of educating students in the appropriate capabilities?

It appears that established programs in other fields and disciplines do not offer the requisite objectives and capabilities of homeland security education.

While it was initially expected that existing programs such as public policy and public administration would better accomplish the two most important elements and cognates could address the remaining ones, examination of the core courses in those disciplines seems to suggest otherwise. Consider one respected program, Harvard University’s, John F. Kennedy School of Government, Master in Public Policy.¹² The core elements addressed by the coursework in that program contribute to the “foundations in three methodological areas:”

• Analysis
• Management
• Leadership

This is accomplished through the following coursework:

• Economics
• Quantitative Analysis
• Politics and advocacy
• Financial management
• Strategic management
• Ethics
• Leadership

Strategic collaboration, largely an affective capability, is not present in this or other similar programs of study. Additionally, valuable but largely extraneous topics are included as core in these degrees, e.g., economics. The conclusion, therefore, is that these parallel programs do not suffice in meeting the needs of homeland security graduate education.

Homeland security education, as the answers to the fundamental questions to our exploration suggest, may not be what many presume it to be. These presumptions are that homeland security, as an academic discipline, is without much coherence, it borrows its personnel from many disciplines (most noticeably law enforcement and fire), and it lacks heritage, theory, and recognition. Consider, for example, the list of things homeland security education is missing, according to Linda Kiltz:

To date, there is no agreed upon definition of homeland security, no grand theory explaining the phenomenon of homeland security, no standardized curriculum, little discussion of the history, paradigms and philosophies of the field, and ill defined faculty roles.¹³

Nonetheless, abandoning homeland security education would widely miss the mark. There is a clear and present need for graduate education focused on homeland security professionals representing the professional disciplines. The data from the nineteen surveys, particularly those since 2007, show convincingly that the objectives and competencies for graduate education for those in homeland security leadership positions within their agencies and organizations are known, can be taught through graduate education, and will produce benefits in the preparedness of those organizations. It would, therefore, be a mistake not to continue that instruction. It would also be ludicrous to replicate the same education at the undergraduate level, since the objectives appear to be at the analysis, synthesis, evaluation, and metacognitive levels, where students are educated in thinking critically and utilizing the affective domain to engage in strategic collaboration. Graduate certificate programs could, and are, being used to address the same key objectives and competencies for those not interested in completing a graduate degree or already holding a graduate degree. The formative environment of the Naval Postgraduate School’s homeland security graduate curriculum remains a viable location to test courses and instructional techniques to infuse the key elements into graduate education and share the successes and failures with others in academe. As curriculum planners engage in the graduate versus undergraduate homeland security education debate, the prevailing question should continue to be: Education for what purpose? Armed with the data from this research, we can articulate both the purposes and the capabilities to address those purposes in the venue of graduate education. There appears to be little vocational support for undergraduate education in homeland security but there may be stronger academic objectives, such as critical thinking and critical writing, embraced and explicitly articulated in the courses developed for advanced undergraduates.

The lingering larger question of our exploration is what direction should homeland security education take if we are to fulfill the promise of moving our nation’s preparedness level beyond its current unacceptable level?¹⁴ Here, the exploration has much to suggest, although it will not be achieved in short order. The way forward suggests a dependence on evidence rather than opinion, and reality rather than belief. The way forward, more specifically, would include efforts to:

• Continue to encourage graduate education, but strongly encourage the inculcation of the key objectives and capabilities identified in this research, particularly the development of strategic collaboration capabilities, the ability to think critically and analytically, and the capability to operate in the ambiguous environment of homeland security;
• Assess the courses and the program using those key variables as dependent variables in the assessment processes;
• Assess the impact of homeland security education using disciplined, reliable methods that can discriminate effects based on the current and future attributes of curricula;
• Disseminate the results to other universities and colleges with recommendations of smart practices, as well as the theories and methods used to develop and test the capabilities in both homeland security and emergency management academic programs;
• Encourage (through special journal issues, fellowships, and proactive recruitment) faculty in existing disciplines to adopt homeland security issues and problems within their research agendas so that those expert in these disciplines can contribute to the progress of research and theory development in homeland security;
• Encourage the Department of Homeland Security to partner with the US Department of Education, Health and Human Services, and other federal agencies, to take a leadership role in a process similar to the Bologna Process.¹⁵ This would involve identifying – with some particularity – the roles and objectives of undergraduate and graduate education, using homeland security education as the example since it is in the germinal stages of development;
• Engage representatives of more mature disciplines, already contributing to homeland security education and research, to be manifestly involved in the development of theories, methods, and analytical capabilities that should be considered in the development of graduate homeland security education. In doing so, these individuals could articulate the theories, analytical capabilities, and research methods appropriate to contribute to the paradigm of homeland security.

CONCLUSION

Much of what is stated above is conclusory and this article is brief enough that no “recap” is needed. One unstated but underling issue has not been broached: Is it time for homeland security to be considered an academic discipline?

Determining the degree to which the “discipline” of homeland security is coalescing is more difficult and methodologically uncharted. Rollins and Rowan’s work represented a strong methodological model. The literature on “model” curricula,¹⁶ along with the cross tabulation-like tables used to present some of the academic homeland security programs and public policy/public administration programs mentioned in the literature, was used in interpreting some of the survey data. This type of assessment is not likely to provide “clear and convincing” insight but will suggest certain recommended steps or actions.

The literature also suggests that academic journals are an element of an academic discipline. The Journal Citation Reports has been used as a bibliometric to assess the progress of some disciplines and the respect the disciplines seem to be developing among researchers.¹⁷ That method to test the development of homeland security (and emergency management) as a viable discipline in academe shows rather starkly, that homeland security is immature. A search of citations in July 2011, using PAIS International,¹⁸ indexed by ProQuest and available through CSA, yielded 409 citations for “homeland security.” These citations began in 2001 and regressed across time. Screening these articles by the category “scholarly,” generally meaning the articles were in peer-reviewed journals, produced a list of ninety-seven articles. While there is no benchmark or standard for the number of scholarly publications necessary as a predicate for an academic discipline, an average of fewer than ten a year is suspect.

Whether homeland security is an interdisciplinary or a multi-disciplinary study area can be debated, but it appears not to have evolved to a point where idiosyncratic theories and methods of research specific to homeland security are better paradigmatically than those of the disciplines initially producing them and coming together to address or assess the issues in homeland security. Homeland security education appears to be too immature and amorphous, with its educational goals in dispute, to merit proceeding vigorously in the development of new programs beyond those providing the knowledge and capabilities needed by those leaders already in defined homeland security roles and key public safety positions.

Faculty in the emerging discipline of homeland security, seeking to craft (or cobble together) courses and coursework may, in their zeal to incorporate and homogenize the theories and research of others, drift away from their areas of expertise and do a less-than-creditable job instructing students when faculty more central to the disciplines being taught are available. A quote attributed to Paul Samuelson, the Nobel Laureate Economist, in his Collected Scientific Papers on the state of the discipline of economics seems appropriate: “Economics has never been a science, and is even less now than a few years ago.”¹⁹

It is more useful to doubt the coagulation of courses and curricula into a discipline or a science than to proclaim success and rigidly hold to rapidly outdated foundations. At this stage in the development of “homeland security education” a wiser approach would be to capitalize upon the development of homeland security imperatives and research within existing disciplines, thus building a firm foundation for a more mature discipline of homeland security. To do otherwise risks taking the path away from science and a discipline, as observed by Samuelson.

About the Authors

William V. Pelfrey, Sr., of Pelfrey and Associates LLC, has engaged in academic curriculum development, instruction, and training for thirty years, holding faculty and administrative positions in seven major universities before retiring in 2005. He has authored articles in Evaluation Review: A Journal of the Institute for Law and Public Policy, Homeland Security Review, The Journal of Homeland Security and Emergency Management, Criminology, Criminal Justice Review, Journal of Criminal Justice Education, other journals, and law reviews, as well as book chapters and books. He is coauthor of the WMD Training Strategy 2002 and ODP Prevention and Deterrence Guidelines. His research has been in the areas of homeland security, curriculum development, crime analysis and assessment, criminological theory, trends and patterns in violent crime, policing, and victimology. He holds the PhD in Criminology from Florida State University.

William D. Kelley, Jr. has a distinguished record of service in operations, administration, teaching, and research concentrating on public safety and domestic preparedness. Mr. Kelley is currently senior advisor/consultant for the Center for Homeland Defense and Security (CHDS) at The Naval Postgraduate School(NPS). Prior to 2010, he was a consultant to the US Department of Homeland Security’s (DHS) Federal Emergency Management Agency (FEMA) National Preparedness Directorate (NPD) in Washington, DC. His work has focused on research and strategic planning regarding prevention, preparedness, mitigation, response and recovery relative to incidents and acts of terrorism with special emphasis on evidence based education and training strategies and programs for executive level personnel. His experience has included similar consecutive positions beginning in 1998 contiguous with the creation of a key DHS predecessor agency at the US Department of Justice. Mr. Kelly has been directly involved in supporting the management of the Center for Homeland Defense and Security (CHDS) at the Naval Postgraduate School (NPS) since its inception in 2002.

1. See, for example: Frances Young, “Educating for Excellence” British Journal of Educational Studies 26, no. 2 (1988): 100-110; and, more recently, Philip Altbach, Liz Reisberg and Laura E. Rumbley, Trends in Global Higher Education: Tracking an Academic Revolution: A Report Prepared for the UNESCO 2009 World Conference on Higher Education (Paris, France: United Nations Educational, Scientific and Cultural Organization, 2009).↵
2. The reliance on respondents associated with the graduate program at the Naval Postgraduate School is likely to be a methodological concern. The disciplinary and geographic diversity make these graduates a particularly relevant population and the fact that they were exposed to the same curriculum improves inter-rater reliability. Additionally, the high importance ratings of items they were not exposed to in the curriculum increases the criticality of those assessments and neutralizes the argument of favorable bias. Absent data to the contrary, these results should serve as the basis for hypothesis testing in other populations.↵
3. W.V. Pelfrey, W. D. Kelley, Jr. and J. W. May, Jr., The Office for Domestic Preparedness WMD Training Strategy. Executive Summary, Volumes 1 and 2, and Appendices, prepared for The Office for Domestic Preparedness, Office of Justice Programs (Washington, DC: US Department of Justice, 2002).↵
4. Tom Ridge, The Office for Domestic Preparedness Guidelines for Homeland Security June 2003: Prevention and Deterrence (Washington, DC: Department of Homeland Security, 2003).↵
5. Committee on Educational Paradigms for Homeland Security, National Research Council, Frameworks for Higher Education in Homeland Security (National Academy of Science Press, 2005), 19, http://www.nap.edu/catalog/11141.html.↵
6. Michael Lamport Common, “Introduction to the Model of Hierarchical Complexity and its Relationship to Postformal Action,” World Futures 64 (2008): 305-320; and Common, Implications of Hierarchical Complexity for Social Stratification,” 430-435.↵
7. Brian A. Reaves, Local Police Departments, 2007 (Washington, DC: Bureau of Justice Statistics, 2010), 32.↵
8. Government Accountability Office (July 23, 2010), http://www.gao.gov/products/GAO-07-299.↵
9. USA Jobs, “Working for America,” retrieved on July 23, 2011 at http://jobview.usajobs.gov/GetJob.aspx?JobID=100988898&JobTitle=Transportation+Security+Officer+(TSO)&q=tsa&where=&brd=3876&vw=b&FedEmp=N&FedPub=Y&x=120&y=12&AVSDM=2011-07-18+11%3a42%3a00.↵
10. Italicized items were scored highest in importance by the expert panels as well as appearing on the list of highest importance on the practitioner surveys.↵
11. John Rollins and Joseph Rowan, The Homeland Security Academic Environment: a Review of Current Activities and Issues for Consideration (Homeland Security and Defense Education Consortium, 2007).↵
12. HARVARD Kennedy School website, http://www.hks.harvard.edu/degrees/masters/mpp.↵
13. Linda Kiltz, “The Challenges of Developing a Homeland Security Discipline to Meet Future Threats to the Homeland,” Journal of Homeland Security and Emergency Management 8, no. 2 (2011), Article 1: 13.↵
14. See, for example, Progress Made and Work Remaining in Implementing Homeland Security Missions 10 Years after 9/11, GAO-11-940T (Washington, DC: Government Accountability Office, September 8, 2011).↵
15. Clifford Adleman, The Bologna Process for U.S. Eyes: Re-learning Higher Education in the Age of Convergence (Washington, DC: Institute for Higher Education Policy, 2009), www.ihep.org/Research/GlobalPerformance.cfm.↵
16. See for example, Harvey Alverch and Milan Dluhy, “Teaching Public Administration, Public Management, and Policy Analysis: Convergence or Divergence in the Masters Core, Journal of Policy Analysis and Management 11, no. 3 (192): 541-551.↵
17. See for example, Benjamine M. Althouse, Jevin West, Carl T. Bergstrom, and Theodore Bergstrome, “Differences in Impact Factor Across Fields over Time,” Journal of the American Society for Information Science and Technology 60, no. 1 (2009): 27-34; Duncan MacRae Jr. and Irwin Feller, “The Structure of and Prospects for Policy Research as Suggested by Journal Citation Analysis,” Policy Studies Review 15, no. 1 (1998): 115-135; Jong Yong Abdiel Foo, “A Study on Journal Self-Citation and Intra-Citing Within the Subject Category of Multidisciplinary Science,” Scientific Engineering Ethics 15 (2009): 491-501.↵
18. The PAIS International database from CSA contains citations to journal articles, books, government documents, statistical directories, grey literature, research reports, conference reports, publications of international agencies, microfiche, Internet material, and more. Newspapers and newsletters are not indexed. PAIS International includes publications from over 120 countries throughout the world. An enhanced subscription to PAIS International on CSA Illumina is also available that includes access to the PAIS Archive. The PAIS Archive provides retrospective coverage of the PAIS Annual Cumulated Bulletin. The database back file dates to 1972: PAIS International includes records from the print PAIS Bulletin 1977 and forward; and from the PAIS print Foreign Language Index published 1972-1990, at which time it merged with the PAIS Bulletin. Major areas of PAIS Subject Coverage include: oAdministration of justice oAgriculture, forestry and fishing oBanking and finance oBusiness and service sector oCulture and religion oEconomic conditions oEducation oEnergy resources and policy oEnvironment oGovernment oHealth conditions oHuman rights oInternational Relations oLabor conditions and policy oLaw and ethics oManufacturing and industry oMedia and communications oMilitary and defense policy oPolitics oPopulation and demographics oScience and technology oSocial conditions oTrade oTransportation ↵
19. Paul A. Samuelson, Collected Scientific Papers (Cambridge, MA: MIT Press, 2011)↵

---

This article was originally published at the URLs https://www.hsaj.org/?article=9.1.3 and https://www.hsaj.org/?fullarticle=9.1.3.

Copyright © 2013 by the author(s). Homeland Security Affairs is an academic journal available free of charge to individuals and institutions. Because the purpose of this publication is the widest possible dissemination of knowledge, copies of this journal and the articles contained herein may be printed or downloaded and redistributed for personal, research or educational purposes free of charge and without permission. Any commercial use of Homeland Security Affairs or the articles published herein is expressly prohibited without the written consent of the copyright holder. The copyright of all articles published in Homeland Security Affairs rests with the author(s) of the article. Homeland Security Affairs is the online journal of the Naval Postgraduate School Center for Homeland Defense and Security (CHDS). https://www.hsaj.org