Word count 3000 Please see attached files. look at Question.pdf

The Bayesian Approach to Security Effectiveness through metrics, modeling and decision -making (BASE m 2d) concep tual framework was developed by this research to address this pervasive problem. Specifically, this research illustrates the framework by examining multi -order effects on hospital operations, thereby assessing the likelihood of impact to a patient’s health given a successful cyber or physical (natural or man -made) attack on a dependent CI. A survey was provided to medical professionals at 10 different hospitals to help identify current risk management processes used by the medical professionals to understa nd, assess , and validate patient impact given DoS to a dependent CI (Power, Water, and Communications). The viii probabilistic Bayesian module allowed for a scenario -based, what -if impact analysis, given limited available data. This research revealed, d espite t he known dependence on CIs, no standardized metrics or processes are used to assess patient impact for risk mitigation given a DoS. Also noted was a lack of general preparedness, training , and methods of sharing information in the event of a DoS on a depe ndent CI. Consequently, the findings of this research resulted in a hybrid, hierarchical, multi -dimensional approach grounded in systems engineering principles . ix TABLE OF CONTENTS DEDICATION ................................ ................................ ................................ ................................ ....... IV ACKNOWLEDGMENTS ................................ ................................ ................................ ........................... V ABSTRACT ................................ ................................ ................................ ................................ ........ VII TABLE OF CONTENTS ................................ ................................ ................................ .......................... IX LIST OF FIGURES ................................ ................................ ................................ ............................... XII LIST OF TABLES ................................ ................................ ................................ ............................... XIII LIST OF ACRONYMS ................................ ................................ ................................ .......................... XIV CHAPTER 1 - INTRODUCTION ................................ ................................ ................................ ................ 1 1.1 Research Background ................................ ................................ ................................ ....... 5 1.1.1 Overview of CI ................................ ................................ ................................ ............... 5 1.1.2 Healthcare and Public Health ................................ ................................ ........................ 7 1.2 Motivation ................................ ................................ ................................ ........................ 10 1.3 Research Problem ................................ ................................ ................................ .......... 11 1.4 Research Objectives ................................ ................................ ................................ ....... 12 1.5 Scope and Limitations ................................ ................................ ................................ ..... 13 1.6 Research Contribution ................................ ................................ ................................ .... 14 1.7 Significance/Implicatio ns of Research ................................ ................................ ............. 15 1.8 Definitions of Key Concepts ................................ ................................ ............................ 17 1.8.1 Security Effectiveness for the Operational Environment ................................ ............. 17 1.8.2 Hierarchical Holographic Modeling (HHM) ................................ ................................ .. 18 1.8.3 Bayesian Belief Network (BBN) ................................ ................................ ................... 20 1.8.4 Systems Security Effectiveness Index (SSEI) ................................ ............................. 21 1.9 Organization and Outline ................................ ................................ ................................ . 22 CHAPTER 2 - LITERATURE REVIEW ................................ ................................ ................................ ..... 24 2.1 Review of Related Works ................................ ................................ ................................ 24 2.2 Overview of CIP/R Analysis ................................ ................................ ............................ 25 2.3 Approaches to CI Prot ection & Resiliency (CIP/R) ................................ .......................... 28 2.3.1 Implementations of Decision Analysis Tools for CIP/R ................................ ................ 31 x 2.3.2 Implementations of Probabilistic Risk Modeling for CIP/R ................................ ........... 32 2.4 Approaches to Measuring Security Effectiveness for CI ................................ .................. 34 2.5 Sum mary ................................ ................................ ................................ ......................... 34 CHAPTER 3 - RESEARCH METHODOLOGY ................................ ................................ ............................ 35 3.1 Research Design ................................ ................................ ................................ ............. 36 3.2 Expert Elicitation ................................ ................................ ................................ ............. 37 3.3 Survey Scale ................................ ................................ ................................ ................... 40 3.4 Expert Elicitation Calibration ................................ ................................ ........................... 40 3.5 Survey Instrume nt ................................ ................................ ................................ ........... 41 3.6 Validity of Survey Instrument ................................ ................................ ........................... 43 3.7 Validity of Conceptual Framework ................................ ................................ ................... 44 3.8 Data Collection ................................ ................................ ................................ ................ 45 CHAPTER 4 – HOSPITAL CASE STUDY ................................ ................................ ................................ . 46 4.1 Case Study Background ................................ ................................ ................................ .. 46 4.2 Case Study Application ................................ ................................ ................................ ... 47 4.2.1 Step 1 and 2: Define Operational Environment and Security Goals ........................... 49 4.2.2 Step 3: Identify Dependencies ................................ ................................ ................... 50 4.2.3 Step 4: Assess/Measu re the Security Posture ................................ ........................... 50 4.2.4 Step 5: Assess multiple dimensions/perspectives [CI -HHM] ................................ ...... 53 4.2.5 Step 6: Assess Strength/Weakness [Calculate the SSEI] ................................ .......... 56 4.2.6 Step 7: Assess Impact Likelihood [Construct the BBN] ................................ .............. 59 4.2.7 Steps 8 -10: Decision Ana lysis ................................ ................................ .................... 65 CHAPTER 5 - DATA ANALYSIS AND RESULTS ................................ ................................ ....................... 65 5.1 Analysis Objectives ................................ ................................ ................................ ......... 65 5.2 Demographic Data ................................ ................................ ................................ .......... 66 5.3 Metrics ................................ ................................ ................................ ............................ 68 5.4 Descriptive Data ................................ ................................ ................................ .............. 70 5.5 Reliability and Validity of Survey Instrument ................................ ................................ ... 72 5.6 Validity of Conceptual Framework ................................ ................................ ................... 74 5.7 Threats to Internal Validity ................................ ................................ ............................... 75 5.8 Threats to External Validity ................................ ................................ ............................. 75 xi CHAPTER 6 – CONCLUSION ................................ ................................ ................................ ................ 76 6.1 Conclusion with Respect to Study Hypotheses ................................ ............................... 77 6.2 Conclusion with Respect to Study Questions ................................ ................................ .. 77 6.3 Discussion ................................ ................................ ................................ ....................... 78 CHAPTER 7 - FUTURE RESEARCH ................................ ................................ ................................ ....... 81 REFERENCES ................................ ................................ ................................ ................................ .... 83 APPENDIX A ................................ ................................ ................................ ................................ ...... 96 APPENDIX B ................................ ................................ ................................ ................................ .... 102 xii LIST OF FIGURES Figure 1. CI Interdependency Multi -order Effects ................................ ................................ ............ 9 Fig ure 2. HHM for Critical Infrastructure or Dependent Resources (CI/DR -HHM) .......................... 19 Figure 3. Research Focus Areas ................................ ................................ ................................ .... 27 Figure 4. General Flow of BASE m2d Framework ................................ ................................ ......... 48 Figure 5. Simplified BBN ................................ ................................ ................................ ................ 60 Figure 6. Representation o f BASE m2d Model of DoS attack on CIs (w/o SSEI) ........................... 62 Figure 7. Representation of SSEI Analysis given DoS Attack on Power CI ................................ ...63 Figure 8. Survey: Years of Experience in Medical Field ................................ ................................ 67 Figure 9. Survey: Medical Field Profession ................................ ................................ ................... 67 Figure 10. Survey: Experience in Intensive Care Unit (ICU) ................................ .......................... 67 xiii LIST OF TABLES Tab le 1. Healthcare and Public Health Sector CI Dependency (DHS.gov) ................................ ..... 8 Table 2. Literature Review CI Model Comparative Analysis ................................ ........................... 30 Table 3. Reference Metrics for Critical Infrastructure/Dependent Resource Protection ................. 52 Table 4. Reference Metrics for Critical Infrastructure/Dependent Resource Resilience ................. 53 Table 5. CI/DR -HHM SSEI Scoring Scale (by Category) ................................ ............................... 55 Table 6. CI/ DR -HHM SSEI Scoring Scale (Total) ................................ ................................ ........... 56 Table 7. Area of Improvement/Deficiency (Calculated SSEI) ................................ ......................... 57 Table 8. Exemplar Stakeholder Question Categories ................................ ................................ .... 58 xiv LIST OF ACRONYMS AHP Analytical Hierarchy Process BASE m 2d Bayesian Approach to Security Effectiveness with metrics, modeling, and decision - support BBN Bayesian Belief Network CI Critical Infrastructure CIP/R Critical Infrastructure Protection and Resiliency CPT Conditional Probability Table DA Decision Analysis DAG Direct ed Acyclic Graph DHS Department of Homeland Security DOS Denial of Service DR Dependent Resource EO Executive Order GAO Government Accountability Office HHM Hierarchical Holographic Model ICU Intensive Care Unit MAUT Multi -Attribute UtilityTheory NIPP National Infrastr ucture Protection Plan PDD Presidential Decision Directive PPD Presidential Policy Directive SCADA Supervisory Control and Data Acquisition SSEI Systems Security Effectiveness Index SSP Sector Specific Plan 1 CHAPTER 1 - INTRODUCTION The defin ition of security is the state of being free from danger or threat. While this defined state of being can never be achieved within the cyber domain, the knowledge of one’s security posture (as it relates to exposur e to danger or threat ) is paramount to ul timately understanding cyber and/or physical security and implementing appropriate, timely , and necessary security measures. To that end, there has been an avalanche of standards, tools , and compliance guidance added to the cadre of weapons for cyber -warf are; inevitably developed to afford a perceived sense of security that the devices, networks, systems, and enterprises are secure. Still, organizations’ relative security posture (i.e. assessed security effectiveness per defined security goals) remains un known despite Information Security Professionals having implemented the suggested/recommended security configurations, tools, policies , and plans in an effort to defend against various known threats . The risks of misinterpreting one’s security posture can have devastating consequences in terms of misallocation of resources, loss of revenue, loss of competitive advantage, loss of privacy, loss of trust, damage to reputation, destruction of property, and in some cases , potential loss of life. These conseque nces are amplified when they are associated with the elements of the nation’s Critical Infrastructures (CIs) . CIs, as defined by the Department of Homeland Security (DHS), are the assets, systems , and networks, whether physical or virtual, so vital to th e United States that their incapacitation or destruction would have a debilitating effect on security, national economic security, national public health or safety, or any combination thereof [DHS.gov 2013]. CIs are large, complex, adaptive , and highly in terconnected; they are ripe with potential areas where system knowledge can easily go unexamined and thus undiscovered. Implementing security measures without considering the multi -dimensional aspect of interdependencies that are both internal and external to CIs can lead to 2 undesirable consequences. This can ultimately imperil efforts to achieve CI protection goals. Moreover, goals can be negatively impacted by providing an illusion of acceptable security where there are actually areas of unacceptable and intolerable risk - in spite of operators having employed every available tool, policy , and standard. Within the multi -dimensional regions of CIs there are cyber -based, temporal, geo -spatial, legislative, societal, economic , and stakeholder attributes that can be explored and considered to uncover unknown insight into a CI and /or dependent resource’s true security posture. According to Government Accountability Office (GAO) reports, over the past decade there has been a sustained increase in malicious pen etrations to government information systems [ICS - CERT 2017, DHS 2016, GAO -15 -573T 2015, GAO -12 -666T 2012]. In 2012 alone, America's power, water, financial institutions, nuclear systems , and other key resources have experienced a 52 percent increase in ta rgeted attacks by cyber criminals seeking to gain (or deny) access to the nation's CI [CERT 2013]. Parties aligned with the Russian government have developed a cyberweapon (CrashOverride) specifically designed to destroy industrial control systems of CIs. Considered Stuxnet 2.0, CrashOverride hackers briefly shut down one -fifth of the electric power generated in Kiev (washingtonpost.com 2017). Recent ransomware cyber -attacks such as WannaCry and Petya have affected more than 200,000 computers, causing chao s and disruption for critical infrastructure and dependent resources ( CI/DR ), including major hospitals, a nuclear disaster site, and electrical grids (CNET 2017). It was noted by Healthcare Informatics: “this attack (WannaCry) shows that interconnected d evices and systems are vulnerable to attack by nations, non -state actors , and just plain crooks. An attack of this scope points to the potential for an entirely different type of damage: shutting down entire businesses, hospital systems, banks, and criti cal infrastructure” (2017). Cyber criminals are demonstrating their ability to access sensitive information, disturb the integrity of personal data , and block the availability of information systems. 3 However, their ability to manifest the physical destruct ion traditionally associated with kinetic warfare has yet to be fully realized. Dually noted by the GAO and U.S. adversaries, the nation’s CIs and Key Resources remain vulnerable to the potential devastation of cyber -attacks. As indicated in the Presiden tial Policy Directive 21 (PPD 21), U.S. CIs remain a high value target [GAO -18 -62 , 201 7]. According to these recent reports (EO, PPD , and GAO), implementations of existing standards, policy, legislation, methodology , and tools have not provided sufficient confidence, guidance , or rigor toward the effective protection against these increasingly frequent and potentially destructive attacks [GAO -13 -462T , 2013 ; GAO -17 -518T , 2017 ]. As noted by these reports and the adversary’s ability to continually penetrate CIs, it is critical not only to have the ability to assess the effectiveness of various security measures, but also to have a method that cogitates (considers/includes) the complexity and interdependencies of CIs. This research offers a case study to demo nstrate such a method - the Bayesian Approach to Security Effectiveness with metrics, modeling , and decision -support (BASE m 2d) conceptual framework. This comprehensive method is used to assess a CI ’s security posture by evaluating the effectiveness of no t only implemented measures but proposed measures, as well; potentially avoiding devastating (unintended or intended) consequences. As a result, the risk of not protecting CIs or providing the resiliency required to maintain services to dependent key reso urces such as hospitals is explored . Specifically, this study examines the likelihood of impact to a patient’s health given a successful cyber or physical (natural or man -made) Denial of Service (DoS) attack on a dependent CI using the BASE m 2d framework (construct). 4 Understanding the potential impacts to elements of the Healthcare and Public Heath (HPH) sector given a successful Denial of Service (DoS) to CIs is imperative to ensure proper security effectiveness measures are in place to avoid poten tially irreversible consequences. Further, a strategic, holistic construct is proposed to properly focus budget constrained resources based on the assessment of potential weaknesses of CIs as defined per their security goals or protection targets (patient s, personnel, medical devices, etc. ). The objective of this study is to provide emergency management personnel a conceptual framework and methodology to evaluate security effectiveness and effects of cascading risk that may result from inadequate security measures. This research is presented as a proof of concept using a combination of real and notional data. Elicitation of specific data from medical experts was limited due to the acknowledged vulnerability of hospitals and the potential insight it may prov ide to adversaries. This research revealed the troubling finding that of the 13 hospitals pursued for expert elicitation, medical professionals from 10 distinct hospitals noted their current risk assessment plans did not include CI interdependency patient impact metrics or analysis given a denial of service on a dependent CI. Consequently, the disparate measures hospital engineers, IT professionals , and physicians use today to protect and maintain services they provide for the ultimate purpose of preserving life are discussed. Further, this framework is being offered as a method that considers those distinct security measures and provides a holistic approach to ensure better protection and resiliency toward patient care. The BASE m 2d framework was validate d with data provided by medical professionals from 10 different hospitals. 5 1.1 RESEARCH BACKGROUND 1.1.1 OVERVIEW OF CI As a result of devastating events (Oklahoma bombing, Katrina, 9 -11, Stuxnet, etc.) and the nation’s lack of preparedness, congressional attention has resulted in the government, academia, and private industry taking action. The dedicated focus on C ritical Infrastructure Protection (CIP) spans across four presidential administrations. In 1996 , President Clinton established the President’s Commissi on on Critical Infrastructure Protection (PCCIP) (E.O. 13010). Although no immediate threats to critical infrastructures were noted upon the release of the PCCIP in 1997, it did stress the importance of CI interdependencies. In 1998 the Presidential Deci sion Directive, number 63 (PDD - 63) was released. This directive sought to protect , by the year 2003, the nation’s CIs from deliberate attacks. PDD -63 was later updated by President Bush through the Homeland Security Presidential Directive HSPD -7 to estab lish a national policy for Federal departments and agencies to identif y and prioritize United States Critical I nfrastructure and Key Resources and to protect them from terrorist attacks (HSPD -7, 2003). The E.O. 13130 established by President Clinton in 19 99 and E.O. 13231 by President Bush in 2001, essentially instituted Information Sharing and Analysis Centers (mostly facilitated by the private -sector) and a National Infrastructure Advisory Council (NIAC). In 2002, the Department of Homeland Security (D HS) was established and charged with the primary responsibilities of protecting the United States and its territories from - and responding to - terrorist attacks , man -made accidents , and natural d isasters (DHS.gov). In 2013, President Obama issued E.O. 13636 and PPD -21. These two directives aim to “enhance the security and resilience of the n ation’s critical infrastructure and to maintain a cyber environment that encourages efficiency, innovation , and economic prosperity while promoting safety, security, business confidentiality, privacy, and civil liberties.” (E.O. 13636, PPD -21 ) Most recently (2017) , 6 President Trump released an Executive Order on Strengthening the Cybersecurity of Federal Netwo rks and Critical Infrastructure. Together, these policies are intended to achieve the following (DHS.gov ): • Encourage the adoption of effective measures across all critical infrastructure sectors to improve security and resiliency and reduce risk from cyb er-attacks to essential functions and services by publishing a Cybersecurity Framework (the Framework , 2017 ) that will provide owners and operators with a prioritized, flexible, repeatable, performance -based, and cost -effective set of validated security co ntrols based upon industry best practices. • Enhance timely, relevant, and accurate information sharing on significant risks by implementing a program to develop and rapidly share unclassified information with critical infrastructure owners and operators, en abling the adoption of effective mitigations to prevent or to reduce the consequences of significant incidents. • Align responsibilities of public and private partners to efficiently allocate risk reduction responsibilities by conducting an analysis of the e xisting critical infrastructure public - private partnership model and recommending options for improving the effectiveness of the partnership in managing both the physical and cyber risks. • Promote innovation in novel risk -reduction solutions by developing a National Critical Infrastructure Security and Resilience Research and Development (R&D) Plan to identify priorities and guide R&D requirements and investments toward those solutions that will help assure the provision of essential functions and services o ver time. • Ensure that privacy, civil rights, and civil liberties are protected as a foundational part of all risk management efforts by conducting an assessment of the privacy, civil rights, and 7 civil liberties implications of all EO 13636 and PPD -21 progr ams and recommending revisions to proposed initiatives as required. The Presidential Directive 21 (PPD 21) identified 16 CI sectors and designated specific federal - agencies to facilitate/head protection and resiliency programs and activities. The sectors are identified as follows: 1. Agriculture and Food 2. Banking and Finance 3. Chemical 4. Commercial Facilities 5. Communications 6. Critical Manufacturing 7. Dams 8. Defense Industrial Base 9. Emergency Services 10. Energy 11. Government Facilities 12. Healthcare and Public Health 13. Information Technology 14. Nuclear Reactors 15. Transportation Systems 16. Water and Sewage It is commonly acknowledged that the interdependencies between critical infrastructure sectors are of great importance to the protection of the nation. 1.1.2 HEALTHCARE AND PUBLIC HEALTH Hea lthcare and public health facilities rely on various CIs in order to maintain daily operations. A cyber or physical attack on any of those interdependent CIs can indirectly have a detrimental impact on a patient’s health or their personal data. Depending on the motivation of the attacker, i.e., to disturb the confidentiality, availability , or integrity of the hospital service or patient’s data, the 8 impact can be irreversible. Table 1 describes the critical dependencies between various sectors to HPH. For e xamples, hospitals rely on critical services such as reliable power, clean water supply , and available communications . An illustration of CI interdependency includes the following: Water and Communication CIs depend on Energy (power) CIs, while Emergency Service (i.e., ambulances) CIs rely on Communication CIs. A DoS of critical services to a hospital from either a cyber or physical attack can have a cascading effect if the proper protection and resiliency security measures are not implemented. Table 1. Healthcare and Public Health Sector CI Dependency (DHS.gov) Understanding how and where to properly address and allocate security measures in a budget -constrained environment will prove invaluable to ensure these critical resou rces/services are uninterrupted (or have minimal or acceptable impact). A case is modeled in this research where a patient in ICU is critically ill and depends on dialysis for survival. Given a DoS on a Water CI, the hospital could be subsequently impact ed if the proper back -up resources are not engaged or available in a timely manner . The calculated System Security Effectiveness I ndex (SSEI) defined in this study uses Systems Engineering /Systems Thinking concepts as a foundation to 9 inform the decision -maker of deficiencies internal and/or external to the infrastructure, enabling a holistic, proactive assessment of their security effectiveness (of current or proposed security measures) before an actual attack occurs. A CI failure can be due to a successf ul attack, whether virtual or physical, that results in a service interruption (partial shutdown or complete shutdown), ultimately having a 2 nd, 3 rd or 4 th order effect [Figure 1 ]. Figure 1. CI Interdependency Multi -order Effect s The National Association of County and City Health Officials (NACCHO), in summary, have identified four categories of cyber -attack impact on healthcare and public health facilities (NACCHO 2014 ): 1. Loss of integrity : Patients and practitioners may lose co nfidence in a healthcare provider’s ability to maintain patient privacy due to perceptions of inadequate security. 2. Loss of availability : Cyber threats to data and operational systems can: take a facility off - line, leading to disruption of care; create los s of access to health records, limiting the 10 provider’s ability to provide appropriate care, shelter , and medicine; disrupt emergency telephone lines and EMS systems; and slow or disable emergency medical response systems. Cyber -attacks can also prevent or impact production and manufacturing of medical equipment or drugs. 3. Loss of confidentiality: The exposure of personal data can trigger ripple effects for victims of cyber -crime, including theft or loss of a patient’s private information or discovery of pat ient information on personal medical devices. 4. Physical destruction of systems : Cyber -attacks could damage physical systems used to perform functions, such as regulate utilities critical to healthcare and public health and could shut down or slow supply cha ins, impair patient care, and impede emergency response, potentially leading to significant loss of life. Public trust depends upon the sustainability, resilience, integrity , and availability of national HPH critical infrastructure [NACCHO 2014]. Researc h results from this study reveal that hospital engineers have performed due diligence to ensure if power is disrupted or water supplies have been tainted or halted, backup generators and alternate water supplies are available to maintain critical services. However, this research further revealed that most hospitals have not performed additional risk assessments to understand or evaluate the potential impact of a DoS (i.e., water, power) to the patient . 1.2 MOTIVATION This study is motivated by the need to do more to protect and maintain the nation’s CIs and the services they provide. In light of the mere possibility that physical destruction or loss of human life 11 can result (directly and/or indirectly) from a successful attack on a CI , is cause enough to paus e and re -assess the effectiveness of security measures in place to protect and provide resilie nc y. 1.3 RESEARCH PROBLEM The indirect consequence of a DoS on a CI can be devastating to those that depend on its services. The need to do more to protect and main tain the nation’s CIs and the services they provide is paramount considering the dire consequences if neglected. Residential communities, hospitals, banking , and government services are examples of resources that require sustained and reliable provisions from CIs such as Energy, Water , and Communications for mere survival. Thus, the ability to protect and/or maintain these critical services to dependent resources is crucial. As CIs strive to institute protective and resiliency measures, the ability to ass ess the effectiveness of those measures becomes more important not only to the CI, but also to the dependent resources. This research explores the growing concern and significance of understanding potential impacts of indirect consequences to dependent r esources given a Denial of Service (DoS) to CIs. To address the persistent challenge of protecting CIs and maintaining the essential services they deliver, this research reveals that a methodology is needed to provide CI owners/stakeholders a tool to evalu ate their security posture while ultimately allowing for proactive provisioning before potential disaster. Current methodologies used to address the complex problem of improving cyber/physical security protection of the enterprise, or specifically in this case , the nation’s CIs, must expand beyond existing traditional approaches. Most security methods used today are considered from a single -dimension. This is normally accomplished by protecting virtual or physical access to architectural elements or compon ents (network of routers, switches, servers, SCADA) from cyber - attacks, e.g., by configuring firewalls, implementing policies, limiting access to data servers, and 12 training users. Although these techniques demonstrate a noble effort, they have proved to b e neither sufficient nor effective. Further, organizations, CI owners , and operators must have a method to assess the effectiveness of the security program they put in place. A serious gap exists in the tools available to assess the effectiveness of secu rity measures which are designed to mitigate disruptions to essential CI services (NIST 2014, GAO -16 -152 , GAO -17 -518T 2017 ). Also lacking are strategic methods to evaluate the subsequent impacts resulting from interruptions to those services. Furthermore , a framework that empowers emergency management personnel to reduce negative impacts of a CI DoS by strategically improving implemented security measures does not readily exist today (NACCHO 2017). Existing literature describe numerous approaches to CI i nterdependency analyses [Zio, Ouyang, Eusgeld, Haimes, DiMase, Borum, Kozik, Sikula, Rinaldi, DiGiorgio]. However, there is limited research on estimating the likelihood of negative impacts from those interdependencies or understanding the effectiveness of security measures designed for their protection. This research fills a gap by providing a framework that allows emergency management personnel to estimate the likelihood of impacts using a construct that dynamically (through scenario analysis) and proacti vely addresses and evaluates the relative effectiveness of implemented and/or proposed security measures designed to help minimize or negate undesirable effects of CI service disruptions. We expand on knowledge, experience, and recommendations offered by p reviously documented research of noted scholars. 1.4 RESEARCH OBJECTIVES The objective of this research is to develop a solution to address the stated research problem. Thus, a conceptual model that comprises a systematic, comprehensive (quantitative/quanti tative), scenario -based tool to proactively assess the effectiveness of implemented or proposed security measures is offered . Specifically, the objectives are outlined as follows : 13 1) develop a strategic methodology to assess the effectiveness of securit y implementations and aid in decision -making with a goal of proactively preparing for the inevitable occurrence of a CI service interruption/disruption. 2) model how successful penetrations of CI vulnerabilities can have life -threatening implications as an indirect or direct result of CI service disruptions ; 3) identify exemplar metrics that could effectively be used to warn, prevent, or absorb CI service interruptions . Research Questions and Hypotheses The following research questions are related to the problem statement and noted hypotheses. Question #1 : How can the assessment of security effectiveness of CI interdependencies and vulnerabilities be modeled proactively (before occurrence or penetration) for improved decision -making ? Question #2 : How can combining probabilistic reasoning and a holistic, systems -thinking based framework facilitate the assessment of relative effectiveness of CI and dependent resources security measures? Question #3 : What metrics are used by hospitals to trigger au xiliary systems in the event of a shutdown (partial or complete)? What metrics, if any are used to warn, prevent, or absorb CI service interruptions? Hypothesis 1a: Probabilistic reasoning and a systems -thinking based framework can be combined to assis t in the overall evaluation (strength or weakness) of CI and/or dependent resources security effectiveness. Hypothesis 1b: Probabilistic reasoning and a systems -thinking based framework can be combined to quantifiably evaluate security effectiveness of P rotection and Resiliency (P/R) for CIs and their dependent elements. 1.5 SCOPE AND LIMITATIONS This research forms the basis and foundation for future research. The value of this research will obtain its greatest return with the continuance of this research through studies in multi -order 14 dependency analysis of CI impacts towards the goal of improved security (protection and resilience) effectiveness. The bounds/limitations identified for this research include the examination of CIs as a black box. It is im portant to note that the same CI/DR -HHM analysis can be performed looking internal to the CI to assess the sub -components and the effects of any CI external influence it may cause. While excluding a detailed assessment of each CI, it is assert ed that the illustration and demonstration of the methodology and conceptual framework is not lost. Ten medical professional s (nurse, physician, or administrator) from ten different hospitals provided expert knowled ge based on the questions asked. T he ir responses are limited to their individual experience and training. Conversely, those respondents that participated in this research consisted of seasoned medical professional s appointed to speak on behalf of their hospital’s practices. 1.6 RESEARCH CONTRIBUTION This re search seeks to contribute to the body of knowledge within multiple disciplines, as follows: • Systems Engineering – provides a systematic, holistic, qualitative , and quantitative approach to CI protection that considers the elements of the greater system, t he element interactions, and their emergent properties . • Engineering Management – supports and informs decision -making using “what -if” scenario analyses within the model to allow for proactive planning and better allocation of resources based on security go als and implemented security measures . • Systems Security Engineering/Information Security – combines security and systems engineering best practices ; this construct enables a comprehensive perspective of the 15 system and its interfaces/ interdependencies to en courage better security in the appropriate areas based on identified security goals. • CI Protection/Resiliency – combines a framework for security professionals and decision - makers to perform scenario -based, quantitative , and qualitative analysis of CIs and dependent resources. Specifically, the novelty expressed in this research expands upon existing CI interdependency studies and analysis, while exploring a unique implementation of HHM and BBN through the proposition of a hybrid multi -dimensional framew ork that generates quantitative and qualitative results to assess security effectiveness. 1.7 SIGNIFICANCE /IMPLICATIONS OF RESEARCH The ability to quantify the potential impacts to a critical hospital patient using numeric and qualitative data can be instrumen tal for decision -makers seeking to preemptively execute necessary security measures in order to demonstrate emergency preparedness, given a CI service disruption. This conceptual framework allows CI owner/operators the capability to proactively assess thei r situational awareness or security posture through scenario analysis, strategically based on the organization’s security goals. For hospitals, maintaining public health by providing continuity of services and ultimately preserving human life is of the hi ghest priority; this is followed closely by the goal of maintaining the confidentiality and integrity of patient records and billing information. This methodology provides CI owners and dependent resources with a tool to assess the exogenous (external) an d endogenous (internal) n th order dependencies and impacts to ensure emergency preparedness through various scenarios or what -if analysis [i.e. assess how the degradation in power and/or water services may impact the ability of a hospital to provide 16 necess ary services to a patient to sustain life]. The approach also allows for an analysis of impact(s) between the CIs [i.e. how degradation in power may impact water resources and/or communications]. The source of these impacts can easily go undetermined witho ut multi -order interdependency risk analyses. Although this methodology and framework is demonstrated using a specific threat (Denial of Service) to assess a specific purpose (impact to public health) given various possible vulnerabilities (people, proce sses, tools, networks , or physical assets), the applications of this approach extend beyond what is demonstrated here. This study approach could aid decision -makers assessing various threats including the following: • Availability of critical services to ho spitals and patients; • Integrity of the services provided by medical staff and vendors (medical devices); • Confidentiality of patient records, billing , and pharmacy data. Furthermore, this research suggests that in order for a complex, interdependent syst em of CIs to effectively provide critical services to dependent resources, more effective and efficient standards need to be implemented. Standards that mandate CIs communicate (share information) across CIs and to those that depend on their services, in accordance with a security effectiveness taxonomy understood by the impacted community of stakeholders. The SSEI concept and construct offered in this research would allow CIs to communicate in a common language at a confidential or sensitive information level, if necessary. This would allow appropriate security measures to be considered from the perspective of their own internal security effectiveness evaluation, as well as the external interdependent sources security effectiveness levels/evaluation. As an example, during the stakeholder risk assessment, knowing the SSEI of other CIs as well as your own SSEI, would aid decision -makers in a more efficient allocation of resources to effectively 17 maintain their security goal/target at an acceptable level of protection with the appropriate resiliency, given a successful penetration or attack. Further, if a CI that is providing critical services has a SSEI of 0.65, a dependent resource may want to ensure that his/her internal/individual SSEI compensates for th e weakness of that CI, potentially with a more immediate failover system for power, or maintain a larger back -up water source or more robust filtration system. 1.8 DEFINITIONS OF KEY CONCEPTS The proposed construct provides guidance on evaluating the security effectiveness of the protective or resilient resources a CI operator/owner has chosen to implement - in a systematic, quantitative, and performance metric -based approach. The following paragraphs describe the essential elements of the combined framework and their relevance. 1.8.1 SECURITY EFFECTIVENESS FOR THE OPERATIONAL ENVIRONMENT Relative effectiveness is best defined in its operational environment. In this study, the operational environment is bounded by the components’ internal and external (interfaces) t o the CIs and dependent resources in question. The following definitions are important to note: Security - the extent to which security measures provide protections that detect, deter, neutralize , and mitigate potential threats, while also providing resi liency measures to resist, respond, recover, absorb , and adapt to availing threats. (DHS NIPP, 2013 ) Security Effectiveness - the degree to which security implementations provide adequate protective and resilient measures, allowing business operations to be maintained at an agreed upon level of service per the enterprise security goal. Risk Management Effectiveness – determined by “whether and how much risk was actually reduced or whether risk was acceptable…” (Hubbard, 2009) 18 Security effectiveness, as defined above, implies that implemented security measures should not impede, interrupt , or disturb critical operations of the enterprise, unless by design in order to protect systems or persons from active attack. Effective protection for one organizatio n or CI may not apply to another organization or CI. Measures applied for a specific threat may not be as effective for a different threat. The same paradigm applies when measuring in different operational environments and for different security goals. A more targeted solution considers what is relative or relational to the problem and specific influences to the overall system. As a result, this research addresses security effectiveness more appropriately as relative security effectiveness. Specifical ly, this research asserts relative security effectiveness is best achieved by first defining the operational environment, understanding associated dependencies, identifying the goal or target to be protected, and evaluating the problem with a specific thre at in mind. 1.8.2 HIERARCHICAL HOLOGRAPHIC MODELING (HHM ) HHM is one approach to multi -dimensional modeling (modeling from various/multiple perspectives). The philosophy of HHM is grounded in the fundamental principle that complex, large -scale systems such as CIs cannot be sufficiently appreciated or modeled in a planar or singular context. Haimes [1981] states: “The HHM approach (philosophy) recognizes that no single vision or perspective of a system is adequate to represent a system and its component parts. Instead, the HHM approach identifies and coordinates multiple, complementary decompositions of a complex system.” HHM was chosen for this study to incorporate societal, legislative, environmental, spatial , and other relevant dimensional perspectives tha t may contribute to the strength or weakness of security posture. A CI/DR -HMM, developed for this research, is defined here as the HHM generated 19 specifically for the purposes of evaluating CIs or dependent resources. The HHM categories and variables were extracted from a multitude of sources, to include the Department of Homeland Security NIPP. The HHM philosophy provides comprehensive, multi -dimensional insight into an otherwise hidden problem/solution space to measure security effectiveness. To demonstr ate the concept, weights are distributed equally among the five (5) categories of the HHM (threats, vulnerabilities, protection, resiliency, interdependencies), resulting in a sum of 20 percent for each hierarchical category – totaling 100 percent for the entire critical infrastructure. Additionally, each category is an aggregate of its components (i.e., protection includes detect, deter, neutralize and reduce.) See Figure 2. Figure 2. HHM for Critical Infrastructure or Depend ent Resources (CI/DR -HHM) 20 This risk -assessment consists of input gathered from multiple stakeholders, such as engineering, IT, and medical professionals. Each CI owner or decision -maker calculates their CI -HHM score as described in subsequent sections. This information is used in the evaluation of the overall SSEI. 1.8.3 BAYESIAN BELIEF NETWORK (BBN ) BBNs are graphical illustrations of probabilistic dependencies (links) between variables (nodes). The graph is a Directed Acyclic Graph (DAG) and the depende ncies are such that any node given its parents in the graph is independent of its non -descendants (Pearl, 1988 ). Like BBNs, attack graphs are visual representations of physical and/or logical access into and within an enterprise, network or CI. An attac k graph, as discussed in Frigault’s work [2014], can be represented as a DAG, coupled with conditional probability tables (CPT) to constitute the BBN. A thorough implementation of attack paths considers all paths that an attacker may exploit, both virtual ly and physically, to access the CI/enterprise, network , or system. To demonstrate the BASE m 2d concept the network was modeled at the highest CI nodal hierarchy (black box) and the attack paths are notionally identified via the DAG. BBNs employ the fund amental premise of the Bayes Theorem: 21 The stated probability of an event or hypothesis is conditional based on the available/known evidence in the relevant context. This condition can be made explicit by the notation P (H|E), which reads as "the probabili ty of event H given the evidence E." BBN is a me thod for understanding evidence in the context of previous knowledge or experience [Pearl 1988]. The utility of BBNs has become increasingly popular over the past decade in various fields of study to demon strate reliability, predictions, diagnosis, and decision analysis, among other uses. If it is accepted that prior knowledge has intrinsic value, there is basis for using BBN. In this study, a BBN is generated illustrating the dependencies and potential i mpacts of successful penetrations originating from CIs. The BBN is used here for its ability to account for uncertainty and limited available data of CI probability of attacks and interdependency/impact data. For this study, the proof of concept is demons trated by using historical and relative notional data (prior probabilities), while the unknown values are calculated through a Bayesian software simulation tool ( Netica v5.15) to infer the CI interdependency impacts to dependent resources. 1.8.4 SYSTEMS SECURITY EFFECTIVENESS INDEX (SSEI ) The SSEI is a calculated value, resulting from risk -assessment performed by stakeholders to understand (quantify) the relative security effectiveness and posture of CIs and/or dependent resources. This index is designed to allow owners/operators the ability to assess and communicate the strength and weakness of implemented and/or proposed security measures. The SSEI serves as an evaluation of the risk mitigation steps a CI or dependent resource has taken to protect against service disruptions. The BASE m 2d framework is the vehicle developed to apply the index. We show how an organization can use their self -evaluation of security effectiveness to estimate the multi -order impact(s) of a C I service disruption. 22 This research is intended to demonstrate how the SSEI (index) can be determined and used to understand and improve security effectiveness. It ranges from 0 to 1 (0 – 100%), with a low index indicating a weak or poor security ef fect iveness rating. Section 4.2.5 describes how the SSEI was constructed to demonstrate the concept of this research. The BASE m 2d conceptual framework is the vehicle developed to exercise the index. The SSEI serves as an evaluation of the risk mitigation ste ps a CI has taken to protect against DoS attacks or whatever threat is being assessed . For example, based on the measures taken in various areas of security, per the CI /DR HHM categories, a CI or DR would self -assess their overall security effectiveness ( SSEI) to determine their current posture or where they could improve. The BASE m 2d framework is provided for the CI owner to assess various scenarios given their current or objective SSEI. Alternatively, a n objective or threshold (minimum) SSEI can be ob tained from a trade analysis to determine the index required so as not to negatively impact the security goal (or have an impact of an acceptable level). A general scale was developed for this study and used to illustrate the SSEI concept. 1.9 ORGANIZATION AND OUTLINE This document consists of nine sections: Introduction; Literature Review; Research Method ology ; Hospital Case Study; Data Analysis and Results; Conclusions; Recommendations for Future Wo rk; References; and Appendices. The Introduct ion details the research background, motivation , hypotheses/questions , and the significance of the study. The Literature Review examines relevant studies on critical infrastructure protection and resiliency , implementation of BBNs and HHMs for CI analyses, and securit y effectiveness approaches . The Research M ethodology describes the use of survey research for studying existing CI interdependent patient -impact risk assessments performed at hospitals , data collections , and sources and methods . The Hospital Case Study ap plies the 23 proposed methodology and implementation of the BASE m2d construct. The Data Analysis and Results describe the statistical analysis of the data and results of the hypotheses testing . The Conclusion section detail s the findings of this study from t he perspective of the research questions and hypotheses . The Recommendations for Future Work describes potential research directions suggested to further this study. The References section contain a bibliography of the reso urces used throughout this resea rch . Finally, the Appendices provide supplemental material as a result of this research . 24 CHAPTER 2 - LITERATURE REVIEW The purpose of this chapter is to review, assess, synthesize , and critique existing literature with a goal of furthering the body of knowledge in the field of evaluating and improving CIP/R security effectiveness. To address the stated research problem in accordance with the established research goals, e xisting tools (models, techniques , and approaches) developed fo r the purpose of per forming risk analysis of CI interdependencies were review ed and compared against criteria colle cted from current literature. This chapter evaluates how current approaches assess CIP/R sec urity effectiveness, given the complex, interdependent nature of CIs. Ultimately, models were reviewed for their ability to assess multi -order (hidden) effects inherent in CI interdependencies combined with the capability to insert mitigation, scenario -bas ed modeling to potentially reduce vulnerabilities. While compiling th is lit erature review , it was necessary not only to express the gaps noted in existing literature but to clearly articulate the distinctions of my research objectives , while specifically stating how this study expands upon existing research. The hierarchica l models chosen for hybridization are stated up front , while justification for that selection is supported in subsequent section s that detail the review of related works. 2.1 REVIEW OF RELATED WORKS Review of literature from noted scholars (Ayyub, DiMase, B orum, Ryan, Di Giorgio, Pettigrew, Bayuk, Haimes, Satumtira, Ghorbani, Rinaldi) identified relevant attributes/criteria to effectively achieve the goal of generating a comprehensive framework, given complex adaptive systems, such as CIs: 25 1. Align/trace to s ecurity goals 2. Strategic/systems engineering approach 3. Performance based metrics 4. Quantitative and qualitative assessment 5. Scenario or what -if analysis for decision making 6. Accommodates uncertainty 7. Allows for limited data 8. Assess security effectiveness o f security measures 9. Extensible application 10. Assess indirect consequences/Interdependency analysis Although these scholars have acknowledged the criteria above as imperative components to model/provide/improve effective security - current models, methods , and techniques at most only incorporate two or three components. Thus, the ability to assess the effectiveness of security implementations on a hol istic level has been limited. 2.2 OVERVIEW OF CIP/R ANALYSIS Government agencies, private sectors and noted sch olars have done a thorough job identifying the importance of CIP (GAO, 2008 -20014; PPD21; Zimmerman, 2001; Rinaldi, 2001, 2004; Moteff, 2005; Huang et.al .) Al. , 2014; Cummings, 2014; Ezell, 2005; George, n.d. ; Richard , 2008). Others have gone further to n ote various ways to incorporate risk management in CIP analysis (Bensi, 2013; Haimes, 1995, Chittister, 2012; Kjølle, 2012 ). S ome have emphasized that without considering the interdependence of CIs, an analysis would be inadequate (Gheorghe 2005; Haimes, 2 004; Zimmerman, 2001; Santos, 2006; Zhang, 2011; Zio, 2013). Also noted is the fact that more quantitative methods should be developed, implemented, and accompanied by 26 qualitative analysis ( Kjølle, 2012; Ryan, 2005; Di Giorgio, 2012 ). It has been predomi nantly discussed by those in academia and the government that modeling and simulation techniques are effective in doing predictive, scenario -based analysis of CIP (Ouyang, 2014; P. Pederson, 2006; Di Giorgio, 2012 ). Additionally, there has been a recent s urge in adding resilience techniques based on the realization that techniques or tools will never fully protect CIs or dependent resources (Ouyang, 2012, 2014; Kahan, 2009; Vugrin, 2010, Little, 2013; PPD21, 2013; NIPP, 2012; Biringer, 2010). Each of th e aforementioned scholars acknowledge that more needs to be done to protect and maintain our nation ’s CIs; however, review of these related works primarily revealed that prevailing methodologies do not provide a systematic, comprehensive approach towards a ssessing and acquiring security effectiveness for CI protection and resiliency. In light of the heightened focus on CIP and evidence of repeated penetrations, standards, protocols, and procedures have been developed - only to provide a false sense of secu rity. This study expands on knowledge/experience and recommendations offered by previously documented research. The following graphic dep icts areas of existing research; while the area in grey denotes the gaps that, if filled would constitute a compreh ensive solution. This research target s the noted gaps. 27 Figure 3. Research Focus Areas Review of literature revealed both qualitative and quantitative methods are most effective when combined to aid in reducing the likelihood of the undesired consequences of a successful attack on a C I (Adar, 2005) . Thus, the concept of combining HHM and BBN were explored; two hierarchical models, one allowing for a qualitative analysis (HHM), while the other provides for a quantitative assess ment (BBN). The fundamental philosophy of HHM and the probabilistic backbone of BBN were combined, with the “tuning” mechanism of a Systems Security Effectiveness Index (SSEI), in an effort to develop a more comprehensive approach to ultimately evaluate a nd improve the effectiveness of security measures. The BASE m 2d conceptual framework is offered as a contribution to the CIP/R crusade. 28 In this chapter we discuss the comparative analysis performed to select models chosen for this research . Additionally , e xisting literature employing applications of HHM and BBN for CIP/R, and other methods, their uses , and individual limitations are reviewed. This literature review is organized as follows: Approaches to CI protection and resiliency; Approaches to Mea suring Security Effectiveness for CIs; Implementations of HHM for CIP/R; Implementations of BBN for CIP/R; Literature Review Summary. 2.3 APPROACHES TO CI PROTECTION & RESILIENCY (CIP/R) Successful attacks on CIs resulting in physical destruction have raise d growing concerns regarding the effectiveness of various techniques implemented for the purpose of providing protection. This research examined existing methods, techniques, and strategies used to evaluate the effectiveness of security measures for Criti cal Infrastructure Protection and Resiliency (CIP/R). Specifically, this study sought to understand if and how these methods considered or extended measures to protect dependent (2nd/3rd order) resources in a manner that is systematic, proactive , and holi stic , such that it assists in effective, efficient, and informed decision -making. As noted by many scholars researching CI protection, there does not exist today a silver bullet approach that completely protects and prevents the interruption or denial of CI services (Biringer et. al. , 2013). Instead , a more strategic and effective approach is needed, to include having the ability to provide resiliency to maintain service in accordance with security metrics and stakeholder goals. Biringer explain s that s ecurity systems must be designed relative to the specific security concerns of the infrastructure, the threat to the infrastructure, the security concerns of the infrastructure, and the protection goals of the security system. They further elaborate that each owner of the site, facility, or system must specify or describe the protection goals of its security system to allocate sufficient financial resources and labor to meet goals with a clear understanding 29 of the level of consequences that are acceptable if the protection goals cannot be met (Biringer et. al. , 2013). Similarly, this study asserts that within the design , the CI owner/operator must consider how a realized vulnerability of the infrastructure may affect the services provided by the infrastruc ture, ultimately i mpacting dependent resources. Over thirty -five models, techniques , and approaches were reviewed, wit h data sourced from Ouyang (2014 ), Eusgeld (2010 ), Idaho National Labs (Pederson, 2006) and Satumtira (2010), and Vugrin (2010), to as sess how each technique synergized the multi -dimensional, multi -objective, qualitative, stochastic, and hierarchical nature of CIs and dependent resources (CI/DR). Although each model was in various stages of maturity (R&D, Internal -only, operational), th ey were each designed for the purpose of analyzing CI interdependencies. Many were design ed for a specific CI (internal dependencies), others intended to manage cross -sector dependencies. It was unclear which tools were designed to evaluate multi -order e ffects (indirect consequences of negative events), which is of great interest to this study. In the models reviewed, resiliency was handled, at most from the perspective of redundancy. Many of the tools had the ability to perform sensitivity analysis and the ability to determine various “strength” of dependencies (i.e. which dependency had the greatest impact on another CI). Various decision analysis techniques were built in to determine or indicate priority and relative importance; however, very few of the models illustrated the ability to assess dimensional interdependencies (legislative, societal, economic, stakeholder, etc.). At most, a few models had the ability to incorporate temporal, spatial, and geographic data. Five models were highly regarded by the DHS: Athena, CARVER, Critical Infrastructure Modeling system (CIMS), Knowledge Display and Aggregation System (KDAS), and Maritime Security Risk Analysis Model (MSRAM). Each of the five models are considered Model -Based Risk Analysis (MBRA) tool s, known for their ability to aid in risk -informed decisions (Lewis, 2012). CARVER, 30 Athena, KDAS and MSRAM were each designed specifically for military and government entities, while CIMS targeted emergency planners and responders as end users. Of all the models evaluated, no tool clearly articulated how or if effectiveness was assessed; and the extent to which resiliency was addressed , it was limited to identifying redundant components/measures. No tool addressed resiliency as it is defined by the NIPP ( Ouyang, 2013). The ability to handle minimal data and account for uncertainty was only managed by tools with a stochastic engine, however, even those tools did not address effectiveness from the perspective of both protection and resiliency, nor from vario us dimensions, as previously described. Table 2. Literature Review CI Model Comparative Analysis Although unable to physically manipulate the models evaluated for this research, the data available served well in filtering vario us capabilities and limitations of each tool. While each tool appeared to serve a valuable fit for its purpose, it did not appear evident that they lent themselves to trivial modification for extensibility to incorporate additional/lacking features. A to ol is most valuable and effective when designed from its core to allow for modular growth that enhances or provides Criteria/Features BASEm 2d Bologna NIPP PCCIP PPD63 Briere Rinaldi Peerenboom Haimes Zimmerman Mendonca Osorio Bensi Haimes AIMS Athena CARVER CI3 CIMS CIP/DSS CIPMA CISIA DEW EMCAS FAIT FINSIM Fort Future IEISS IIM KDAS KMV MIN MSRAM MUNICIPAL N-ABLE NEMO Net-Centric GIS NEXUS Fusion FrameworkTM Ngtools NSRAM PFNAM TRAGIS TRANSIM WISE Conceptual Modeling and/or S imulation Tools Applied 1Align/trace to security goals x x x x x x 2Strategic approach x x x x x x x x x 3Performance-based metrics x x x x 4Quantitative and qualitative x x x 5Scenario or what-if analysis x x x 6Accommodates uncertainty x x x x x 7Allows for limited data x 8Assess security effectiveness x x x x 9Extensible application x x x 10 Interdependency Analysis x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 11 Protection x x x x x x x 12 Resiliency x x x x x x Hybrid 31 additional capability, vice adding on to a tool that was not built to be dynamically modified. Consequently, I sought to establish the BASE m2d conceptual framework, a tool that asserts to provide what existing tools lack, a comprehensive approach to assess security effectiveness of measures proposed/implemented for CI protection and resiliency; while also providing modularity for future impro vements and/or additional functionality. 2.3.1 IMPLEMENTATIONS OF DECISION ANALYSIS TOOLS FOR CIP/R HHM has been used extensively in assessing and identifying sources of risk, complex interdependencies of water resource CIs (Chittister e.t al. 2012; Haimes 1995) , and Supervisory Control and Data Acquisition (SCADA) networks (Haimes 2005). However , this research will be the first implementation of HHM to assist in the identification of strength or weakness weights/index, coupled with using BBN analysis to ass ess and improve security effectiveness of CI protections and resiliency. Haimes originally designed HHM to identify sources of risk. Although modified in this study from its original design, I maintain and leverage the integrity of the HHM philosophy a nd concept for its ability to not only identify sources of risk , but to assist in the identification of relevant variables hierarchically, in various categories , and from multiple dimensions. Approaches such as Analytical Hierarchy Process (AHP), Multi -Attribute Utility Theory (MAUT) , or Multi -Criteria Decision Analysis were evaluated as comparable models; and although each of these methods also assist in decision making and allow for a structured way of framing the problem, the need to assign relevant wei ghts to each criteria to show importance was not necessary to meet the objectives of this conceptual study. Instead, to demonstrate the preliminary concept, variables were elicited from existing research previously collected from experts at the DHS and do cumented in the NIPP. CI experts determined that the variables identified within the NIPP held equal weight at the highest 32 level of evaluation. In an effort to scope individual components of this framework I leverage (extend) vetted research to focus thi s research on demonstrating the comprehensive framework. This research acknowledges that criterion weights are a reality (all variables may not be equally important), hence they may vary depending on budget constraints and available resources. Variables should be re -evaluated/weighted on a case by case basis, which would suggest the use of the aforementioned decision analysis models such as AHP and others. This is recommended as a future enhancement to the BASE m 2d preliminary framework. The HHM compone nt of this tool is a modular component which can be modified or replaced as the user desires. 2.3.2 IMPLEMENTATIONS OF PROBABILISTIC RISK MODELING FOR CIP/R Graphical probabilistic models such as Markov Random Fields (Markov Networks) and Bayesian Belief Network s were explored to address the uncertainty of complex system interdependencies and the very real occurrence of limited data; specifically, a probabilistic model that allows for scenario/what -if analysis, with a user -friendly interface, and one that does no t require great statistical knowledge. Markov Networks are known for their power and flexibility (undirected, allowing cycles); BBNs are considered to be restricted by comparison (directed, acyclic). It was discovered that both Markov Networks and BBNs w ould suffice for this study, however, BBN was chosen simply based on its immediate capabilities, the researcher’s familiarity with the method, and its ease of use. A review of existing literature indicates BBNs have been implemented to model and analyze the interdependencies of CI (Di Giorgio, 2011 ). BBNs have also been used to predict the likelihood of terrorist attacks on CIs (Johan 2009; Haimes 2004 ). Kazak (2010 ) uses BBN with a security ontology to assess the severity level of detected threats, wh ile Faribault (2014 ) demonstrates the utility of BBN to assess vulnerability in computer networks. Additionally, Queiroz (2013 ), et. al . use 33 BBN to evaluate information diversity within SCADA systems . There is also extensive research available implementi ng BBNs to provide earthquake decision -support systems for seismic infrastructure risk assessment, to include Bensi (2010 ), Bayraktarli (2005 ), and Kheun (2009 ). In the medical field, BBNs are currently applied to assist in more accurate diagnoses (Forsbe rg 2011 ), to predict the occurrence of cancers (Burnside, n.d. ), and to estimate patient survival given the presence of certain cancer prognostic factors (Forsberg 2012 ). Although it was determined through literature review that BBN is effective and most appropriate to evaluate CI vulnerabilities and medical prognosis/diagnosis, gaps exist in its implementation as a catalyst for stimulating decision analysis for the purpose of assessing and quantifying security effectiveness toward improved CI protection and resiliency. Further, no comprehensive approach that generates an effectiveness index based on existing implemented security measures for the purpose of further improving security posture was found. As previously discussed, various CI interdependency analyses/approaches have been extensively explored by scholars such as Rinaldi, Haimes, Di Giorgio, Zhi -yan , Macaulay , and others. However, this research extends upon that research with provisions for a holistic framework to address relative security effectiveness, given CI interdependencie s, to make informed decisions. It is acknowledged that even the best efforts will not afford absolute protection, thus the need to simultaneously prepare for resiliency (i.e., adaptability, recoverability, absorption, etc.) of a cyber or physical attack. The BASE m 2d framework is aimed at providing both proactive and responsive measures toward better P/R. Subsequent sections of this paper detail the methodology and the combined implementation of the associated models (HHM and BBN). 34 2.4 APPROACHES TO MEASURI NG SECURITY EFFECTIVENESS FOR CI Methods used today to evaluate security effectiveness include variations on vulnerability analysis, penetration testing, threat analysis , and/or risk analysis. It would be expect ed , at a minimum, for organizations to perfo rm some aspect of each of these activities before allocating resources to improve their security effectiveness. Conversely, review of literature has revealed that those employing any one of the aforementioned analyses, most often do not necessarily do so in a systematic, holistic manner (Biringer, 2013; Pettigrew , 2009; Ryan , 2008. ). For example, the assessment of various threats/vulnerabilities from the perspective of various stakeholders predicated on the organization’s security goals, using qualitative or quantitative performance -based metrics to determine the effectiveness of the protection and/or resilience measures they have in place. Additionally, until recently, those efforts that focused on vulnerabilities, pen etration testing, threats , and risk did so only as it related to protection. More recently, there has been an exertion toward the realm of implementing practices to ensure resilience. Currently, t he review of related works primarily identifies that existing methodologies do not provide an approach toward assessing and acquiring security effectiveness for CI protection and resiliency. 2.5 SUMMARY In Chapter 2 the problem currently faced by the nation to protect and maintain CIs from successful cyber or physical attacks was discussed . Severa l models were reviewed to evaluate their ability to address the multi -dimensional, multi -objective, quantitative/qualitative, and hierarchical nature of CIs and dependent resources. Of the models reviewed , no tool clearly articulated how or if effectivene ss was assessed; and the extent to which resiliency was addressed was limited to identifying redundant components/measures. The ability to handle minimal data and account for 35 uncertainty was only managed by tools with a stochastic engine, however, even th ose tools did not address effectiveness from the perspective of both protection and resiliency, nor from various dimensions, as previously described. Consequently, this research implemented the integration (hybrid) of two hierarchical techniques that encom pass each of the ten (10) aforementioned criteria. It was hypothesized that by combining multidimensional modeling such as HHM and BBN, a conceptual framework could be developed to enable scenario -based analysis to assess the effectiveness of implemented or proposed security measures. Chapter 2 also discussed a detailed overview of existing techniques and models used today to provide protection and resiliency to CIs and those that depend on its services. As previously discussed, various CI interdepende ncy analyses/approaches have been extensively explored by scholars such as Bloomfield, Eusgeld, Haimes, Min, Ouyang, Kjolle, Liberati, Satumtira, Zimmerman, Rinaldi, Haimes, Di Giorgio, Zhi -yan , Macaulay , and others. However, this research extends upon that research with provisions for a met hodology and framework to address relative security effectiveness, given CI interdependencies, to make informed decisions. CHAPTER 3 - RESEARCH METHODOLOGY This chapter describes the research methodology chosen to investigate how two hierarchical mod eling techniques can be combined to provide a quantitative and qualitative conceptual framework (BASE m 2d) that can be used to assess and improve the relative security effectiveness of CIs and dependent resources. Described herein, is the approach taken t o develop, test , and validate the holistic framework. The research method chosen consist s of a survey designed to elicit probability distributions from experts, which are subsequently employed to validate the BASE 36 m2d framework. To test the hypotheses, a hospital case study was designed to demonstrate the frameworks’ utility, flexibility , and limitations. 3.1 RESEARCH DESIGN This study utilized a focused cross -sectional survey to collect data from medical professionals employed at 10 distinct hospitals. Th e survey was used to capture the knowledge and experience of medical professionals regarding the use of metrics implemented by their respective hospital facility to measure effectiveness and/or emergency preparedness. Data was also collected to understand various impacts to a critically ill hospital patient (dependent resource) given a CI failure or Denial of Service (DoS). Survey responses were collected via a web -based, electronic tool (Survey Monkey). Using the expert elicitation method, the data was f urther used to validate the BBN model and expected results. Elicitation of specific data from medical experts was limited due to the acknowledged vulnerability of hospitals and potential insight it may provide to adversaries.

Thus, we use a combination of real and notional data and present this research as a proof of concept. A survey was used to evaluate various impacts to a hospital patient given a DoS to a CI and to understand the knowledge of medical professionals regarding metrics implemented by the ir employment facility (hospital). Surveys are often used to gather statistical data about demographics, actions, techniques, perceived effectiveness, characteristics or attributes from a selected or random community , or category of a specific population (Babbie, 2010; Creswell, 2009; Salkind, 2009). Descriptive and inferential analysis was performed on the data collected using Netica, SPSS , and Minitab . The research survey process for this study maintained the strictest confidence of respondent inform ation and identities, as required by the GWU Institutional Review Board (IRB). No personally 37 identifiable information (PII) was acquired, and all metadata associated with each respondent was securely discarded. 3.2 EXPERT ELICITATION Expert elicitation (EE) is often used when it is not feasible to collect empirical data for statis tical analysis ( Cooke 1991) . According to Ryan, et. al., EE is designed to elicit, codify, and combine the knowledge of people who have significant experience or expertise in a defined field in order to assess unknown quantities or parameters (Ryan 2012). These scholars go on to state that the use of “expert judgment is justified when quantitative data is missing, of dubious quality, or is insufficient for obtaining reasonable statistical results .” Various methods are noted in scholarly research of how to elicit, codify, and combine expert knowledge, as described by Cooke (199 1). For this research effort, method s detailed by Buede , Renooij (2001) , and Pitchforth (2011) was used based on its extensive application in Bayesian analysis . Bayesian Networks are often created through the process of EE (Pitchforth 2012). Studies have shown that experts tend to be overconfident about their judgments (Tversky 1974, Lin 2008, Flandoli 2011). Heuristics and biases are issues that often arise when eliciting information from experts. To overcome these relevant concerns, scholars recommend a well - structured elicitation process. According to Kahneman et. al. , (1982) and Renooij (2001), bi as is a systematic tendency to take into account factors that are irrelevant to the task at hand, or to ignore relevant facts, thereby failing to make an inference that any appropriate normative theory, for example probability theory, would classify as nec essary. There are two types of biases to consider: motivational bias , which is caused by personal interest or circumstances experienced by the expert; and cognitive bias which a rise during the processing of expert information with the use of heuristics ( availability, anchoring, representativeness , and control ) (Kahneman et. al. , 1982). 38 Motivational bias can be mitigated by reassuring the respondents that their responses are for information only and not a promise or commitment. Cognitive bias can be mini mized by letting the respondent know that they exist and by subjecting the experts to calibration (Hubbard, 2016) . The survey should consider these concern s in deciding upon the survey method used (Renooij 2001). This research has taken specific steps to minimize bias , inconsistencies, potential errors , and overconfidence. The following process was used to elicit relevant data from respondents: Step 1: Expert Selection Experts were selected by various representatives of each hospital solicited for data on this study (Hospital Risk Assessment Office, Administrator of hospital, Chief Information Technology Officer, etc.) . Each respondent was selected based on their domain knowledge of hospital operations and their ability to assess patient impacts. The respondents were advised that their individual participation would remain anonymous and data provided would be voluntary, with no obligations or retribution to their employment. They were also assured that the accuracy of their information need only be ba sed on their experience (Renooij 2001) . Step 2: Set Foundation for the Expert During initial contact, the expert was advised of the purpose of the study, to include background information, definitions , and scope. Expectations were detailed and the confid entiality of personal information was emphasized. Special care was taken to ensure that the same contextual information was provided to each participant in a common manner so as not to introduce bias in judgment ( Boring, 2005) . 39 Step 3: Train the Expert Participants were subjected to a n electronic survey which included an introduction and described the purpose of the study. The introduction described the basic process and flow of the survey , and each section provided necessary instructions for completion . The goal of this activity was to ensure the expert felt comfortable with the process and understood with clarity the objective of the survey instrument. The multiple choice, five -point Likert scale questions eased the burden of the respo ndents who may have not been com fortable with providing direct probability distributions. Step 4: Elicitation of Judgments Responses were collected via a web -based, electronic tool (Survey Monkey). Experts were asked to provide minimal demographic information (i.e. yea rs of experience, medical profession, and experience in ICU or other emergency patient care). Expert responses were calibrated based on absolute agreement in accordance with Intra -class Correlation Coefficient (ICC). Probability distributions were indirec tly elicited from each expert, capturing the likelihood CI service disruption impacts to patients. The elicited data was used to validate the behavior of the BBN model. Step 5: Aggregating the data for Consistency Upon completion of the survey , verificat ion consisted of checking whether the elicited probabilities are coherent, obey the laws of probability, and are reliable (Fenton, 1998). According to Renooij, an indication of the validity of the assessments can also be obtained by entering observations into the belief network and computing the effect of the observations on the probabilities for certain variables of interest. The outcomes for these variables can then be checked against available data or presented to the expert (Renooij 2001). 40 3.3 SURVEY SCALE The scale used for this survey instrument consisted of a five -point Likert scale. The questions were rank ordered from lowest (very unlikely) to highest (very lik ely). As noted by Fowler, Hayes, Nunnally, Punch, Weisberg, a graduated scale will maximize the degree of variability in the responses and lend itself to an analysis of continuous variable data ( Fowler, 1995; Hayes, 1998; Nunnally, 1978; Punch, 2003; Weisberg, 1977). The chances of misclassifying the data due to reverse coding will be significa ntly reduced with rank ordered responses (Punch, 2003; Weisberg, 1977). Each rank order corresponded /coded to a probability distribution as follows: 1 2 3 4 5 Very unlikely Unlikely Need More Info Likely Very Likely (0-20% ) (21 -40% ) (41 -60% ) (61 -80% ) (81 -100% ) 3.4 EXPERT ELICITATION CALIBRATION Various methods exist on how to elicit, structure, combine, and calibrate an expert ’s response to a survey (Ryan 2012; Flandoli 2011; and Cooke 1991) . Cooke’s Classical Method (1991) suggest s weighting expert’ s opinions and experience with “seed questions ,” to which the answers are generally available and typically known by subject matter experts in the field to be studied. How the experts respond are scored in accordance with the true answers and consequently weighted. The weights are derived from a combination of the expert response to the seed questions and are used to calibrate the “accuracy” of the experts’ opinion. Calibration measures the statistical likelihood that a set of experimental results correspo nds with the experts’ assessments (Cooke 2004) . Alternatively, Hubbard (2016) discusses how he facilitates workshops to “c alibrate the expert” to improve one’s ability to subjectively assess odds. He teaches various methods to 41 reduce overconfidence and u nderconfidence in responses – know n challenges to eliciting expert judgment. Expert responses used in this study were not calibrated as offered by Cooke , given no seed questions were offered within the survey to calculate weights . To account for this , we attempted to take extreme care during the expert training period , as preferred by Hubbard – calibrate the expert . The probability distribution s collected from experts were used to confirm BBN validation , i.e. , does the model behave as it should; are the s imulated results as expected. This overall elicitation approach was taken with caution and considered a sound way to minimize biases and under/ overconfidence . The following considerations were taken into account to proceed with this method, when no oth er data exist (Johnson 2010) : • Detailed training of experts prior to the s urvey • Clear, standardized instruction script • Provided Likert scale with corresponding probability distributions • Avoiding use of scenarios or anchoring data • Allow for comments or feedb ack 3.5 SURVEY INSTRUMENT This research leveraged the flexibility and convenience of a professional online tool (Survey Monkey) used to collect data, extract, and compile the final results. Survey Monkey is a convenient, web -based tool, available 24/7 to resp ondents during the data collection period. Although many researchers find several advantages to using the online survey tool, such as being cost efficient and providing quick results. Survey Monkey also allows for various design methods to present/displa y questions and collect responses, it has a global (electronic) reach to extend beyond places a researcher could physically be present, and it also tends to have a higher 42 response rate than other popular methods. Detailed benefits are described below: Ben efits of Electronic/Online Survey: http://writing.colostate.edu/guides/page.cfm?pageid=1406&guideid=68 • Cost -savings: It is less expensive to send questionnaires online than to pay for postage or for interviewers. • Ease of Editing/Analysis: It is easier to make changes to questionnaire and to copy and sort data. • Faster Transmission Time: Questionnaires can be delivered to recipients in seconds, rather than in days as with traditional mail. • Easy Use of Preletters: You may send invitations and receive respons es in a very short time and thus receive participation level estimates. • Higher Response Rate: Research shows that response rates on private networks are higher with electronic surveys than with paper surveys or interviews. • More Candid Responses: Research s hows that respondents may answer more honestly with electronic surveys than with paper surveys or interviews. • Potentially Quicker Response Time with Wider Magnitude of Coverage: Due to the speed of online networks, participants can answer in minutes or hou rs, and coverage can be global. • Same strength as written survey • Ability to consistently track responses • Automatic randomization of questions and answers choices to remove potential biases Drawbacks to Electronic/Online Survey: • Sample Demographic Limitati ons: Population and sample limited to those with access to computer and online network. • Additional Orientation/Instructions: More instruction and orientation to the computer online systems may be necessary for respondents to complete the questionnaire. • Pot ential Technical Problems with Hardware and Software: Computers have a much greater likelihood of "glitches" than oral or written forms of communication. • Potential for SPAM 43 • Potential for survey fatigue The survey questions were designed to be simple, stra ight forward, and elicit an intuitively positive response (Punch, 2003; Weisberg, 1977). The questions were also designed to be without negative statements, which have been known to confuse respondents and lead to non -normal, diverged, and skewed distribut ions and result in weaker statistical correlations (Hayes, 1998). 3.6 VALIDITY OF SURVEY INSTRUMENT Purpose : To collect relevant data for the purpose of understanding how hospitals assess, measure , and/or utilize the following: • Hospitals utilization/impleme ntation of metrics to trigger alternate critical systems for water, power , and communications • Likelihood of Impact to hospital and/or critically ill patients, given a limited or complete service disruption of power, water , or communications to the areas of cardiac care, dialysis , and oxygen/ventilator – services to which the critically ill patient is dependent. Approach : The results of the survey were used to validate the findings of the conceptual framework. To determine if the results of the survey inst rument are valid and reliable , the respondent data was evaluated against the data produced by the tool. If an instrument is unreliable, it is also invalid, because accurate findings cannot be obtained with inconsistent data (Carmines, 1979). A valid surv ey instrument serves the purpose it is intended to serve and provides correct information (Fink, 2003). This survey instrument was evaluated based on the following criteria/test s: 44 RELIABILITY ▪ INTERNAL CONSISTENCY or Homogeneity using a Cronbach coeffic ient (α >0.7) o Measures the extent to which a technique, experiment , or measuring procedure assess the same characteristic or quality . ▪ INTER - AND INTRARATER RELIABILITY o Measures the extent to which multiple respondents agree in their ratings of given items. VALI DITY ▪ CONTENT VALIDITY o Measures the extent to which the question/items thoroughly and appropriately assess the characteristics or qualities it purports to measures. ▪ FACE VALIDITY o Measures the appearance of the metric/question on the surface. 3.7 VALIDITY OF CONCEPTUAL FRAMEWORK Purpose : The objective of the conceptual framework is to provide relevant information for the purposes of providing proactive protection and resilience to CIs and dependent resources. In doing so, the validity and reliability of the c onceptual framework was evaluated. Specifically, the following approach was used to measure the tools’ effectiveness for its purpose , and its ability to perform as a hybrid model to measure and assess security effectiveness of CIs and dependent resources . Approach : Previous research performed by the DHS identified relevant variables used to measure effectiveness for CI protection and resiliency. Th is re search was furthered by using those variables within the conceptual framework (BASE m 2d) to demonstrat e the ability to quantitatively 45 and qualitatively measure and assess security effectiveness of CIs and dependent resources. Further, the data collected from medical professionals was used to validate the results of the hybrid conceptual model. 3.8 DATA COLLEC TION Data was collected and provided from various sources as described herein. Ten medical professionals from 10 distinct hospitals provided data through expert elicitation, to validate the BBN model and expected results. DHS and various sources discover ed through research provided the relevant variables used for the SSEI and HHM. The specific CI categories and hierarchical relationships used in this study were extracted from the Department of Homeland Security (DHS) National Infrastructure Protection Pl an (NIPP) and their associated Sector Specific Plans (SSP). Historical and relational data was used as prior probabilities for the BBN analysis from sources such as the medical professionals, DHS , Verizon 2013, 2014 , and 2015 Breach Reports, Symantec Annu al Security Threat Report , and Verisign. Specific data for each CI was collected from government sources such as DHS in collaboration with the Department of Energy (DOE), Water Waste and Sewage (Environmental Protection Agency) , and the Communications sec tor (National Communications Systems). Previous research performed by the DHS identified relevant variables for CI protection and resiliency (CIP/R). Patient impact data was collected per the Institutional Review Board (IRB) guidelines, ensuring no person ally identifiable information was acquired, attributed, or misused. Survey Monkey, a web based renowned vendor often used in academic research, was employed during this study as the survey instrument to facilitate the collection of patient impact data fro m medical professionals. Results of the survey were used to validate the BBN model and overall conceptual framework. 46 CHAPTER 4 – HOSPITAL CASE STUDY 4.1 CASE STUDY BACKGROUND Public trust depends upon the sustainability, resilience, integrity , and availabil ity of national Healthcare and Public Health (HPH) critical infrastructure [NACCHO 2014]. Continuity of healthcare and public health services are critical to response and recovery following a disaster or emergency [NACCHO 2016]. Results from this research reveal that hospital engineers have performed due diligence to ensure if power is disrupted or water supplies have been tainted or halted, backup generators and alternate water supplies are available to maintain critical services. However, this research f urther revealed that most hospitals have not performed additional risk assessments to evaluate implemented security measures to allow them to understand, evaluate , and reduce the potential impact of a CI DoS (e.g., water, power) to a patient or medical dev ice. Understanding how and where to properly address and allocate security measures in a budget -constrained environment will prove invaluable to ensure these critical resources/services are uninterrupted (or have minimal or acceptable impact). This case s tudy models a critically ill patient in ICU who depends on medical equipment (dialysis, defibrillator, etc.) for survival. Given a DoS to a CI (power, water, etc.), the hospital could be subsequently impacted if the proper back -up resources are not engage d or available in a timely manner Multi -order, cascading effects are important to understand given they can occur as a result of a direct or indirect attack or occurrence. A CI failure or DoS (resulting in a partial shutdown or complete shutdown) can be du e to a successful attack, whether virtual or physical, intentional or unintentional, potentially having a 2 nd, 3 rd, or 4 th order effect [Figure 2]. The case study modeled in this study examines the 4 th order effect to a patient that is depending on CI serv ices to sustain life. 47 Figure 2. CI Interdependency Multi -Order Effects 4.2 CASE STUDY APPLICATION The Bayesian Approach to Security Effectiveness with metrics, modeling and decision -support (BASE m 2d) framework uses a hospital case study to demon strate the overall methodology. Described herein are the steps to perform a risk -assessment using the CI/DR -HHM, calculate the SSEI , and perform “what -if” analyses to determine the potential impact to a dependent resource (hospital or patient) given a suc cessful DoS to one or more CIs (water, power, communications). Figure 4 represents the general flow of the BASE m 2d framework. 48 Figure 4. General Flow of BASE m2d Framework The following sections describe the ten -step methodo logy of the BASE m 2d framework. The first three steps define the problem -solution space; Steps 4 -7 assess the current security posture using metrics, the operational environment from multiple stakeholder perspectives using HHM, the overall relative effect iveness using SSEI, and the likelihood of impact using BBN; while Steps 8 -10 estimate the relative risk, and describe the iterative decision analysis process. 49 4.2.1 STEP 1 AND 2: DEFINE OPERATIONAL ENVIRONMENT AND SECURITY GOALS The strategic implementatio n of this framework begins by establishing security goals within the operational environment. The Department of Homeland Security National Infrastructure Protection Plan (NIPP) has developed value propositions and/or security goals for each CI identified in the plan. Establishing security goals includes documenting what is important to the organization and considers perspectives of all stakeholders, to include CI owners/operators, dependent CI owners/operators, vendors, consumers/customers, etc. Many deci sion -makers focus resources in areas that have little to no impact on what they value, and without the full consideration of the operational impact; thus, leading to security compliance with little security effectiveness. Once goals are established within the operational environment, measures can be directly implemented to monitor progress toward achieving those goals in accordance with the value proposition. HPH Security Goal : For the HPH sector or hospitals, it is crucial that they maintain (resilien ce) a certain level of business continuity to preserve human life and to protect the confidentiality of information for their patients and personnel. The HPH security goal identified above can be divided into two separate targets to protect: human life a nd patient information. This case study focuses on preserving human life to demonstrate the capabilities of the model; however, the model is extensible to include an analysis of the impact of a successful attack on patients’ personal information, as well. That case is excluded here based on the threat (DoS) in question, which is often employed for purposes other than to exploit personal data. 50 4.2.2 STEP 3: IDENTIFY DEPENDENCIES Internal and external dependencies (also known as potential attack paths) to the p rotection target or security goal (hospital, patient), within the operational environment can indicate vulnerabilities. Specifically, if there is a penetrable entry or exit point (link) to or from the target, strategic consideration should be given to app ly appropriate measures of protection or resiliency to prevent undesired effects in the event of an attack or natural disaster. Links/dependencies may be identified as having virtual and/or physical access and should be prioritized, especially in a budget constrained environment. Identifying dependencies between CIs (water, power, communications) and dependent resources (hospital, medical devices, patient) provides the topology to construct the general Bayesian Belief Network. THE METRICS 4.2.3 STEP 4: ASSESS /M EASURE THE SECURITY POSTURE Relevant metrics should be identified and implemented to assess the current security posture of the organization, particularly metrics that acknowledge and indicate internal/external degradation of the infrastructure. Metrics chosen should pass the “so -what” test and should be selected in relation to the security goal within its operational environment. They should be identified and defined around the problem space. Without metrics, organizations will find it difficult to accu rately gauge effectiveness and articulate improvement. When metrics can be quantified as a number or percentage, are contextually relevant, and measured consistently, they confer credibility to the overall assessment (Jaquith, 2007). Jaquith goes on to st ate that good metrics should facilitate discussion, insight, and analysis. 51 Security metrics are the servants of risk management, and risk management is about making decisions. Therefore, the only security metrics we are interested in are those that sup port decision making about risk for the purpose of managing that risk (Jaquith, 2007) Getting the right metrics requires asking the right questions. Cyber -attacks may cause a temporary disruption (minutes to hours), while natural disasters such as tornado es or hurricanes may cause long -term outages (weeks to months). A question such as “are failover settings for the backup generator sufficient to not have a negative effect on the patient, in the event that power is disrupted at the hospital’s main plant,” should lead to identifying the appropriate metrics that ensure the ability to proactively monitor status and plan accordingly. Medical devices dependent on CI services would benefit from metrics such as mean time between failure (MTBF) for monitoring or alarm if a failover system (power, water) exceeds a certain value. Asking how long a backup generator can support a critically ill patient surviving on a ventilator before asphyxiation or brain damage occurs should also result in relevant metrics. Although the answer to this question may depend on the severity of the patient’s illness, the reliability and sustainability of the medical equipment should be measured, understood , and baselined accordingly, to assist in proper planning. Penetration testing shou ld also be performed and results included in the overall assessment to continuously monitor and capture anomalies due to unsolicited or unintentional physical or cyber access. These combined techniques provide an understanding of the current security post ure and insight into the strength or weakness of the enterprise. The BBN model developed for this research is structured such that if current measures indicate an unacceptable impact to the patient, additional or alternative security measures should be i mplemented and assessed. Threshold and objective parameters should be considered for 52 each metric (there may be cases where threshold parameters may be sufficient). Identifying these parameters and monitoring for trends or outliers will allow emergency per sonnel to take appropriate actions prior to an undesirable event. Such metrics would be useful to ensure resiliency. Step 4 results in the identification and assessment of relevant performance metrics by stakeholders that further assist in understanding a nd monitoring the health of the enterprise. Below, we provide an example set of performance metrics to be used for the protection and resilience of CIs or dependent resources is provided in Tables 3 and 4. Table 3. Reference Me trics for Critical Infrastructure/Dependent Resource Protection 53 Table 4. Reference Metrics for Critical Infrastructure/Dependent Resource Resilience 4.2.4 STEP 5: ASSESS MULTIPLE DIMEN SIONS /PERSPECTIVES [CI -HHM] [Steps 5 and 6 ar e closely linked. Step 5 details the risk -assessment process using HHM which is required to calculate the Systems Security Effectiveness Index (SSEI), discussed in Step 6.] This extensible CI/DR -HHM is provided with hierarchical and dimensional categori es to allow the CI/DR owner/operator to perform a risk -assessment of his/her infrastructure. Each category (threats, vulnerability, protection, resiliency , and interdependency) is used by the stakeholders to assess the security measures currently in place or proposed. Each stakeholder responds to a series of the same questions from the perspective of their own discipline (e.g.

engineer, IT specialist, doctor/nurse, hospital administrator, etc.). This includes a combined stakeholder threat and vulnerability analyses, for example, to determine the extent to which each discipline effectively implements measures to reduce vulnerabilities, employ s threat modeling, plan s for protection and resiliency, and considers interdependencies of other CIs and relevant reso urces. The strength (or weakness) of security measures employed by an organization is 54 calculated based on a combined stakeholder risk -assessment of each CI/DR -HMM category. The quantified results identify areas of deficiency, thus implying areas where impr ovements can be made. The CI owner or dependent resource facilitator should evaluate (weight and priority) each CI/DR -HHM category per their risk tolerance and goals. Figure 2 illustrates the third order (hierarchical) graphical representation of the CI/D R-HHM, with general weights uniformly distributed among its five categories. Adjacent to each sub - category is the max weight a CI/DR owner/operator would self -assess the effectiveness of the security measures they have in place. The decision -maker combine s the results of the risk - assessment completed by multiple stakeholders (internal and external), such as risk managers, engineering, IT , and medical professionals to calculate the SSEI, described in Step 6. Figure 2 . HHM for Critical Infrastructure or Dependent Resources (CI/DR -HHM) 55 Using the CI/DR -HHM categories provided in Figure 2 , relevant questions should be devised by the organization and each question should have a measurable component for monitoring and improvement. Qualitative responses to questions such as “do we have measures in place to protect (blank)” (yes/no); “if so, what are they and are they sufficient?” Table 5 provides a general CI/DR -HHM scale to score the risk -assessments for each category. A general SSEI scale was developed fo r this study to illustrate the concept. [Table 5 and 6 ]. Table 5. CI/DR -HHM SSEI Scoring Scale (by Category) Categories are identified as the main variables within the CI -HHM (threats, vulnerabilities, protections, resilience , and interdependencies). Each of the five CI subset categories can have a maximum value of 20 percent (0.20). A CI or enterprise can have a strength/weakness value per category in the range from 0 – 20 percent (0 – 0.20) as shown in Table 5. This weight or value implies that the enterprise has implemented security measures (for that category) to a certain level of effectiveness in accordance with their value proposition or goals. The results from the CI/DR -HHM category assessment ultimately contribute to th e overall SSEI “score” [Table 6] or level of the strength/weakness of an organization’s security posture. Subsequently, the SSEI is used within the BBN analysis to estimate impact. This step results in an understanding of specific areas of strength and we akness of the enterprise and facilitates the ability to better allocate resources to security measures identified as deficient so as to reduce negative consequences. 56 4.2.5 STEP 6: ASSESS STRENGTH /W EAKNESS [C ALCULATE THE SSEI] The SSEI is a calculated value that indicates/quantifies the risk management steps taken to reduce the probability of a successful attack. Emergency management personnel or decision -makers are to assess their effectiveness index based on the degree to which they implement measures and red uce risk identified in the CI/DR -HHM categories [Figure 3]. Security measures should include people, process, technology, e.g., “what, if any, security measures do we have in place, in the form of people, processes , and technology, to detect, deter, neutr alize , and reduce cyber/physical attacks?” Table 6. CI/DR -HHM SSEI Scoring Scale (Total) The individual scores of the five categories are calculated from Figure 3 and Table 5 , then combined for the overall CI - HHM score. This t otal will be used to select the current effectiven ess level of the SSEI in Table 6 . Table 7 illustrates scores elicited from a Department of Energy (power) engineering expert. The CI/DR -HHM risk -assessment scores resulted in an overall SSEI score of 0.7 45 (75%) out of a total possible score of 1.0 (100%). This particular CI owner, with appropriate stakeholders, assessed their category relative effectiveness scores as follows: Threat (0.155 of max possible 0.20), Vulnerability (0.120 of max 0.20), Protec tion (0.178 of max 0.20), Resilience (0.156 of max 0.20) and Interdependency (0.136 of max 0.20). Table 4 denotes that a score of 0.745 falls in the range of “good security posture.” 57 Table 7. Area of Improvement/Deficiency (Cal culated SSEI) Table 7 also identifies a difference score (column 3), which denotes area or room for improvement. Given the overall score of 0.745, there is a total area of improvement of 0.255. Specific areas can be identified to improve as the result s are incorporated into the BBN model, e.g., the Vulnerability category risk -assessment resulted in a difference score of 0.080 (having the highest difference score). With an overall SSEI of 0.745, the BBN evaluation infers a “degraded” impact to the pat ient. To improve this potentially unacceptable impact, the CI operator would use the difference scores to take appropriate measures to increase overall security effectiveness, 58 ultimately improving their protection and resiliency. Further explanation is pr ovided in BBN section to follow (Step 7). The examination of dimensional (category) elements of the CI/DR -HHM is critical to the overall comprehensive security effectiveness evaluation. This research reveals that societal, environmental, legislative , and stakeholder perspectives and actions contribute to either strengthening or weakening protection and/or resiliency measures. An example to assess security measures from a dimensional (stakeholder) perspective follows. Example: HHM -Stakeholder Dimension : The questions in Table 8 should be asked from the perspective of each stakeholder (engineer, IT professional, CI owner/operator, Physical security, etc.). Table 8. Exemplar Stakeholder Question Categories A complete risk -assessm ent would follow the same logic and questioning from the perspective of other dimensions, e.g., for the legislation/governance/policy dimension, “are there laws/standards/policies in place (in the form of people, processes , or technology) that enable or prevent our ability to detect, deter, neutralize , or reduce cyber/physical threats?” If so, what are they and are they strategically employed to protect our security goal(s)? This step exits with an SSEI score to be incorporated next into the BBN analysis . 59 THE MODEL 4.2.6 STEP 7: ASSESS IMPACT LIKELIHOOD [C ONSTRUCT THE BBN] In this study, a Bayesian Belief Networks (BBN) are generated illustrating the dependencies and potential impacts of successful penetrations targeting CIs. The BBN is used here for its abi lity to account for uncertainty and limited available data of CI probability of attacks and interdependency/impact data. In general, BBNs can be used as visual representations of physical and/or logical access into and within an enterprise, information sy stem network, CI, or dependent resource. A BBN attack graph is provided at its highest level of nodal hierarchy, mapping the CI interdependencies of an Healthcare and Public Health (HPH) CI sector element (e.g., hospital). Figure 5 is offered as a simpl ified example to help understand and elucidate this research’s use of BBN. The fundamental question for analysis is “what is the probability of a hospital patient’s degraded health (with the possibility of death) given an attack on the power plant and/or t he water facility on which the patient ultimately depends?” This is determined by identifying through expert judgment or historical data the marginal probability of an attack on nodes B, C , and D; and subsequently calculating the joint probability to dete rmine the potential impact to node E (using Bayes Theorem). Upon calculating the probability of an attack on the hospital, that assessment is propagated toward computing the probability of its associated links (F and G). The joint probability of nodes F and G is then calculated to assess the vulnerability of H, the hospital patient’s degraded health or death. Netica BBN software uses the Joint Tree algorithm to make inferences that propagate calculated probabilites to adjoining nodes . The assessment of e ach node is iteratively improved given updated data/knowledge for that node, thus continuously reducing the uncertainty and adding more information fidelity for 60 decision making. Although this study considers each CI as a black box, it should be noted that one will increase the fidelity of a node by examining the hierarchy within each node. Internal to each node one would consider an aggregate of factors such as security measures currently in place, historical attack data that may be available for that nod e, etc. – ultimately providing additional insight into the “strength” weight or security effectiveness level for that node – assessing it to be less/more vulnerable to penetration. Figure 5. Simplified BBN For this study, the proof of concept is demonstrated by using historical and relative notional data (prior probabilities), while the unknown values are calculated through the Bayesian software simulation tool Netica v5.15 to infer the CI interdependency impacts to dependent r esources. The case is considered where essential interdependent CIs to the hospital have performed their risk -assessment of security effectiveness per the CI -HHM provided and have scored 61 themselves accordingly. The model structure and preliminary resu lts were vetted by medical professionals to confirm the model and various scenarios correctly represented their expectations. The Conditional Probability Tables (CPT) and prior probabilities used in the hospital patient healthcare BBN model were generated using data obtained from combined sources: medical professionals, NIPP, DoE, DHS, Verisign, SSP, Verizon Breach Reports, Symantec, and EPA along with best estimates and theoretical data from scenario analysis. Illustrated in the Figure 7 BBN, are various security effectiveness states of the CIs, given a successful DoS Attack or No_Attack (on Power CI), resulting in either No_Effect, Partial_Shutdown, or Complete_Shutdown.

The following definitions provide further understanding of the “what -if” scenario a nalysis model: No_Effect is defined as an event/attack having no significant impact/disruption, while a Partial_Shutdown indicates that the main source has been impacted and services are only being provided by the backup or a temporary alternate source. A Complete_Shutdown indicates that both the main source and the backup are no longer providing service. In Figure 7 , the basic structure of the BBN with links is illustrated from the interdependent CIs and their association to the hospital, subsequently no ting the links from the hospital to the medical devices on which a patient may be dependent to sustain life. Also noted, the network with nodes that indicate a successful Attack or No_Attack on a CI, highlighted in red. 62 Figure 6. Representation of BASE m2d Model of DoS attack on CIs (w/o SSEI) The CI_DoS Attack nodes in in Figure 6 are shown in red. Upon compilation, the dependent nodes are then calculated to assess the potential impact to the hospital and subsequently to the patient. The patient is assumed to be in the ICU and totally dependent on the medical device (dialysis, ventilator, etc.). This model structure allows one to assess the impact to either the hospital or patient target node of interest. Additional nodes c an be added, ultimately adding to the complexity of the interdependent nature of the CI, enterprise or organization. The estimated risk -assessment from each CI is now folded into the SSEI nodes indicating the CI strength or weakness. Until the SSEI rat ing/score is entered, the BBN model assumes a uniform distribution, essentially stating that the score is unknown at the time of compilation. Upon knowing/evaluating the respective CI SSEI or posture, that value is inserted as a Finding or as Evidence in the model. However, the other CIs will be estimated or varied to perform an appropriate “what -if” scenario analysis. As more evidence or data is known or observed, the Water No_Effect Partial_Shutdown Complete_Shutdown 100 0 0 Power No_Effect Partial_Shutdown Complete_Shutdown 100 0 0 Communications No_Effect Partial_Shutdown Complete_Shutdown 100 0 0 Hospital None Limited_Service No_Service 100 0 0 DoS_Attack_Power No_Attack Attack 100 0 DoS_Attack_Water No_Attack Attack 100 0 Ventilator None Degraded Out_of_Service 100 0 0 Dialysis_M achine None Degraded Out_of_Service 100 0 0 DoS_Attack_Comm No_Attack Attack 100 0 Ventilator_Patient None Degraded Critical 100 0 0 Dialysis_Patient None Degraded Critical 100 0 0 63 results of the model are improved. Performing t his step results in estimating the lik elihood of impact to a patient given a cascading failure from a DoS attack on a CI. Figure 7. Representation of SSEI Analysis given DoS Attack on Power CI BASE m 2d allows the decision -maker to assess from CI or dependent re source, i.e. “illustrate how the CI’s SSEI strength/weakness potentially impacts the hospital or the patient .” An operator could go further to assess or determine “what is the minimum SSEI one could have so as to not have a critical impact?” The scenario s to be evaluated are numerous, each potentially enabling the decision -maker to make more informed and proactive improvements for better protections. Water No_Effect Partial_Shutdown Complete_Shutdown 15.9 40.9 43.2 Power No_Effect Partial_Shutdown Complete_Shutdown 47.9 31.3 20.8 SSEI_Water Excellent Good Fair Poor 0 0 0 100 SSEI_Power Excellent Good Fair Poor 16.7 25.0 50.0 8.33 SSEI_Communications Excellent Good Fair Poor 0 0 0 100 Communications No_Effect Partial_Shutdown Complete_Shutdown 39.8 28.9 31.3 Hospital None Limited_Service No_Service 31.8 22.7 45.5 DoS_Attack_Power No_Attack Attack 0 100 DoS_Attack_Water No_Attack Attack 50.0 50.0 Ventilator None Degraded Out_of_Service 17.3 23.2 59.5 Dialysis_M achine None Degraded Out_of_Service 37.6 19.2 43.2 DoS_Attack_Comm No_Attack Attack 50.0 50.0 Ventilator_Patient None Degraded Critical 17.3 23.2 59.5 Dialysis_Patient None Degraded Critical 37.6 19.2 43.2 64 Figure 7 illustrates how the use of SSEI HHM (blue nodes) assessments can be used to make decisions to ul timately improve a CI element’s security posture. Upon determining the SSEI score from the risk -assessment, the CI -HHM BBN model was used to evaluate impacts to a CI facility, dependent hospital , or patient given a successful DoS. Figure 7 also illustrate s how the Water CI and Communication CI have experienced a successful penetration, which yields a “Poor” relative effectiveness index. The model indicates that given a “Poor” index, the probability of Complete_Shutdown on Water and Communications is 43.2% and 31.3%, respectively, given a successful DoS attack on Power (due to their interdependence). Additionally, the impact to the hospital is noted as having a 45.5% probability of providing “No_Service” to dependent resources given the CI vulnerabilities. Subsequent actions by the hospital to improve the index should result in a lower likelihood of providing “No_Services” to the hospital. Further analysis revealed a 43% probability of potential “critical” impact to the patient. A “critical” impact score to the patient indicates a life -threatening result due to the grave nature of the patient’s condition and their total dependence on the medical device that is providing services. Given an unsatisfactory potential patient impact score , the SSEI risk -assess ment (difference scores) that was previously performed is re -assessed. This re -assessment specifically seeks to target/improve security measures that would strengthen or reduce the impact to the target goal (patient). This evaluation is an iterative proc ess with the goal of continual, targeted, security effectiveness improvement. Decisions can be made given various trades from the scenario analysis, taking into account the security goal, risk priorities , and budget constraints. The “ difference scores ” indicated in the SSEI table allow the CI operator to strategically target specific areas of improvement for better protections. 65 4.2.7 STEPS 8-10: DECISION ANALYSIS The decision -maker has now examined, based on his security goals, “What can go wrong and its impac t?” (steps 1 -6), “How likely is it to go wrong?” (step 7), and “What are the possible outcomes?” (Steps 1 -7). From Step 7, we have learned the likelihood of impact to the patient in the event of a CI DoS attack. Step 8 -10 uses this information to make s trategic decisions based on relevant goals. Upon estimating the risk to the patient, as categorized in the model (Step 8), the decision -maker proceeds to Step 9 to determine if the risk is acceptable to sustain life until alternative protective or resilie nt measures take effect. Step 10 responds to the respective answer: If “yes ,” re -assess periodically for new threats or assessments; If “No ,” implement new/additional measures based on self -assessment risk analysis (difference scores). The BASE m 2d metho dology seeks to empower emergency management personnel with the ability to make risk -informed decisions given the realities of the world we live in today – uncertainty; sophisticated adversaries; limited data; and adaptive, complex, and interdependent infr astructure. With the insight gained from using the multi -dimensional approach, we have potentially uncovered otherwise hidden areas of risk. Coupled with BBN, the user can operationalize the stakeholder information to make strategic decisions. CHAPTER 5 - DATA ANALYSIS AND RESULTS 5.1 ANALYSIS OBJECTIVES This section describes the analysis of data gathered and presented by this research. Data analysis consisted of analyzing demographic, descriptive data, inferential data, response data, and reliability data (i.e., Cronbach alpha). Statistical software (i.e., Minitab, SPSS and Netica) was employed to 66 perform the necessary analysis. There were two main objectives identified for the analyses performed in this research: 1.) Reliability and Validity of Surve y Instrument; 2.) Validi ty of Conceptual Framework. Ten (10) respondents/representatives from ten distinct hospitals participated in the online survey. The analysis revealed internal reliability and validity of the survey instrument and the framework. Th e findings of each are detailed in the following sections. 5.2 DEMOGRAPHIC DATA Demographic data were collected on respondents on specific job function in the medical profession, with options of physician, nurse practitioner , or administrator. Research reveal ed that any of these professions would provide relevant data regarding the objectives of this study. Years of experience and experience in ICU data were also collected. Thirteen hospitals were solicited for expert knowledge on risk management, metrics, procedures , and policies regarding CI service interruption impacts to patients. Ten hospitals responded by completing a 60 -question electronic survey, to include comment sections of voluntary information allowing for elaboration of selected answers or to p rovide additional details. A m ajority (7 of 10) of the survey respondents were tenured medical professionals (10 years or more experience in a medical field). All respondents were identified as personnel experienced and qualified to speak on behalf of th eir hospital. 67 Figure 8. Survey: Years of Experience in Medical Field Figure 9. Survey: Medical Field Profession Figure 10 . Survey: Experience in Intensive Care Unit (IC U) 68 In this research, experts are loosely defined as a nurse, physician, or administrator knowledgeable in the hospital’s risk management practices, and having a clinical understanding of patient impact given specific service interruptions . Respondents ex perience in ICU was relevant to understand their appreciat ion of emergency equipment operations and significance to patient’s survival . Although ten experts were surveyed, five experts are usually sufficient for most elicitation efforts. Clemen and Winkl er (1985) discussed that data saturation can occur and that diminishing returns have been observed when including additional experts. 5.3 METRICS Basic and publicly available metric data was elicited from respondents for two purposes: 1 ) understand knowled ge of medical professional staff of hospital protocols and procedures regarding measures used to trigger alternate systems, and 2) Calibration of expert judgment information, if desired. Questions and responses are summarized below. What metric is used t o trigger backup POWER generator? 69 What metric is used to trigger backup WATER supply? What metric is used to trigger alternate COMMUNICATIONS? In summary, it is interesting to note elicited response s regarding knowledge of metrics used : 50% were no t aware of metrics used to trigger alternate power sources or generators; a disparity in knowledge existed for metrics used for water; and more than 70% indicated they had no knowledge of alternate measures or metrics for communications given a service out age. 70 As indicated in this research, metrics are imperative to demonstrate emergency preparedness for the public healthcare community; just as important is assessing those measures periodically and ensuring that the staff is aware and trained to respond ac cordingly in the event alternate comms are needed. 5.4 DESCRIPTIVE DATA Data were collected from experts on likelihood of CI service disruption impacts to patients depending on the following medical devices: cardiac equipment, dialysis systems, and oxygen/ven tilators. Ten medical professional respondents from different hospitals provided expert judgment on likelihood of impact (critical, degraded, no impact) given a CI (power, water, comms) service disruption. A summary of analysis was performed on data col lected from the respondents. The data was grouped by CI service disruption (power, water, comms) against likelihood of impact (critical, degraded, no impact) to patients depending on various medical equipment (cardiac care, dialysis, ventilator ). The resu lts were used to validate the behavior of the BBN model. The data was collected from the respondents used a 5 point Likert scale that correlated to probability distributions as follows: 1 2 3 4 5 Very unlikely Unlikely Need More Info Likely Very Likely (0-20%) (21 -40%) (41 -60%) (61 -80%) (81 -100%) Power Disruption Impact : Respondents were mostly consistent in their responses regarding the likelihood of “critical” impact of disrupted power to medical devices. Respondents tended to agree (between likely and very likely) that any disruption of service to power (limited or complete shutdown) would have a higher likelihood of having a “critical” impact on a patient depending on 71 dialysis, cardiac care , and/or ventilator. More variance was noted among respond ent s in regards to assessing “degraded” impact to patient s given a service disruption. Respondent ability to assess “degraded” impact to patient given power disruption exhibited a large variance, indicating responses in this category to be inconclusive. N evertheless, it was noted that fifty percent of the respondents selected “need more info” to determine the likelihood of impact to a cardiac care patient given limited power. Respondents tended to agree that “no impact” to a patient would be less likely (b etween very unlikely and unlikely) given service disruption in power to any of the medical devices. This was consistent with the BBN model results. Water Disruption Impact : Respondents were mostly consistent in their responses regarding the likelihood of “critical” impact of disrupted or contaminated water supply. Respondents tended to agree (between likely and very likely) that any disruption of service to water (limited or complete shutdown) would have a higher likelihood of having a “critical” impact t o a patient depending on dialysis, cardiac care , and/or ventilator. Respondent s had a general agreement and a tendency toward unlikely “degraded” impact to patients given water disruption, although a variance spread was dually noted in this category. A stronger agreement among respondents (between very unlikely and unlikely) was noted in assessing likelihood of “no impact” to patient s given a water disruption/contamination. This was consistent with the BBN model results. Communication Disruption Impac t: Respondents were generally consistent in their responses regarding the likelihood of “critical” impact of disrupted c omms unlikely to very likely. Meaning, each respondent was consistently explicit that disrupted comms would have some measure of impact to patient care. This category may have been less intuitive to respondents which is indicated by the disparity in their response. Comms is represented here as the electronic 72 response system used to communicate with emergency staff to provide the status o f patient s connected to a medical device, such as an alarm to notify service has been disrupted. If communication systems are down , medical professionals may be unaware that help is needed and services are no longer being provided or have been degraded. Re spondents tended to agree (between unlikely and very unlikely) that any disruption of service to comms (limited or complete shutdown) would have a higher likelihood of having a “degraded” impact to a patient depending on dialysis, cardiac care , and/or vent ilator. More variance was noted among respondent s in regards to assessing “no impact” to patient s given a comms service disruption, indicating more info would be needed to determine increase or decrease in “no impact” likelihood. This was consistent with the BBN model results. 5.5 RELIABILITY AND VALIDITY OF SURVEY INSTRUMENT To determine if the results of the survey instrument are reliable and va lid, respondent data is evaluated against the data produced by the tool (instrument). Specifically, did the surv ey measure what it was intended to measure . In review, the instrument was evaluated as follows: RELIABILITY ▪ INTERNAL CONSISTENCY or Homogeneity using a Cronbach coefficient (α >0.7) o Measures the extent to which a technique, experiment, or measuring proce dure assess the same characteristic or quality ▪ INTRARATER RELIABILITY o Measures the extent to which multiple respondents agree in their ratings of given items. 73 Results revealed that the survey instrument tested reliable in its ability to assess the impact to a patient given a service disruption in power, water, and communications. Each category demonstrated a Cronbach alpha as follows: ➢ Critical α = 0.936 ➢ Degraded α = 0.924 ➢ No impact α = 0.931 These results indicate the strong internal consistency of the tools ’ ability to measure the aforementioned attributes. To determine the intrarater reliability, the Intra -class Correlation Coefficient (ICC) was used. This statistical measure the proportion of variance of an observation due to between -subject variability in the true score (Fink 2003). ➢ Critical ICC = 0.932 ➢ Degraded ICC = 0 .929 ➢ No impact ICC = 0.925 The results indicate consistent responses between the respondents. VALIDITY ▪ CONTENT VALIDITY o Measures the extent to which the question/items thoroughly and appropriately assess the characteristics or qualities it purports to measures. ▪ FACE VALIDITY o Measures the appearance of the metric/question on the surface. 74 Content validity is a subjective measure of how appropriate ite ms or scales seem to a set of reviewers identified as Subject Matter Experts (SMEs) (Litwin 2003). It typically involves an evaluation of the survey’s content to ensure it incl udes relevant information or questions. Content validity is not quantified with statistics (Litwin 2003). Instead it is an opinion from subjective experts. Th e survey was reviewed by administrators of each hospital for relevance before appointing an app ropriate Respondent. The survey was deemed relevant and appropriate for its purpose. Face Validity is determined by an observer with an untrained eye and is considered a casual assessment of the appropriateness of the survey instrument (Litwin 2003). 5.6 VALIDITY OF CONCEPTUAL FRAMEWORK A major goal of this study was to develop a model that is valid in assessing the ultimate purpose of evaluating system security effectiveness of CIs and dependent resources , given noted interdependencies . Events and actions that occur from a dependent or linked CI ha ve the ability to influence the likelihood of occurrence and impact on an adjoining dependent resource. It is demonstrated here that this phenomena can be modeled using BBN nodes with various alternating states. The direction of influence is identified with parent -child or CI -hospital and/or hospital - medical device references as required by one of the rules of BBN using Directed Acyclic Graphs (DAG). Although the direction of influence and dependence often flows from parent to child, inferences can be made in the opposite direction. The model learns from this data and can generate Conditional Probability Tables (CPTs) that are often used to evaluate the model performance. The CPTs used in this research were gene rated from historical and relational data. As mentioned previously, this conceptual framework is validated with the use of data identified in the Data Collection section and correlated with data provided by experts. 75 5.7 THREATS TO INTERNAL VALIDITY Threats t o validity for this research include the following: • Internal validity demonstrates a causal relationship between variables. Medical professionals are influenced (bias and overconfidence) by many factors , and this is often reflected in their responses. The approach used to mitigate this threat to internal validity was to take care in training or calibrating the expert with the use of mixed method techniques (as described in the Research Methodology section). Respondents were also allowed to provide open comm ents to elaborate or ask questions, if needed. Clear instructions and training are imperative to mitigating this threat. • Reliability of the measurements can threaten validity . The data collection instruments included: knowledge questions to establish the credibility of the respondent, consistent measurement scales, clear and unambiguous questions, concise and efficient questionnaire design, and sufficient time to fill -out the questionnaire (Babbie, 1990, Sudman and Bradburn, 2004; Valerdi, 2005). 5.8 THREATS TO EXTERNAL VALIDITY External validity refers to the ability to apply the research results to other contexts or domains. The external validity of this study extends beyond CIs and hospitals. Another application within the space community is to evaluate p otential impacts to satellites given a successful attack to interdep end ent ground stations. The general concept has extensibility to assess and evaluate how a successful threat to a third party’s vulnerability can impact a primary or secondary source and vice versa . 76 CHAPTER 6 – CONCLUSION This research establishes a systematic, multi -dimensional, quantitative, metric -based framework required to effectively assess, measure , and ultimately improve an organization’s security posture. The utility of multi -dimensional models such as HHM and BBN and applying them to CI cyber - physical security protection and resiliency is examined . To achieve greater se curity effectiveness for CIs, a systematic approach , using Systems Engineering principles, within the operat ional environment by first identifying security goals and objectives based on what is to be protected was suggested . Second, identify all paths that lead to the CIs that need protecting, including internal and external access and dependencies (via people, processes, technology, infrastructure, i.e., the entire enterprise). Third, articulate what security means are implemented from the perspective of all stakeholders. In the absence of this type of systematic approach, organizations, enterprises , and infra structures will remain vulnerable. This research postulated that the understanding and influence of the strengths/weaknesses of the enterprise are enhanced with the use of inferential statistics coupled with multi -dimensional, metrics -based analysis. It a lso is important to position security professionals and other key decision makers to measure the effectiveness of implemented security controls. Further, this framework and methodology empowers leaders to make more informed decisions as to where resources should be focused for more effective security. The methodology used to address this complex problem of improving cyber -physical security protection of the enterprise, or specifically in this case the nation’s CIs, must expand beyond traditional approach es used today. Further, organizations, CI owners , and operators must have the ability to assess the effectiveness of the security program they put in place. Measures of effectiveness (MOE) identified in the proper context (based upon security goals and t he operational 77 environment), using relevant performance metrics for proactive monitoring and vulnerability assessment, can be used to not only understand security posture but also to highlight deficiencies that would otherwise prevent the making of informe d decisions toward the implementation of more cost -effective solutions. 6.1 CONCLUSION WITH RESPECT TO STUDY HYPOTHESES In review, the following hypotheses were asserted: Hypothesis 1a: Probabilistic reasoning and a systems -thinking based framework can b e combined to assist in the overall evaluation (strength or weakness) of CI and/or dependent resources security effectiveness. Hypothesis 1b: Probabilistic reasoning and a systems -thinking based framework can be combined to quantifiably evaluate security effectiveness of Protection and Resiliency (P/R) for CIs and their dependent elements. It was shown that HHM (systems -thinking based framework) and BBN (probabilistic reasoning) could be combined to demonstrate the strength or weakness of CI security eff ectiveness by implementing the HHM qualitative component to evaluate various elements of the CI from multiple dimensions, and using the BBN module of the framework to dynamically assess the potential impact of a threat in accordance with available data and identified dependencies. It was further determined that the hybrid model demonstrated the ability to quantify the effectiveness of protection and resiliency for CIs with the use of metrics, the SSEI , and the inherent nature of BBNs. 6.2 CONCLUSION WITH RESPE CT TO STUDY QUESTIONS In review, the following questions were asserted for this research: Question #1 : How can the assessment of security effectiveness of CI interdependencies and vulnerabilities be modeled proactively (before occurrence or penetration) f or improved decision - making? 78 The use of BBNs allowed for the most appropriate assessment of indirect consequences of a direct attack. The very nature of BBN development is to establish hierarchical dependencies, often to multiple orders (3 rd, 4 th order, etc.) . It was demonstrated through scenario analysis within the framework and a hospi tal case study that a successful DoS attack on a CI (power, etc.) could be modeled proactively (before occurrence or penetration) for improved decision -making Question #2 : How can combining HHM and BBN facilitate the assessment of relative effectiveness of CI security measures? The BASE m 2d framework integrated both qualitative (HHM) and quantitative (BBN) modules/components to exercise the SSEI concept, allowing a CI own er/operator the ability to self - assess the current security posture of their infrastructure based on their security goals and tolerance. This self -assessment with the conceptual framework enables the CI to ultimately make improvements to prevent or mitiga te (direct and indirect) negative impacts. Question #3 : What metrics are used by hospitals to trigger auxiliary systems in the event of a shutdown (partial or complete)? What metrics, if any are used to warn, prevent, or absorb CI service interruption s? As a result of the Respondents survey, it was surprisingly noted that many medical professionals were unaware of any metrics used to trigger auxiliary systems in their respective hospitals. 6.3 DISCUSSION In this research, the question “how secure am I?” is not asked. Although a relevant question, a more realistically attainable and measurable question is posed – “how effective are the security 79 measures I have put in place?” More specifically, “what systematic construct or metrics have we employed to as sess that effectiveness in accordance with our value proposition or security goals?” For example, if you have removed unnecessary accesses (via physical hardware or personnel (root access)), you have essentially reduced your attack surface; in turn, you c an measure the effectiveness of that security implementation. Although many organizations work diligently to ensure they are in compliance with the latest standards, it has been proven by our adversaries that being in compliance does not guarantee secur ity. To effectively achieve the stated resear ch goals, this research proposed the integration of two hierarchical techniques to enable a qualitative and quantitative assessment of security measures. This framework provides a risk assessment via scenario o r what -if analysis to inform the decision -maker on where best to allocate proper resources that maintain a specific and appropriate level of service to preserve human life in the event of a CI failure due to an attack vector (partial shutdown or complete s hutdown). In addition, it is assert ed that metrics should be in place to account for minimum mean time between failures to ensure resiliency measures are effective. For example , a back -up generator must be triggered within a certain timeframe to ensure a ventilator patient does not asphyxiate due to oxygen deprivation . Additionally, the significance of attack graphs and understanding attack paths to protect an enterprise security goal is a strategic and important step in effective and efficient use of resources. Attack paths/graphs are to be produced from various scenarios within the CIs’ operational environment, to include interdependencies with other CIs. BBNs are then generated from each attack graph to identify potential areas of vulnerability. HH M has been used to identify dimensional variables (temporal, geo -spatial, legislative, societal, stakeholders’ perspective) from which strengths and weaknesses of each CI node are to be evaluated. Finally, a simplified BBN threat scenario is provided that is used to demonstrate and evaluate the CI security effectiveness per the 80 identified metrics. The overall methodology is iterative to incorporate new data or knowledge (via BBN), to produce results with greater accuracy and fidelity; this construct is sc alable and extensible in that this method can be used for the largest CI to the smallest architectural element (network or enterprise) and various CI domains; and a general process i s provided for repeatability. This study expands on knowledge, experienc e, and recommendations offered by previously documented research. The hybrid methodology assesses the potential impacts to patients, medical devices and the emergency preparedness of a healthcare facility given a CI service disruption. Additionally, it provides a holistic, strategic, quantitative, and measurable approach that permits one to perform relative security effectiveness analysis in the face of uncertainty and where limited data prevents necessary measures from being executed to ensure proper prot ection and resiliency of the enterprise. I bound and strategically target the problem -solution space by defining the operational environment and identifying the security goals that are to be protected. This research postulate s that the understanding and influence of the strengths/weaknesses of the enterprise are enhanced with the use of inferential statistics coupled with multi -dimensional, metrics -based analysis. Standardized use of the SSEI would allow CIs and/or key resources to maintain and communica te a certain “acceptable” index that merits or ensures an agreed -to level of service to its dependent resources. It is acknowledged that current methodologies used to address the complex problem of improving cyber/physical security protection of the en terprise, or specifically in this case the nation’s CIs, must expand beyond existing traditional approaches. Most security methods used today are considered from a single -dimension. This is normally accomplished by protecting architectural elements or com ponents (network of routers, switches, servers) from cyber -attacks, i.e., by configuring firewalls, implementing policies , and training users. Although these techniques 81 demonstrate a noble effort, they have proved to be neither sufficient nor effective. Further, organizations, CI owners , and operators must have the ability to assess the effectiveness of the security program they put in place. Measures of effectiveness (MOE) identified in the proper context (based upon security goals and the operational e nvironment), using relevant performance metrics for proactive monitoring and vulnerability assessment, can be used to not only understand security posture but also to highlight deficiencies that would otherwise prevent the making of informed decisions towa rd the implementation of more cost -effective solutions. Finally, this approach allows organizations, CI owners , and operators the ability to assess and reduce vulnerabilities within and across the security domain to which they interface and are responsible . CHAPTER 7 - FUTURE RESEARCH The SSEI scale proposed in this research could be improved, verified , and validated by CI owners/operators and further assessed against previous cases of enterprises or CIs that have experienced DoS attacks to test predictab ility and reliability. This research also recommends that for a complex, interdependent system of CIs to effectively provide critical services to dependent resources, more effective and efficient standards need to be implemented. Standards that mandate CI s communicate (share information) across CIs and to those that depend on their services, in accordance with a security effectiveness taxonomy understood by the impacted community of stakeholders. The SSEI concept introduced in this research, if further de veloped, would allow CIs to communicate in a common language at a confidential or sensitive information level, if necessary. This would allow appropriate security measures to be considered from the 82 perspective of their own internal security effectiveness evaluation, as well as the external interdependent sources security effectiveness levels/evaluation. This research has laid the ground work for future studies in the area of understanding, assessing , and improving system security effectiveness for CIP/R . The following recommendations are offered as potential areas of research: • The Systems Security Effectiveness Index can be furthered developed by generating a more relevant scale of measurement. This research provided a general scale to demonstrate the c oncept of the hybrid framework by equally dividing each category . For example, further research can determine if a scoring range of .95 – 100 percent should be considered ‘Excellent’, or a range of 0 - 0.50 should be considered ‘Poor’ security effectivene ss. • The CI -HHM provided in this research established an equal scale for the variables identified from the NIPP. Additional research could establish priorities for a specific CI function (Water, Power, etc.) or specific enterprise in question and place rel evant weights using models such as AHP or MAUT . • This conceptual framework was demonstrated using a hospital case study however, the researcher believes the concept is extensible to other disciplines with varying threats • Establish an SSEI ontology that can be generalized to a relevant community and mandated to be shared . 83 REFERENCES Adar, E., & Wuchner, A. (2005). Risk management for critical infrastructure protection (CIP) challenges, best practices , & tools. Paper presented at the 8 pp. doi:10.1109/IW CIP.2005.18 Alden, J. (2006) Measuring the “unmeasurable .” Perf. Improv., 45: 7–11. Alqahtani, Abdulrahman, (2015) "Towards a Framework for the Potential Cyber -Terrorist Threat to Critical National Infrastructure." Information and Computer Security 23 (5 ): 532 -569 Annex, A., (2010) Infrastructure, N., & Plan, P. Healthcare and Public Health Sector -Specific Plan Amass F. S., Bhunia, A. K, Chaturvedi, A. L., Dolk, D. R., Peeta, S., and Atallah, M. J. Advances in Homeland Security Series: The Science of Hom eland Security (Volume 1), West Lafayette, IN: Perdue University Press, 2006 227 pp, introduction, notes, b ibliography, index (hardcover) Assessment of Key Risks for Hospitals and Healthcare Systems – Spring 2010, Ayyub, B. M. (2001). Elicitation of expert opinions for uncertainty and risks , CRC Press. Babbie, E. (2010). The basics of social research (5th ed .). Belmont, CA: Wadsworth Publishing. Bayraktarli, Y. Y., J. Ulfkjaer , et. al. . (2005) On the application of Bayesian probabilistic networks for earthquake Bayuk , Jennifer and Ali Mostashari. (2013) "Measuring systems security." Systems Engineering : 1 - 14. Bensi, Michelle Terese. (2010) A Bayesian Network Methodology for Infrastructure Seismic Risk Assessment and Decision Support 84 Biringer, B., E. Vugrin, et. al. . (2013). Critical infrastructure system security and resiliency , CRC Press. Bloomfield R , Chozos N , Nobles P . (2009) Infrastructure interdependency analysis: introductory research review. Adelard LLP; Boring, R., Gertman, D., Joe, J., Marble, J., Galyean, W., Blackwood, L., & Blackman, H. (2005). Simplified Expert Elicitation Guidelines For Risk Assessment Of Operating Events . (INL/EXT -05 -00433). Idaho Falls, Idaho: Department of Energy, U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, Division of Risk Analysis & Applications. Burnside ES, Rubin DL, Shachter RD. (2004) Using a Bayesian Network to Predict the Probability and Type of Br east Cancer Represented by Microcalcifications on Mammography” in: Fieschi, M., Coiera, E., and Li, Y.J., Eds., Medinfo 2004, Proceedings of the 11th World Congress on Medical Informatics, Sept. 7 -11, 2004, IOS Press, 13 -18. Carmines, E. and R. Zeller, Rel iability and Validity Assessment . Quantitative Applications in the Social Sciences, ed. M.S. Lewis -Beck. 1979, Thousand Oaks, California: SAGE Publications. Chittister, Clyde C., and Yacov Y. Haimes. (2012) "Risk to cyberinfrastructure systems served by cloud computing technology as systems of systems." Systems Engineering 15: 213 -224. Clemen, R. T. and R. L. Winkler (1985). "Limits for the Precision and Value of Information from Dependent Sources." Operations Research 33 (2): 427 -442. Cooke, R.M. 1991. Ex perts in Uncertainty: Opinion and Subjective Probability in Science Cooke, R. M., & Goossens, L. H. (2004). Expert judgement elicitation for risk assessments of critical infrastructures. Journal of risk research , 7(6), 643 -656. 85 Cronbach L. J. (1971) "Test Validation," R.L. Thorndike (ed.) Educational Measurement Washington, DC: American Council on Education Department of Homeland Security, http://www.dhs.gov/healthcare -and -public -health -sector Di Giorgio, Alessandro, and Francesco Liberati. (2011) "Interde pendency modeling and analysis of critical infrastructures based on Dynamic Bayesian Networks." In Control & Automation (MED), 2011 19th Mediterranean Conference on , pp. 791 -797. IEEE. Dimase, D., Collier, Z. A., Heffner, K., & Linkov, I. (2015). Systems e ngineering framework for cyber physical security and resilience. Environment Systems & Decisions, 35(2), 291 -300. Druzdzel, MJ and Van der Gaag, LC, 1995 “Elicitation of probabilities for belief networks:

combining qualitative and quantitative informatio n,” Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence 141 –148 Eusgeld, I., C. Nan, S. Dietz, (2011) ‘‘System -of-systems” approach for interdependent critical infrastructures, Reliability Eng. Syst. Saf. 96 679 –686. v Eusgeld, I., D. Henzi, et. al. . (2008). "Comparative evaluation of modeling and simulation techniques for interdependent critical infrastructures." Scientific Report, Laboratory for Safety Analysis, ETH Zurich . Executive Order 13636 (2013) Improving Critical Infr astructure Cybersecurity, President Barack Obama Executive Order 13010 (1996) . Critical Infrastructure Protection. Federal Re gister. Vol. 61. No. 138. Executive Order 13130 (1999). National Infrastructure Assurance Council, Federal Register, Vol. 64. 86 Executive O rder 13231 (2001) Critical Infrastructure Protection in the Information Age. Federal Register. Vol.66. Ezell, B. C. (2007). Infrastructure Vulnerability Assessment Model (I ‐VAM). Risk Analysis, 27(3), 571 -583. Fink, A. (2003). The survey h andbook, Sage. Fink, A. (2003). The Survey kit:How to manage, analyze and interpret survey data . Thousand Oaks, CA: SAGE Publications Ltd doi: 10.4135/9781412984454 Fink, A. (2003). The survey kit: How to manage, analyze, and interpret survey data, Thousa nd Oaks, CA: Sage Publications, Inc. Flandoli, F., E. Giorgi, et. al. (2011). "Comparison of a new expert elicitation model with the Classical Model, equal weights , and single experts, using a cross -validation technique." Reliability Engineering & System S afety 96 (10): 1292 -1310. Forsberg JA, Eberhardt J, Boland PJ, Wedin R, Healey JH. (2011) Estimating Survival in Patients with Operable Skeletal Metastases: An Application of a Bayesian Belief Network. PLoS ONE 6(5): e19956. Forsberg, J. A., Healey, J. H., & Brennan, M. F. (2012) A probabilistic analysis of completely excised high -grade soft tissue sarcomas of the extremity: an application of a Bayesian belief network. Annals of Surgical Oncology, 19(9), 2992 –3001. Fowler, F. J. (1995). Improving survey qu estions : Design and evaluation . Thousand Oaks, CA: Sage Publications. Fowler, F. J. (2009). Survey research methods (4th ed .) Thousand Oaks, CA: Sage Publications. 87 Frigault, Marcel. (2010) Measuring network security using Bayesian network -based attack grap hs. Ph.D. diss., Concordia University (Canada), http://search.proquest.com/docview/794810164?accountid=11243 Gass, Saul I. 2005. Model world: The great debate -MAUT versus AHP. Interfaces 35, (4) (Jul): 308 -312, http://search.proquest.com/docview/217112431 ?accountid=11243 (accessed February 27, 2015). Ghorbani AA, BagheriE. (2013) The state of the art in critical infrastructure p rotection : a framework for convergence. International Journal of Critical Infrastructures 2008;4:215 –44. Grover, Jeff. (2013) Str ategic economic decision -making using Bayesian belief networks to solve complex problems . New York, NY: Springer Haimes, Yacov Y. (1981) "Hierarchical Holographic Modeling." IEEE Transactions On Systems, Man, and Cybernetics : 606 -617. Haimes, Yacov Y. an d Chittester, Clyde G. (2005) "A Roadmap for Quantifying the Efficacy of Risk Management of Information Security and Interdependent SCADA Systems," Journal of Homeland Security and Emergency Management: Vol. 2: Iss. 2, Article 12. Haimes, Yacov Y. (2004) “ Risk modeling, assessment, and management” . Hoboken, N.J.: Wiley - Interscience Haimes YY,Horowitz BM, Lambert JH, Santos JR,Crowther KG, and Lian C. Inoperability input – output model for interdependent infrastructure sectors . II: ase studies . Journal of Inf rastructure Systems,11;80 –92. Haimes, Y.Y., J. Lambert, Duan Li Duan Li, R. Schooff, and V. Tulsiani. (1995) “ Hierarchical holographic modeling for risk identification in complex systems . 1995 IEEE International Conference on Systems, Man , and Cybernetics” . Intelligent Systems for the 21st Century. 88 Harrison, Keith and White, Gregory B. (2010) "An Empirical Study on the Effectiveness of Common Security Measures." Lecture, 2010 43rd Hawaii International Conference on System Sciences, Koloa, Kauai, Hawaii Hay es, N. (Ed.) (1998). Doing qualitative analysis in psychology. Hove: Psychology Press. Hokstad, P., I.B. Utne, J. Vatn,( 2012) Risk and Interdependencies in Critical Infrastructures: A Guideline for Analysis, Springer. Hubbard, D. W. (2009). The failure o f risk management: Why it's broken and how to fix it . John Wiley & Sons. Hubbard, Douglas W. (2010) How to measure anything finding the value of "intangibles" in business . 2nd ed. Hoboken, N.J.: Wiley. Hubbard, Douglas W. (2016 ) How to measure anything in cybersecurity risk . Hoboken, N.J.: Wiley. http://ics -cert.us - cert.gov/sites/default/files/Monitors/ICS -CERT_Monitor_Oct -Dec2012.pdf http://www.dhs.gov/xlibrary/assets/NIPP_Overview.pdf HTTP ://WWW .DHS .GOV /NEWS /2013/07/18/ WRITTEN - testimony -nppd -house -home land -security - subcommittee -cybersecurity https://www.washingtonpost.com/world/national -security/russia -has -developed -a-cyber -weapon - that -can -disrupt -power -grids -according -to-new -research/2017/06/11/b91b773e -4eed -11e7 - 91eb -9611861a988f_story.html?utm_term= .89e606950738 (2017) https://www.whitehouse.gov/the -press -office/2017/05/11/presidential -executive -order - strengthening -cybersecurity -federal (2017) https://www.healthcare -informatics.com/article/cybersecurity/exclusive -report -what -can -us - healthcare -it-lead ers -learn -wake -wanna -cry (2017) https://www.cnet.com/news/petya -goldeneye -wannacry -ransomware -global -epidemic -just -started/ (2017) 89 Jha, Manoj K.(2009) "Dynamic Bayesian network for predicting the likelihood of a terrorist attack at critical transportation infrastructure facilities." Journal of Infrastructure Systems 15, no. 1: 31 -39. Jaquith, Andrew. (2007) Security metrics: replacing fear, uncertainty, and doubt . Upper Saddle River, NJ: Addison -Wesley. Kahan, Jerome H.; Allen, Andrew C.; and George, Just in K. (2009) "An Operational Framework for Resilience," Journal of Homeland Security and Emergency Management: Vol. 6: Iss. 1, Article 83. Kjølle, G. H., I. B. Utne, et. al. . (2012). "Risk analysis of critical infrastructures emphasizing electricity supply and interdependencies." Reliability Engineering & System Safety 105 : 80 - 89. Kohavi, R. (1995). A study of cross -validation and bootstrap for accuracy estimation and model selection . Kozik, Rafał, Michał Choraś, and Witold Hołubowicz. (2010) "Fusion of Bayesian and Ontology Approach Applied to Decision Support System for Critical Infrastructures Protection." In Mobile Lightweight Wireless Systems . : Springer Berlin Heidelberg. Kuehn, N. M., Riggelsen, C., and Scherbaum, F. (2009). ―Facilitatin g Probabilistic Seismic Hazard Analysis Using Bayesian Networks. ‖ Seventh Annual Workshop on Bayes Applications (in conjunction with UAI/COLT/ICML 2009). Laconte, P., Y. Y. Haimes, et. al. . (1982). Water resources and land -use planning : a systems approach : Proceedings of the NATO Advanced Study Institute on: "Water Resources and Land -Use Planning," Louvain -la-Neuve, Belgium, July 3 -14, 1978. The Hague ; Boston Hingham, MA, USA, M. Nijhoff ; Distributors for the U.S. and Canada, Kluwer Boston. 90 Laskey, K. B. , & Mahoney , S. M. (2000). Network Engineering for Agile Belief Network Models, 12(4), 487 –498. Lewis, T. G. (2006). Critical infrastructure protection in homeland security: defending a networked nation , John Wiley & Sons. Liberati, A. D. G. a. F. "." IEEE SYSTEMS JOURNAL VOL. 6, (NO. 3): 510 -519. Little, Richard G. (2003) "Toward More Robust Infrastructure: Observations on Improving the Resilience and Reliabi lity of Critical Systems." Lecture, System Sciences. Proceedings of the 36th Annual Hawaii International Conference on, Big Island, Hawaii. Litwin, M. S. and Fink, A. (2003). How to assess and interpret survey psychometrics, Sage. Mahoney, SM and Laskey, KB, 1996, “Representing and combining partially specified CPTs” Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence 391 –400. Marcot, B. G., Steventon, J. D., Sutherland, G. D., & Mccann, R. K. (2006). Guidelines for developin g and updating Bayesian belief networks applied to ecological modeling and conservation McGee, Sibel, Jaime Frittman, Seongjin James Ahn, and Susan Murray. 2016. Implications of cascading effects for the hyogo framework. International Journal of Disaster R esilience in the Built Environment 7, (2): 144 -157 Min, H. -S. J., W. Beyeler, et. al. . (2007). "Toward modeling and simulation of critical national infrastructure interdependencies." Iie Transactions 39 (1): 57 -71. Netica v5.15, http://www.norsys.com/index. html 91 National Association of County and City Health Officials (NACCHO), (2014) Cyber Attack on U.S. Hospital Group Highlights Vulnerability of Critical Infrastructure http://nacchopreparedness.org/ Nati onal Association of County and City Health Officials (NACCHO), (2015) The Role of Local Public Health in Healthcare Critical Infrastructure Protection http://nacchopreparedness.org/ National Institute of Standards and Technologies (NIST), (2014) ) Framewor k for Improving Critical Infrastructure Cybersecurity National Infrastructure Protection Plan (2013) http://www.dhs.gov/publication/nipp -2013 -partnering - critical -infrastructure -security -and -resilience Nunnally, J. C. (1978). Psychometric theory (2nd ed .). New York, NY: McGraw -Hill. Ouedraogo, Moussa; Savola, Reijo M.; Mouratidis, Haralambos; Preston, David; Khadraoui, Djamel and Dubois, Eric. (2013) "Taxonomy of quality metrics for assessing assurance of security correctness." Software Quality Journal : 67 -97. Ouyang, M. (2014). "Review on modeling and simulation of interdependent critical infrastructure systems." Reliability Engineering & System Safety 121 : 43 -60. Ouyang, M., Dueñas -Osorio, L. (2012). “A three -stage framework resilience analysis framework for urban infrastructure systems.” Structural Safety 2012; 36: 23 –31. Pearl, Judea. (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference . San Mateo, Calif.: Morgan Kaufmann Publishers. Pederson, P., D. Dudenhoeffer, et. al. . (2006). "Critical infrastructure interdependency modeling: a survey of US and international research." 92 Pettigrew, J., Ryan, J., Salous, K., Mazzuchi, T., & Dc, W. (2009). Decision -Making by Effective Information Security Managers. Pfleeger, Shari Law rence and Cunningham, Robert K., (2010) "Why Measuring Security Is Hard." IEEE Security & Privacy Magazine : 46 -54. PPD -21 (2013) Critical Infrastructure Security and Resilience , President Barack Obama Punch, K. F. (2003). Survey research : The basics . Lond on, England: Sage Publications. Queiroz, C.; Mahmood, A.; Tari, Z., (2013) "A Probabilistic Model to Predict the Survivability of SCADA Systems," Industrial Informatics, IEEE Transactions on , vol.9, no.4, pp.1975,1985 Radvanovsky, R. S. and A. McDougall ( 2013). Critical infrastructure: homeland security and emergency preparedness , CRC Press. Renooij, S. (2001). "Probability elicitation for belief networks: issues to consider." The Knowledge Engineering Review 16 (03): 255 -269. Riegel, C., Risk Assessment and Critical Infrastructure Protection in Health Care Facilities: Reducing Social Vulnerability, Retrieved March 10, 2013 From Rinaldi , S. m., Peerenboom J. p. and Kelly, T.k. (2001) "Identifying, understanding, and analyzing critical infrastructure interdependencies." IEEE Control Systems Magazine : 11 -25. Robert, Benoit (2008). Modelling interdependencies among critical infrastructures. International journal of critical infrastructures. , 4 (4), p. 392. (ISS N: 1475 -3219) Roberts, Steven (2004) "Tips and Trends for Homeland Security and Critical Infrastructure Protection," Journal of Homeland Security and Emergency Management: Vol. 1: Iss. 4, Article 405 93 Ryan, J.J.C.; Ryan, D.J., (2008) "Performance Metrics for Information Security Risk Management," Security & Privacy, IEEE , vol.6, no.5, pp.38,44. Ryan, Julie J.c.h., Thomas A. Mazzuchi, Daniel J. Ryan, Juliana Lopez De La Cruz, and & Cooke, Roger. (2012) "Quantifying information security risks using expert judgment elicitation." Computers & Operations Research : 774 -784. Ryan, J.j.c.h. (2004) "Information security tools and practices: what works?." IEEE Transactions on Computers : 1060 -1063. Salkind, N. J. (2009). Exploring research (7th ed. ). Upper Saddle Ri ver, NJ: Pearson Education. Sanders, W, (2014) . “Quantitative Security Metrics: Unattainable Holy Grail or a Vital Breakthrough Within Our Reach”, Security & Privacy, IEEE,vol 12 no.2, pp 67 -69 Santos, Joost R., Haimes, Yacov Y. and Lian, Chenyang. (2007 ) "A Framework for Linking Cybersecurity Metrics to the Modeling of Macroeconomic Interdependencies." Risk Analysis : 1283 -1297. Santos JR (2006). Inoperability input –output modeling of disruptions to interdependent economic systems. Systems Engineering pp20 –34. Satumtira G, Dueñas -Osorio L. (2010) Synthesis of modeling and simulation methods on critical infrastructure interdependencies research. In: Gopalakrishnan K, Peeta S, editors.

Sustainable infrastructure systems: simulation, imaging, and intelli gent engineering. New York: Springer -Verlag. Sikula, Nicole R., James W. Mancillas, Igor Linkov, and John A. McDonagh. "Risk management is not enough: a conceptual model for resilience and adaptation -based vulnerability assessments." Environment Systems & Decisions 35, no. 2 (2015): 219. SPSS. (2007). SPSS survey tips guide . Retrieved from: http://www.spss.com 94 Symantec 2010 Critical Infrastructure Protection Study Global Results. (2010), (October). The White House. (2013) "Executive Order -- Improving Criti cal Infrastructure Cybersecurity." http://www.whitehouse.gov/the -press -office/2013/02/12/executive -order -improving -critical - infrastructure -cybersecurity . Tversky, A., D. Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science 185(415 7): 1124 -1131. U.S. Government Accountability Office. (2011) "Cybersecurity: Continued Attention Needed to Protect Our Nation's Critical Infrastructure and Federal Information Systems." GAO -11 - 463T U.S. Government Accountability Office. (2012) "Critical Infrastructure Protection: Cybersecurity Guidance Is Available, but More Can Be Done to Promote Its Use." GAO -12 -92 U.S. Government Accountability Office. (2015) "Critical Infrastructure Protection: Measures Needed to Assess Agencies’ Promotion of the Cybe rsecurity Framework. GAO -16 -152 U.S. Government Accountability Office. (2017) "Critical Infrastructure Protection: Needs to Better Measure Cybersecurity Progress.” GAO -16-79 U.S. Government Accountability Office. (2017) "Information Security: DHS Needs to Continue to Advance Initiatives to Protect Federal Systems." GAO -17 -518T U.S. Governm ent Accountability Office. (2017 ) "Critical Infrastructure Protection: DHS Risk Assessments Inform Owner and Operator Protection Efforts and Departmental Strategic Plannin g.” GAO -18 -62 Vira, C. and Y. Y. Haimes (1983). Multiobjective decision making : theory and methodology . New York, North Holland. 95 Vugrin, E., D. Warren, et. al. . (2010). A Framework for Assessing the Resilience of Infrastructure and Economic Systems. Sus tainable and Resilient Critical Infrastructure Systems . K. Gopalakrishnan and S. Peeta, Springer Berlin Heidelberg : 77 -116. Weisberg, H. F. (1977). An introduction to survey research and data analysis . San Francisco, CA: Freeman & Co. Wright, P. D., M. J. Liberatore, et. al. . (2006). " A Survey of Operations Research Models and Applications in Homeland Security." Interfaces 36 (6): 514 -529. Written testimony of NPPD Executive Order 13636 and Presidential Policy Directive 21 Integrated Task Force Director R obert Kolasky for a House Committee on Homeland Security, Subcommittee on Cybersecurity, Infrastructure Protection, and Security Technologies hearing titled “Oversight of Executive Order 13636 and Development of the Cybersecurity Framework” (2013) Zim merman, R. (2004). Decision -making and the vulnerability of interdependent critical infrastructure . Systems, Man and Cybernetics, 2004 IEEE International Conference on, IEEE. Zio E, FerrarioE. (2013) A framework for the system -of-systems analysis of the r isk for a safety - critical plant exposed to external events. Reliability Engineering and System Safety;114:114 –25. Zio, Enrico (2016) Challenges in the vulnerability and risk analysis of critical infrastructures, In Reliability Engineering & System Safet y, Volume 152, , Pages 137 -150, ISSN 0951 -8320, https://doi.org/10.1016/j.ress.2016.02.009. 96 APPENDIX A Survey on Critical Infrastructure Protection and Resiliency Vulnerability Impact to Hospital/Patient A successful Denial of Service (DoS) attack on a Critical Infrastructure (CI) can indirectly have devastating and irreversible effects to those that depend on its services. Healthcare and public health facilities rely on various CIs in order to maintain daily operations. A cyber or physical attack on a ny interdependent CI (water, power, communications, etc. ) can indirectly have a detrimental and irreversible effect on a patient’s health. This survey is constructed to identify metrics used by hospitals to protect and/or maintain patient healthcare and t o assess the impact to a patient given a DoS or interruption to power, water or communications to a hospital. The following questions should be answered from the expert knowledge of a physician/nurse or administrator caring for a critically/gravely ill pa tient in Intensive Care Unit (ICU) totally dependent on a device serviced by power, water or communications. Thank you in advance for sharing your knowledge/expertise . No personal information will be elicited or shared in this survey. Background Informa tion (Anonymous) 1. What is your medical profession? a. Nurse b. Physician c. Administrator 2. How long have you practiced in the medical field? a. Less than 1 year b. 1-5 years c. 6-10 years d. 11 -15years e. 16 or more years 3. Do you have experience in the Intensive Care Unit (ICU)? a. Yes b. No Hospital Metrics 4. What metric is used to trigger backup POWER generator? a. Main source off for greater than 5 minutes b. Main source off for less than 5 minutes c. Unknown 97 d. No metric used 5. What metric is used to trigger backup WATER supply? a. Main source off or co ntamination detected in greater than 5 minutes b. Main source off or contamination detected in less than 5 minutes c. Unknown d. No metric used 6. What metric is used to trigger alternate COMMUNICATIONS ? a. Main source off for greater than 5 minutes b. Main source off for less than 5 minutes c. Unknown d. No metric used INSTRUCTIONS: Assessing Impact to Patient Answers to each of the following questions should add/total to 100%. Answers may be provided in the form of a check mark or percentages . An example is provided belo w. Example : Given a denial of service of POWER to a hospital, resulting in a complete shutdown (No power generator or main source power), what is the likelihood of impact to a patient depending on cardiac care (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical ✓ Degraded ✓ No Impact This answer is interpreted as “given a complete shutdown to power, the likelihood of impact to a patient’s health is very likely to have a critical impact and 0 -20% likely to have a degraded impact ”. The likelihood of no impact is assumed zero, unless answers are provided in percentages. The answer must total 100%. 98 Denial of Service (DoS) - POWER 1. Given a DoS/interruption of POWER to a hospita l, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on cardiac care (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Deg raded No Impact 2. Given a DoS/interruption of POWER to a hospital, resulting in complete shutdown (No power generator or main source power), what is the likelihood of impact to a patient depending on cardiac care (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 3. Given a DoS/interruption of POWER to a hospital, resulting in limited power (power generator only and no main source power), what is the likelihood of impact to a patient depending on dialysis (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 4. Given a DoS/interruption of POWER to a hospital, resulting in a complete shutdown (No power generator or main source power), what is the likelihood of impact to a patient depending on dialysis (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Li kely (81-100% ) Critical Degraded No Impact 5. Given a DoS/interruption of POWER to a hospital, resulting in limited services (power generator only and no main source power), what is the likelihood of impact to a patient depending on oxygen /ventilator (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 99 6. Given a DoS/interruption of POWER to a hospital, resulting in a complete shutdown (No power generator or main source power), what is the likelihood of impact to a patient depending on oxygen /ventilator (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact Denial of Service (DoS) - WATER 7. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on cardiac care (ICU) Very unlikel y (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 8. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown (No filtered/uncontaminated water o r water from main source), what is the likelihood of impact to a patient depending on cardiac care (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 9. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on dialysis (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 10. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown ( No filtered/uncontaminated water or water from main source), what is the likelihood of i mpact to a patient depending on dialysis (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 100 11. Given a DoS/interruption of WATER to a hospital, result ing in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on oxygen /ventilator (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 12. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown ( No filtered/uncontaminated water or water from main source), what is the likelihood of impact to a patient depending on oxygen /ventilat or (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact Denial of Service (DoS) - COMMUNICATIONS 13. Given a DoS/interruption of COMMS to a hospital, resu lting in limited emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on cardiac care (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 14. Given a DoS/interruption of COMMS to a hospital, resulting in a complete shutdown of COMMS (No emergency communications (i.e. ambulance to hospital, patient to nurs e/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on cardiac care (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 15. Given a DoS/interruption of COMMS to a hospital, resulting in limited emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on dialysis (ICU) Ver y unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical 101 Degraded No Impact 16. Given a DoS/interruption of COMMS to a hospital, resulting in a complete shutdown of COMMS (No emergency com munications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on dialysis (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 17. Given a DoS/interruption of COMMS to a hospital, resulting in limited emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of im pact to a patient depending on oxygen /ventilator (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 18. Given a DoS/interruption of COMMS to a hospital , resulting in a complete shutdown of COMMS (No emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on oxygen /ventilator (ICU) Very unlikely (0-20% ) Unlikely (21-40% ) Need more info (41-60% ) Likely (61-80% ) Very Likely (81-100% ) Critical Degraded No Impact 102 APPENDIX B Survey on Patient Impact due to Critical Infrastructure Service Interruption/Denial 1. Which of the follow ing best describe your job function? Answer Options Response Percent Response Count Physician 10.0% 1 Nurse 80.0% 8 Administrator 10.0% 1 answered question 10 skipped question 0 2. How long have you prac ticed in the medical field? Answer Options Response Percent Response Count Less than 1 year 0.0% 0 1-5 years 20.0% 2 6-10 years 10.0% 1 11 -15 years 20.0% 2 16 or more years 50.0% 5 answered question 10 skipped questi on 0 3. Do you have experience in the Intensive Care Unit (ICU)? Answer Options Response Percent Response Count Yes 66.7% 6 No 33.3% 3 Other (please specify) 2 answered question 9 skipped question 1 103 4. What metric is used to trigger backup POWER generator? Answer Options Response Percent Response Count Main source off for greater than 5 minutes 0.0% 0 Main source off for less than 5 minutes 50.0% 5 Unknown 50.0% 5 No metric used 0.0% 0 answered question 10 skipped question 0 5. What metric is used to trigger backup WATER supply? Answer Options Response Percent Response Count Main source off or contamination det ected in greater than 5 minutes 10.0% 1 Main source off or contamination detected in less than 5 minutes 20.0% 2 Unknown 70.0% 7 No metric used 0.0% 0 answered question 10 skipped question 0 104 6. Wha t metric is used to trigger alternate COMMUNICATIONS? Answer Options Response Percent Response Count Main source off for greater than 5 minutes 30.0% 3 Main source off for less than 5 minutes 30.0% 3 Unknown 40.0% 4 No metric used 0.0% 0 answered question 10 skipped question 0 7. Given a DoS/interruption of POWER to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on cardiac care ( ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 1 1 2 2 3 9 Degraded 1 2 4 1 0 8 No Impact 3 0 1 0 0 4 answered question 10 skipped question 0 8. Given a DoS/interruption of POWER to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on cardiac care (ICU)? Answer Options Very Unlikely (0- 20% ) Un likely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 0 0 3 7 10 Degraded 2 0 1 1 1 5 No Impact 4 0 0 0 0 4 answered question 10 skipped question 0 105 9. Given a DoS/interru ption of POWER to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on dialysis (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 2 2 5 1 10 Degraded 1 2 2 1 0 6 No Impact 2 1 2 0 0 5 answered question 10 skipped question 0 10. Given a DoS/interruption of POWER to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on dialysis (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 0 0 5 5 10 Degraded 1 2 1 1 1 6 No Impact 2 2 0 0 0 4 answered question 10 skipped question 0 11. Given a DoS/interruption of POWER to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 2 2 4 2 10 Degraded 1 1 4 1 0 7 No Impact 2 1 1 0 0 4 answered question 10 skipped question 0 106 12. Given a DoS/interruption of POWER to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 0 0 2 7 9 Degraded 2 0 2 2 1 7 No Impact 2 1 1 0 0 4 answered question 10 skipped question 0 13. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on cardiac care (ICU)? Answer Options Very Unlikely Un likely Need More Info Likely Very Likely Response Count Critical 0 2 1 4 2 9 Degraded 1 3 1 3 0 7 No Impact 2 1 0 1 0 4 answered question 10 skipped question 0 14. Given a DoS/interruption of WATER to a hospital, resulting in a comp lete shutdown (No filtered/uncontaminated water or water from main source), what is the likelihood of impact to a patient depending on cardiac care (ICU) Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 2 1 2 5 10 Degraded 2 3 0 1 1 7 No Impact 2 0 0 2 0 4 answered question 10 skipped question 0 107 15. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup w ater supply only), what is the likelihood of impact to a patient depending on dialysis (ICU)? Answer Options Very Unlikely Unlikely Need More Info Likely Very Likely Response Count Critical 0 2 1 4 3 10 Degraded 0 4 1 1 1 7 No Impact 1 1 1 1 0 4 answe red question 10 skipped question 0 16. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown (No filtered/uncontaminated water or water from main source), what is the likelihood of impact to a patient dependi ng on dialysis (ICU)? Answer Options Very Unlikely Unlikely Need More Info Likely Very Likely Response Count Critical 0 2 0 2 6 10 Degraded 2 2 0 1 1 6 No Impact 2 1 1 1 0 5 answered question 10 skipped question 0 17. Given a D oS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)? Answer Options Very Unlikely Unlikely Need More Info Likely Very Likely Response Count Critical 0 2 3 3 2 10 Degraded 1 4 1 0 1 7 No Impact 1 1 2 0 0 4 answered question 10 skipped question 0 18. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown (No filtered/unconta minated water or water from main source), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)? Answer Options Very Unlikely Unlikely Need More Info Likely Very Likely Response Count Critical 0 2 0 2 6 10 Degraded 3 3 0 0 1 7 No Impact 1 2 1 0 0 4 answered question 10 skipped question 0 108 19. Given a DoS/interruption of COMMS to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on c ardiac care (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 2 1 5 2 10 Degraded 1 4 2 0 0 7 No Impact 0 2 1 1 0 4 answered question 10 skipped quest ion 0 20. Given a DoS/interruption of COMMS to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on cardiac care (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 1 1 2 6 10 Degraded 2 1 3 0 0 6 No Impact 0 1 4 0 0 5 answered question 10 skipped question 0 21. Given a DoS/interruption of COMMS to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on dialysis (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 3 2 2 3 10 Degraded 1 5 1 0 0 7 No Impact 0 2 1 1 0 4 answered question 10 skipped question 0 109 22. Given a DoS/interruption of COMMS to a hospital, resulting in compl ete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on dialysis (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Respons e Count Critical 0 3 0 2 5 10 Degraded 2 4 1 0 0 7 No Impact 0 2 1 1 0 4 answered question 10 skipped question 0 23. Given a DoS/interruption of COMMS to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 2 2 3 3 10 Degraded 1 5 0 0 1 7 No Impact 1 2 0 1 0 4 answered question 10 skipped question 0 24. Given a DoS/interruption of COMMS to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)? Answer Options Very Unlikely (0- 20% ) Unlikely (21 -40% ) Need More Info (41 - 60% ) Likely (61- 80% ) Very Likely (81- 100% ) Response Count Critical 0 1 1 2 6 10 Degraded 2 3 1 0 1 7 No Impact 1 2 1 0 0 4 answered question 10 110 skipped question 0