— This paper introduces an innovative approach to analyzing safety in the next generation of air ... more — This paper introduces an innovative approach to analyzing safety in the next generation of air traffic management systems. The proposed method is based on systems and control theory and is able to capture system design and component interaction causes that are increasingly frequent in accidents. The new methodology is applicable during the entire design lifecycle from early concept selection through final certification. Hazard analysis of a completed NextGen concept, In-Trail Procedure, is demonstrated as well as use in the early concept development of
Since its founding, NASA has been dedicated to the advancement of aeronautics and space science. ... more Since its founding, NASA has been dedicated to the advancement of aeronautics and space science. The NASA scientific and technical information (STI) program plays a key part in helping NASA maintain this important role. The NASA STI program operates under the auspices of the Agency Chief Information Officer. It collects, organizes, provides for archiving, and disseminates NASA’s STI. The NASA STI program provides access to the NASA Aeronautics and Space Database and its public interface, the NASA Technical Report Server, thus providing one of the largest collections of aeronautical and space science STI in the world. Results are published in both non-NASA channels and by NASA in the NASA STI Report Series, which includes the following report types:
One contributor to hazards in complex systems arises out of unsafe interactions among multiple co... more One contributor to hazards in complex systems arises out of unsafe interactions among multiple controllers. The basic problem is that in complex systems, hazards can be created by interactions among components that are each operating “correctly.” STPA is a new hazard analysis that includes both system hazards caused by component failures (as do the traditional analysis techniques) and also those caused by unsafe interactions among components that may not have individually failed. The first descriptions of STPA, however, did not include examples of how to handle potential problems that occur between multiple controllers. We have created an approach to identify possible unsafe interactions among multiple controllers so that the system can be designed to eliminate any ambiguity or potential for unsafe controller interactions. In this paper, we describe the analysis technique and demonstrate its use for the HTV during the critical approach phase. Once these hazardous interactions are id...
Safety should be designed into future air traffic management systems from their very conception, ... more Safety should be designed into future air traffic management systems from their very conception, which can be achieved by integrating powerful hazard analysis techniques into the general systems engineering process. The primary barrier to achieving this objective is the lack of effectiveness of the existing analytical tools during early concept development. This paper introduces a new technique, which is based on a more powerful model of accident causality—called systems-theoretic accident model and process (STAMP)—that can capture behaviors that are prevalent in these complex, software-intensive systems. The goals are to (1) develop rigorous, systematic tools for the analysis of future ATM concepts in order to identify potentially hazardous scenarios and undocumented assumptions, and (2) extend these tools to assist stakeholders in the development of concepts using a safety-driven approach. Keywords--safety; human factors; trajectory management; separation; air-ground integrated co...
component of the NASA Aviation Safety Program under contract NNL10AA13C. The views and conclusion... more component of the NASA Aviation Safety Program under contract NNL10AA13C. The views and conclusions in this report are those of the authors alone. Approval by NASA as a NASA Technical Report is still in process. 2 Electronic versions of this technical report can be obtained from
Recent issues with the Boeing 737-MAX point to challenges in software engineering and the algorit... more Recent issues with the Boeing 737-MAX point to challenges in software engineering and the algorithms that the software must implement. However, the 737-MAX is a microcosm of several other problems that continue to arise in many applications of autonomy, control, and cyber-physical systems. There is a deep coupling between the underlying physics of a system, its various modes of actuation, and the many nested or parallel control systems that comprise any complex robotic system. Many accidents – the speaker will claim all accidents – arise due to an inability to understand and then manage this coupling in a way that scales to complex systems; conversely, many benefits occur when we adequately manage and design for these factors. These issues become even more challenging with the rise and ubiquity of autonomy and automation. This talk will begin by exploring these problems, with several relevant examples to the mechanical engineering community, including aviation, launch vehicle contro...
Requirement decomposition is a widely accepted Systems Engineering practice for Requirements Engi... more Requirement decomposition is a widely accepted Systems Engineering practice for Requirements Engineering. Getting the requirements correct at the very beginning of the lifecycle is crucial for the success of engineering a correct system. This is especially the case for safety-critical complex systems, where incorrect or clashing requirements can lead to accidents. While there is a large volume of work on the formal verification for the bottom-up composition of requirements, there are very few works on how these requirements are rigorously decomposed top-down in the first place. This paper tackles this problem. Inspired by Contract-Based Design, we develop a formalism for requirement decomposition, which can mathematically guarantee a satisfactory system implementation if certain conditions are respected. A systematic methodology is then designed to semi-automatically search for the optimal sub-requirements and guarantee their correctness upon definition. The proposed approach is sup...
Safe navigation of autonomous agents in human centric environments requires the ability to unders... more Safe navigation of autonomous agents in human centric environments requires the ability to understand and predict motion of neighboring pedestrians. However, predicting pedestrian intent is a complex problem. Pedestrian motion is governed by complex social navigation norms, is dependent on neighbors’ trajectories, and is multimodal in nature. In this work, we propose SCAN, a Spatial Context Attentive Network that can jointly predict socially-acceptable multiple future trajectories for all pedestrians in a scene. SCAN encodes the influence of spatially close neighbors using a novel spatial attention mechanism in a manner that relies on fewer assumptions, is parameter efficient, and is more interpretable compared to state-of-the-art spatial attention approaches. Through experiments on several datasets we demonstrate that our approach can also quantitatively outperform state of the art trajectory prediction methods in terms of accuracy of predicted intent.
Our work focuses on modeling security of systems from their component-level designs. Towards this... more Our work focuses on modeling security of systems from their component-level designs. Towards this goal we develop a categorical formalism to model attacker actions. Equipping the categorical formalism with algebras produces two interesting results for security modeling. First, using the Yoneda lemma, we are able to model attacker reconnaissance missions. In this context, the Yoneda lemma formally shows us that if two system representations, one being complete and the other being the attacker’s incomplete view, agree at every possible test, then they behave the same. The implication is that attackers can still successfully exploit the system even with incomplete information. Second, we model the possible changes that can occur to the system via an exploit. An exploit either manipulates the interactions between system components, for example, providing the wrong values to a sensor, or changes the components themselves, for example, manipulating the firmware of a global positioning sys...
Currently, perimeter-based approaches are the mainstay of cybersecurity. While this paradigm is n... more Currently, perimeter-based approaches are the mainstay of cybersecurity. While this paradigm is necessary, there is mounting evidence of its insufficiency with respect to sophisticated and coordinated attacks. In contrast to perimeter-based security, mission-centric cybersecurity provides awareness of how attacks can influence mission success and therefore focuses resources for mitigating vulnerabilities and protecting critical assets. This is strategic as opposed to tactical perimeter-based cybersecurity. We propose MISSION AWARE, which assists in the identification of parts of a system that destabilize the overall mission of the system if compromised. MSSION AWARE starts with a structured elicitation process that leads to hazards analysis. It employs hierarchical modeling methods to capture mission requirements, admissible functional behaviors, and system architectures. It then generates evidence---attacks applicable to elements that directly correlate with mission success. Finall...
Cyberphysical systems require resiliency techniques for defense, and multicriteria resiliency pro... more Cyberphysical systems require resiliency techniques for defense, and multicriteria resiliency problems need an approach that evaluates systems for current threats and potential design solutions. A systems-oriented view of cyberphysical security, termed Mission Aware, is proposed based on a holistic understanding of mission goals, system dynamics, and risk.
As aerospace systems become increasingly complex and the roles of human operators and autonomous ... more As aerospace systems become increasingly complex and the roles of human operators and autonomous software continue to evolve, traditional safety-related analytical methods are becoming inadequate. Traditional hazard analysis tools are based on an accident causality model that does not capture many of the complex behaviors found in modern engineered systems. Additionally, these traditional approaches are most effective during late stages of system development, when detailed design information is available. However, system safety cannot cost-effectively be assured by discovering problems at these late stages and adding expensive updates to the design. Rather, safety should be designed into the system from its very conception. The primary barrier to achieving this objective is the lack of effectiveness of the existing analytical tools during early concept development. This thesis introduces a new technique, which is based on a more powerful model of accident causality that can capture ...
Author(s): Ferlez, James; Elnaggar, Mahmoud; Shoukry, Yasser; Fleming, Cody | Abstract: In this p... more Author(s): Ferlez, James; Elnaggar, Mahmoud; Shoukry, Yasser; Fleming, Cody | Abstract: In this paper, we consider the problem of creating a safe-by-design Rectified Linear Unit (ReLU) Neural Network (NN), which, when composed with an arbitrary control NN, makes the composition provably safe. In particular, we propose an algorithm to synthesize such NN filters that safely correct control inputs generated for the continuous-time Kinematic Bicycle Model (KBM). ShieldNN contains two main novel contributions: first, it is based on a novel Barrier Function (BF) for the KBM model; and second, it is itself a provably sound algorithm that leverages this BF to a design a safety filter NN with safety guarantees. Moreover, since the KBM is known to well approximate the dynamics of four-wheeled vehicles, we show the efficacy of ShieldNN filters in CARLA simulations of four-wheeled vehicles. In particular, we examined the effect of ShieldNN filters on Deep Reinforcement Learning trained controll...
Most of the basic design decisions affecting safety are made in the concept development stage of ... more Most of the basic design decisions affecting safety are made in the concept development stage of system development. Once these decisions are made, the cost of changing them later in development is often enormous and perhaps even infeasible. At the same time, most hazard analysis methods require a fairly complete design to be most useful. By the time enough design has been completed for hazard analysis to be able to identify flaws in the design, the cost of rework and changing basic decisions is great. The solution to these problems is to integrate safety tightly into the system development process from the very beginning of system conception. In this paper, we describe a process for tightly intertwining design and analysis starting in the early development stages. The process involves defining safety as a control problem (STAMP) and using model-driven development and executable requirements specifications.
EPTCS, Proceedings 3rd Annual International Applied Category Theory Conference 2020, 2020
Assuring the correct behavior of cyber-physical systems requires significant modeling effort, par... more Assuring the correct behavior of cyber-physical systems requires significant modeling effort, particularly during early stages of the engineering and design process when a system is not yet available for testing or verification of proper behavior. A primary motivation for 'getting things right' in these early design stages is that altering the design is significantly less costly and more effective than when hardware and software have already been developed. Engineering cyber-physical systems requires the construction of several different types of models, each representing a different view, which include stakeholder requirements, system behavior, and the system architecture. Furthermore, each of these models can be represented at different levels of abstraction. Formal reasoning has improved the precision and expanded the available types of analysis in assuring correctness of requirements, behaviors , and architectures. However, each is usually modeled in distinct formalisms ...
— This paper introduces an innovative approach to analyzing safety in the next generation of air ... more — This paper introduces an innovative approach to analyzing safety in the next generation of air traffic management systems. The proposed method is based on systems and control theory and is able to capture system design and component interaction causes that are increasingly frequent in accidents. The new methodology is applicable during the entire design lifecycle from early concept selection through final certification. Hazard analysis of a completed NextGen concept, In-Trail Procedure, is demonstrated as well as use in the early concept development of
Since its founding, NASA has been dedicated to the advancement of aeronautics and space science. ... more Since its founding, NASA has been dedicated to the advancement of aeronautics and space science. The NASA scientific and technical information (STI) program plays a key part in helping NASA maintain this important role. The NASA STI program operates under the auspices of the Agency Chief Information Officer. It collects, organizes, provides for archiving, and disseminates NASA’s STI. The NASA STI program provides access to the NASA Aeronautics and Space Database and its public interface, the NASA Technical Report Server, thus providing one of the largest collections of aeronautical and space science STI in the world. Results are published in both non-NASA channels and by NASA in the NASA STI Report Series, which includes the following report types:
One contributor to hazards in complex systems arises out of unsafe interactions among multiple co... more One contributor to hazards in complex systems arises out of unsafe interactions among multiple controllers. The basic problem is that in complex systems, hazards can be created by interactions among components that are each operating “correctly.” STPA is a new hazard analysis that includes both system hazards caused by component failures (as do the traditional analysis techniques) and also those caused by unsafe interactions among components that may not have individually failed. The first descriptions of STPA, however, did not include examples of how to handle potential problems that occur between multiple controllers. We have created an approach to identify possible unsafe interactions among multiple controllers so that the system can be designed to eliminate any ambiguity or potential for unsafe controller interactions. In this paper, we describe the analysis technique and demonstrate its use for the HTV during the critical approach phase. Once these hazardous interactions are id...
Safety should be designed into future air traffic management systems from their very conception, ... more Safety should be designed into future air traffic management systems from their very conception, which can be achieved by integrating powerful hazard analysis techniques into the general systems engineering process. The primary barrier to achieving this objective is the lack of effectiveness of the existing analytical tools during early concept development. This paper introduces a new technique, which is based on a more powerful model of accident causality—called systems-theoretic accident model and process (STAMP)—that can capture behaviors that are prevalent in these complex, software-intensive systems. The goals are to (1) develop rigorous, systematic tools for the analysis of future ATM concepts in order to identify potentially hazardous scenarios and undocumented assumptions, and (2) extend these tools to assist stakeholders in the development of concepts using a safety-driven approach. Keywords--safety; human factors; trajectory management; separation; air-ground integrated co...
component of the NASA Aviation Safety Program under contract NNL10AA13C. The views and conclusion... more component of the NASA Aviation Safety Program under contract NNL10AA13C. The views and conclusions in this report are those of the authors alone. Approval by NASA as a NASA Technical Report is still in process. 2 Electronic versions of this technical report can be obtained from
Recent issues with the Boeing 737-MAX point to challenges in software engineering and the algorit... more Recent issues with the Boeing 737-MAX point to challenges in software engineering and the algorithms that the software must implement. However, the 737-MAX is a microcosm of several other problems that continue to arise in many applications of autonomy, control, and cyber-physical systems. There is a deep coupling between the underlying physics of a system, its various modes of actuation, and the many nested or parallel control systems that comprise any complex robotic system. Many accidents – the speaker will claim all accidents – arise due to an inability to understand and then manage this coupling in a way that scales to complex systems; conversely, many benefits occur when we adequately manage and design for these factors. These issues become even more challenging with the rise and ubiquity of autonomy and automation. This talk will begin by exploring these problems, with several relevant examples to the mechanical engineering community, including aviation, launch vehicle contro...
Requirement decomposition is a widely accepted Systems Engineering practice for Requirements Engi... more Requirement decomposition is a widely accepted Systems Engineering practice for Requirements Engineering. Getting the requirements correct at the very beginning of the lifecycle is crucial for the success of engineering a correct system. This is especially the case for safety-critical complex systems, where incorrect or clashing requirements can lead to accidents. While there is a large volume of work on the formal verification for the bottom-up composition of requirements, there are very few works on how these requirements are rigorously decomposed top-down in the first place. This paper tackles this problem. Inspired by Contract-Based Design, we develop a formalism for requirement decomposition, which can mathematically guarantee a satisfactory system implementation if certain conditions are respected. A systematic methodology is then designed to semi-automatically search for the optimal sub-requirements and guarantee their correctness upon definition. The proposed approach is sup...
Safe navigation of autonomous agents in human centric environments requires the ability to unders... more Safe navigation of autonomous agents in human centric environments requires the ability to understand and predict motion of neighboring pedestrians. However, predicting pedestrian intent is a complex problem. Pedestrian motion is governed by complex social navigation norms, is dependent on neighbors’ trajectories, and is multimodal in nature. In this work, we propose SCAN, a Spatial Context Attentive Network that can jointly predict socially-acceptable multiple future trajectories for all pedestrians in a scene. SCAN encodes the influence of spatially close neighbors using a novel spatial attention mechanism in a manner that relies on fewer assumptions, is parameter efficient, and is more interpretable compared to state-of-the-art spatial attention approaches. Through experiments on several datasets we demonstrate that our approach can also quantitatively outperform state of the art trajectory prediction methods in terms of accuracy of predicted intent.
Our work focuses on modeling security of systems from their component-level designs. Towards this... more Our work focuses on modeling security of systems from their component-level designs. Towards this goal we develop a categorical formalism to model attacker actions. Equipping the categorical formalism with algebras produces two interesting results for security modeling. First, using the Yoneda lemma, we are able to model attacker reconnaissance missions. In this context, the Yoneda lemma formally shows us that if two system representations, one being complete and the other being the attacker’s incomplete view, agree at every possible test, then they behave the same. The implication is that attackers can still successfully exploit the system even with incomplete information. Second, we model the possible changes that can occur to the system via an exploit. An exploit either manipulates the interactions between system components, for example, providing the wrong values to a sensor, or changes the components themselves, for example, manipulating the firmware of a global positioning sys...
Currently, perimeter-based approaches are the mainstay of cybersecurity. While this paradigm is n... more Currently, perimeter-based approaches are the mainstay of cybersecurity. While this paradigm is necessary, there is mounting evidence of its insufficiency with respect to sophisticated and coordinated attacks. In contrast to perimeter-based security, mission-centric cybersecurity provides awareness of how attacks can influence mission success and therefore focuses resources for mitigating vulnerabilities and protecting critical assets. This is strategic as opposed to tactical perimeter-based cybersecurity. We propose MISSION AWARE, which assists in the identification of parts of a system that destabilize the overall mission of the system if compromised. MSSION AWARE starts with a structured elicitation process that leads to hazards analysis. It employs hierarchical modeling methods to capture mission requirements, admissible functional behaviors, and system architectures. It then generates evidence---attacks applicable to elements that directly correlate with mission success. Finall...
Cyberphysical systems require resiliency techniques for defense, and multicriteria resiliency pro... more Cyberphysical systems require resiliency techniques for defense, and multicriteria resiliency problems need an approach that evaluates systems for current threats and potential design solutions. A systems-oriented view of cyberphysical security, termed Mission Aware, is proposed based on a holistic understanding of mission goals, system dynamics, and risk.
As aerospace systems become increasingly complex and the roles of human operators and autonomous ... more As aerospace systems become increasingly complex and the roles of human operators and autonomous software continue to evolve, traditional safety-related analytical methods are becoming inadequate. Traditional hazard analysis tools are based on an accident causality model that does not capture many of the complex behaviors found in modern engineered systems. Additionally, these traditional approaches are most effective during late stages of system development, when detailed design information is available. However, system safety cannot cost-effectively be assured by discovering problems at these late stages and adding expensive updates to the design. Rather, safety should be designed into the system from its very conception. The primary barrier to achieving this objective is the lack of effectiveness of the existing analytical tools during early concept development. This thesis introduces a new technique, which is based on a more powerful model of accident causality that can capture ...
Author(s): Ferlez, James; Elnaggar, Mahmoud; Shoukry, Yasser; Fleming, Cody | Abstract: In this p... more Author(s): Ferlez, James; Elnaggar, Mahmoud; Shoukry, Yasser; Fleming, Cody | Abstract: In this paper, we consider the problem of creating a safe-by-design Rectified Linear Unit (ReLU) Neural Network (NN), which, when composed with an arbitrary control NN, makes the composition provably safe. In particular, we propose an algorithm to synthesize such NN filters that safely correct control inputs generated for the continuous-time Kinematic Bicycle Model (KBM). ShieldNN contains two main novel contributions: first, it is based on a novel Barrier Function (BF) for the KBM model; and second, it is itself a provably sound algorithm that leverages this BF to a design a safety filter NN with safety guarantees. Moreover, since the KBM is known to well approximate the dynamics of four-wheeled vehicles, we show the efficacy of ShieldNN filters in CARLA simulations of four-wheeled vehicles. In particular, we examined the effect of ShieldNN filters on Deep Reinforcement Learning trained controll...
Most of the basic design decisions affecting safety are made in the concept development stage of ... more Most of the basic design decisions affecting safety are made in the concept development stage of system development. Once these decisions are made, the cost of changing them later in development is often enormous and perhaps even infeasible. At the same time, most hazard analysis methods require a fairly complete design to be most useful. By the time enough design has been completed for hazard analysis to be able to identify flaws in the design, the cost of rework and changing basic decisions is great. The solution to these problems is to integrate safety tightly into the system development process from the very beginning of system conception. In this paper, we describe a process for tightly intertwining design and analysis starting in the early development stages. The process involves defining safety as a control problem (STAMP) and using model-driven development and executable requirements specifications.
EPTCS, Proceedings 3rd Annual International Applied Category Theory Conference 2020, 2020
Assuring the correct behavior of cyber-physical systems requires significant modeling effort, par... more Assuring the correct behavior of cyber-physical systems requires significant modeling effort, particularly during early stages of the engineering and design process when a system is not yet available for testing or verification of proper behavior. A primary motivation for 'getting things right' in these early design stages is that altering the design is significantly less costly and more effective than when hardware and software have already been developed. Engineering cyber-physical systems requires the construction of several different types of models, each representing a different view, which include stakeholder requirements, system behavior, and the system architecture. Furthermore, each of these models can be represented at different levels of abstraction. Formal reasoning has improved the precision and expanded the available types of analysis in assuring correctness of requirements, behaviors , and architectures. However, each is usually modeled in distinct formalisms ...
Uploads
Papers by Cody Fleming