Nationaal Lucht- en Ruimtevaartlaboratorium National Aerospace Laboratory NLR NLR-TP-2005-066 Prototyping interactive cockpit applications R.P.M. Verhoeven and A.J.C. de Reus This report has been based on a paper presented at the 23rd Digital Avionics System Conference, Salt Lake City, Utah, U.S.A., October 24-29, 2004. This report may be cited on condition that full credit is given to NLR and the authors. Customer: Working Plan number: Owner: Division: Distribution: Classification title: Approved by author: National Aerospace Laboratory NLR AT.1.G.3 National Aerospace Laboratory NLR Air Transport Unlimited Unclassified February 2005 Approved by project manager: Approved by project managing department: -2NLR-TP-2005-066 Summary In the early phases of a cockpit application development process prototypes are built for initial evaluation and demonstration purposes. It is in this phase that innovative ideas are made more concrete, and a base for the specification is created. The prototypes are improved by performing several design-review cycles. The iterative nature of prototyping demands a flexible and customizable design and evaluation environment. Using the experience obtained in a variety of research projects during the last decade, NLR has developed a prototyping environment that facilitates the rapid development and evaluation of avionics display formats. It supports a component based design strategy, enabling efficient reuse of previous work and facilitating parallel activities of several project team members. The resulting display formats can be rapidly deployed to the appropriate evaluation platform, ranging from desktop test environments, via cockpit mock-ups, up to high-fidelity flight simulator and even flight test aircraft. The tools have been successfully applied in several cockpit application development projects. They have allowed to drastically reduce the effort and duration involved with implementing the prototypes, so that customer research issues can be addressed earlier and in a more cost-efficient way. Furthermore, the tools enable evaluated prototypes to be used as a base for further development towards a certified cockpit application running on an avionics target system. -3NLR-TP-2005-066 Contents 1 2 3 Introduction 4 1.1 Background 4 1.2 Cockpit applications 4 Design methodology 5 2.1 Product development processes 5 2.2 Research processes 6 2.3 Supporting tool 6 Flexible prototyping 8 3.1 User requirements 8 3.2 Interpreter-based viewer 8 3.3 Integrated graphical and behavioral prototyping 9 3.4 Hybrid data-driven and event-driven communication 11 3.5 Component-based design strategy 12 4 Realistic evaluations 14 5 Base for further development 14 5.1 Using prototyping results 14 5.2 ARINC 661 compatible 14 6 Example application 15 7 Conclusion 16 References (16 pages in total) 16 -4NLR-TP-2005-066 1 Introduction 1.1 Background After the transition from electromechanical instruments to the glass cockpit, a new revolution is rapidly taking place on the civil flightdeck. The introduction of interactive cockpit applications allows the flight crew to operate the aircraft systems using a cursor control device in a way that is very similar to operating personal computers. While this potentially allows a more userfriendly working environment for the pilot, it imposes new and demanding challenges to the industry and the aeronautical research community alike. Various aircraft and avionics manufacturers are currently taking up the task of transforming their design processes that are optimized for one-way information presentation into efficient means to develop interactive cockpits. This includes changes in the interaction between avionics integrators and equipment suppliers. Manufacturers offering aircraft with truly interactive cockpits to their customers include Airbus (with the A380), Dassault Aviation (Falcon 900EX and 2000EX), and Embraer (170 and 190 series), while Boeing will likely join with the 7E7 and former Fairchild Dornier offered the 728JET until its collapse. Not surprisingly, all major avionics manufacturers offer equipment for interactive cockpits. 1.2 Cockpit applications In this paper we are using the term cockpit applications instead of cockpit displays or cockpit instruments. We feel that it better expresses the nature of the nowadays “software” flightdeck and the generally accepted view that user-interface and systems are equally important. In the electromechanical era, the flightdeck featured dedicated instruments for each system. In the current glass cockpit, information is integrated and presented in a graphical way, but essentially it is still one screen per system and dedicated hardware for each system. Modular avionics radically transforms this concept into a network of generic hardware running software applications that are specific for each system. The cockpit display system is part of this network and the pilots are operating software in a way similar to an office worker with a computer connected to a company network. In a user-centered flightdeck design method, development of the user-interface and of the underlying system is done side-by-side. They are equally important for reaching a system that is effective, efficient and comfortable to use. For instance, in a flight management system (FMS) with a very cumbersome user-interface pilots will most likely not exploit the system’s full potential. On the other hand, an FMS with a very convenient user-interface but lacking important functions will not be very helpful in optimizing the flight. -5NLR-TP-2005-066 2 Design methodology 2.1 Product development processes Design of cockpit applications usually consists of several design-review cycles (Figure 1). These cycles are intended to improve and finally validate the user interface, both in a technical sense and for efficient operation by the pilots. The aim is to get the user interface right first time, so that costly re-engineering after the product is introduced on the market can be avoided. These design-review cycles are formalized by new regulation on human factor aspects of flightdeck design. The first cycle normally starts with a very simple prototype, or a quick adaptation of an existing design. Such a design could be done on paper or using a computer presentation tool. The review is then performed utilizing the paper or electronic means. In later cycles the design is gradually improved and completed. The evaluations are also gradually more encompassing or realistic. Naturally the more complete – and often more complex – later cycles are also more costly. This way of working is also known as iterative and incremental development (IID)[1]. By performing evaluations with users early in the process, potential problems can be discovered and solved at a relatively low cost. This way, the number of problems appearing in the later – more costly – cycles can be drastically reduced. A process such as this can only be cost-effective when the prototypes can be efficiently adapted and extended throughout the complete prototyping phase. Optimally, the best prototype should also be directly usable in specification and design, and for (automatic) coding. improved display concept first display concept do establish plan display requirements check human factors evaluation act determine improvement potential Figure 1. Design-review cycle -6NLR-TP-2005-066 2.2 Research processes Although the objectives and the end product are not the same, in itself the design iterations performed in a – customer focused – research environment are not fundamentally different from those in product development. Also in a research environment a requirement exists to develop a first prototype of the humanmachine interface rapidly, at a low cost and often in a few variations. In an efficient process the most promising HMI “sketches” can then be easily extended and completed for evaluation with pilots in a more realistic environment like a flight simulator, and if needed, used as a base for the specification phase. For example, at NLR we develop cockpit human-machine interfaces on desktop PC’s and this is also the platform for informal reviews early in the process. We usually perform the first formal evaluations in a cockpit mock-up. For more advanced evaluations and for validations, we can use a full-flight research simulator or one of our aircraft (Figure 2). With a variety of evaluation platforms, it is evident that we have aimed to optimize our process, so that transitions of HMI concepts from one platform to the other require only a minimal effort. 2.3 Supporting tool To accommodate the prototyping process in a research environment, NLR has developed a prototyping tool - called Vincent 1 – which fulfils three important requirements: • Enable flexible prototyping • Support realistic evaluation • Serve as base for further development The following sections will discuss Vincent in more detail with respect to these requirements. 1 Visual Interface design of New Control concepts for Effective and Natural Task performance, also referring to Vincent van Gogh, Dutch painter, 1853-1890. -7NLR-TP-2005-066 1 2 Desktop Evaluation Environment ( “Airsim”) Cockpit Mock-Up ( “Apero”) • • • • • • Displays and control panels on one PC Typically early, low-fidelity evaluations Review with experts and team members Three LCD’s, one PC per LCD Typically early, medium fidelity evaluations Review with experts and test pilots Vincent 4 3 Laboratory Aircraft Cockpit ( “PH-LAB”) Full-Flight Research Simulator Cockpit ( “Grace”) • • • • • • One large LCD at F/ O position, one LCD in the back, PC-based Typically in-flight validations and demonstrations Evaluation by test or airline pilots Fully reconfigurable cockpit, displays PC-based Typically high-fidelity validations of mature concepts Evaluations with airline pilots Figure 2. Vincent displays in four evaluation environments -8NLR-TP-2005-066 3 Flexible prototyping 3.1 User requirements The prototype phase is an important part of cockpit application development. The flightdeck designer develops ideas and concepts that could meet the requirements and comply with the flightdeck design guide. Usually there is no single solution for the user interface. Ideas and concepts develop over time as insight in the issues grows and as a result of evaluations with test pilots or pilots from the end-user population. Often ideas and concepts are combined to arrive at a more optimal solution. All in all the prototyping phase is a highly creative process, with many informal and formal review moments and consequently frequent changes to the prototypes. Clearly, due to the highly iterative nature of the prototyping phase, it should be easy to build a prototype and straightforward to change it. More specifically, it should be possible to • build or modify graphical design, • add or change prototype behavior, • prototype symbology as well as interactive display applications, • re-use previous work, and • compare alternative designs. During the development of the prototyping tool these requirements were the focus of attention and have resulted in the following key characteristics of Vincent: • The tool is interpreter based. • The tool allows integrated graphical and behavioral prototyping. • The tool supports data-driven as well as event-driven communication. • The tool features an extensible framework of user-defined components. Two important tools used in the Vincent prototyping environment are the Vincent editor and the Vincent viewer (Figure 3). The editor is used to design prototypes. Both graphical as well as behavioral aspects of a prototype can be addressed within the editor. The viewer is used to evaluate the prototype within the desired simulation environment. 3.2 Interpreter-based viewer Because the viewer is interpreting the prototype definition as produced by the editor, there is no need for code generation and compilation to run the prototype. This significantly accelerates and simplifies the design-review cycle, and therefore considerably increases the flexibility of the prototyping process. Interpretation usually comes at a cost: run-time performance is often significantly lower than with code generator based tools. However, the Vincent viewer is highly optimized and it is possible to obtain (very) high update rates on relatively modest hardware: a standard desktop -9NLR-TP-2005-066 PC with hardware support of OpenGL is sufficient. The result is smooth display of even highly dynamic information and more than sufficient responsiveness to user inputs. The interpreter-based viewer has proven itself in a variety of cockpit application development projects. Using the viewer, it was also possible to simply exchange dynamic displays with project partners for review, evaluation and demonstration purposes. Vincent Editor Vincent Viewer Prototype Definition User Application Figure 3. Vincent tool chain with Editor and Viewer 3.3 Integrated graphical and behavioral prototyping A Vincent display prototype is described by a hierarchical scene graph structure. This type of data structure is often used in graphical programming models (i.e. VRML [2]), because it establishes a clean boundary between display representation and display rendering [3]. The scene graph contains nodes that describe objects and their properties. Objects may represent a geometrical primitive (e.g. line, polyline, rectangle, ...) which can be visualized. However, they may also represent other functionality, such as a sensor to detect user input, or a camera to control how the graphical objects should be projected on the screen. Nodes can contain fields of various types, dependent of the object property they represent. The fields are the inputs and the outputs of the nodes. Using the input fields the standard behavior of a node can be overruled. Specific fields may contain for example a position, scaling factor or visibility flag. Other specific fields may contain one or more children nodes. With the exception of the topmost root node, every node in the scene graph describing the prototype has one or more parents (directed acyclic graph). -10NLR-TP-2005-066 Vincent supports a large variety of built-in nodes, giving the HMI designer the required flexibility. One group of nodes was not mentioned yet: the traversal control nodes. An example of a powerful traversal control object is the Loop node. This node has an input field count and an output field i and its children are traversed count times during rendering. Children of the Loop node can refer to i, which is incremented after each iteration. This way rendering can be iteration-dependent. The transformation hierarchy includes all nodes and their relationships that are responsible for the graphical part of the application design. The behavior graph describes the connections between fields and the flow of events through the prototype. It is common practice during the design to build the transformation hierarchy and the behavior graph simultaneously. In the Vincent editor, the transformation hierarchy is built with drag-and drop. Behavior is added by defining the interface fields of the prototype and connecting them to the appropriate node fields within the scene graph. The relationship in between an interface field and a node field may be a direct connection, but it may also involve an expression. Expressions are useful because they allow for defining a purely functional interface. Such an interface is fully tailored to the underlying application, and independent of the coordinate system in which the graphical objects are defined. The ability to model the graphical and behavioral part of a prototype within the same environment eases the design and evaluation process significantly. For example, a Vincent Primary Flight Display (PFD) prototype (Figure 4) can be fully controllable by functional inputs, such as roll in degrees, pitch in degrees, and altitude in feet. Figure 4. PFD with purely functional interface -11NLR-TP-2005-066 3.4 Hybrid data-driven and event-driven communication Generally, two types of cockpit applications can be distinguished: symbology applications and interactive applications. Symbology applications present information one-way to the pilot. Examples of symbology display applications are the Primary Flight Display and Navigation Display (ND) as can be found on most current flightdecks. Typically symbology displays are data-driven applications. Interactive applications enable the pilot to react on the information shown by means of, for example, a toggle button, push button or combo box. This type of information exchange has an event-driven character. Some applications contain both data-driven symbology and event-driven parts. For instance, the navigation symbology in an interactive ND (Figure 5) is data-driven, while it also responds to pilot inputs with the cursor control device. Figure 5. Interactive ND prototype 2 To enable prototyping both types of application, Vincent supports a hybrid data-driven and event-driven communication model. Data-driven behavior is modeled using expressions, as discussed in the previous section. Event-driven behavior can be modeled using scripts. Each time an event occurs, the associated script will be executed. Figure 6 (next page) shows a conceptual execution model of the Vincent viewer. A design-time scene graph is built by parsing the prototype definition. The design-time graph contains the complete prototype, including the prototype behavioral described by expressions and scripts. 2 This display prototype was developed by NLR in the EU-funded MA-AFAS project. -12NLR-TP-2005-066 Synchronized with the User Application providing input data, a run-time scene graph is built by evaluating the expressions contained in the design-time scene graph, and rendered to obtain a visualization of the prototype. The pilot may inject events in the system by moving the cursor or selecting an object. These events are "transmitted" by output event fields of particular nodes. The scripts attached to these fields will be executed a-synchronously, and may cause other scripts to be fired as well. After all scripts have been executed, a new evaluate-render cycle will be made to update the prototype visualization. Design-Time Scene Graph Expressions Expression Evaluation and Scripts Run-Time Scene Graph Visualisation Script Execution Nodes Parse Prototype Definition Data Events User and User Application Figure 6 Execution model of the Vincent Viewer 3.5 Component-based design strategy By allowing for using user-defined nodes besides the built-in nodes in the scene graph, Vincent enables a component-based design strategy, which has many advantages. One of their main advantages is that components hide their internal complexity to the user (encapsulation). Only the component interface is relevant to the user of the component. User-defined components can be of any size from very small to very large, and behave just like built-in nodes. An example of a very small component is a digital readout with a specific formatting according to the value displayed. A somewhat larger example is a complete speed tape. In fact, a display itself can also be used as component in a larger structure, like a complete cockpit layout. Vincent components can be made completely self-contained, due to the ability to hold graphics as well as functional behavior. -13NLR-TP-2005-066 The ability to use components stand-alone or as part of a bigger structure without any modification, increases the flexibility of the prototyping process significantly. Each display component communicates directly with its own User Application, as if it is running stand-alone (multi-channel communication; see Figure 7 and Figure 8). Optionally, the interface of a subcomponent may be extended with interface fields exchanging information with its parent component (in this case the top-level component). Figure 7 Four displays in a larger structure User Application 1 Display (top level component) User Application 2 Component User Application 3 Component Figure 8. Multi-channel communication -14NLR-TP-2005-066 4 Realistic evaluations The Vincent viewer is a high-performance interpreter, built on the advanced graphics library OpenGL. To give an impression of the performance, a fully modeled PFD including behavior and communication (e.g. Figure 4), renders at least with 120 frames per second (fps) with today's PC hardware and graphical board; the display in Figure 7 with 30 fps or more. These performance characteristics of the interpreter enable prototype evaluation in a realistic simulation environment. Depending on the level of realism required, display prototypes can be implemented on a variety of platforms, ranging from a desktop configuration, up to high fidelity moving base flight simulators, like for example the GRACE simulator at NLR (Figure 2). Note that in case of a desktop configuration, the relevant pilot controls are also simulated using Vincent. 5 Base for further development 5.1 Using prototyping results In a product development process, prototyping results serve as a base for further development. In the IMCAD project [4], it is investigated how to improve the cockpit application development process, primarily focussing on the prototyping and specification phases. Effective use of prototyping results reduces overall development costs, and speed-up time-to-market. In the context of this project, Vincent has been made capable of exporting the prototyping results of symbology displays – in an extensible, human readable XML format – for further use in the development process. Also interactive applications requiring compliance with the new ARINC 661 standard [5] can be prototyped using Vincent. 5.2 ARINC 661 compatible The ARINC 661 standard defines two interfaces in between Cockpit Display System (CDS) and avionics equipment (user system). The first one is the interface between the avionics equipment and the display system graphics generators. The second one is a means by which symbology and its related behavior is defined. The standard relies on a basic set of graphical user interface objects, the so-called ARINC 661 HMI Widget Library. The standard describes the widgets in a purely functional way, without any reference to a visual representation, allowing any company style. Because some of these widgets enable user input, by means of cursor control devices or keyboards, interactive displays can be specified based upon the ARINC 661 standard. -15NLR-TP-2005-066 The capability of Vincent to define and instantiate graphical components with encapsulated behavior allows building an ARINC 661 compatible widget library, with a user-defined look and feel. This ARINC 661 compatible widget library can be used to prototype interactive cockpit applications. After design and evaluation, the prototype can be saved in an XML format 3 , and serve as a base for further development. 6 Example application Vincent has been used successfully in a variety of flightdeck design and display development projects, both military and civil. One of the projects exploiting much of the potential of Vincent is the flightdeck design prototype developed in the NEWSCREEN project [6]. In this interactive concept, component-based design and the capabilities of the multi-channel, interpreter-based viewer were fully exploited. This has dramatically reduced the development duration and effort and has resulted in a favorable concept (Figure 9). Figure 9. NEWSCREEN flightdeck concept 3 A draft version of this format has been proposed by the ARINC 661 Working Group, and sent to the ARINC committee for approval. -16NLR-TP-2005-066 7 Conclusion The prototyping phase of a cockpit application development process demands a flexible prototyping environment, in which prototypes can be easily adjusted and realistically evaluated in the appropriate simulation environment. The ability of Vincent to prototype graphical and behavioral aspects within a single hierarchical component-based data structure has proven to be very effective. Also the use of an interpreter to evaluate the candidate prototypes has significantly increased the flexibility of the prototyping process. Furthermore the possibility to define hybrid data-driven and event-driven communication has broadened the application domain. Finally, re-use of prototyping results in the next phases of a cockpit application development process, makes the prototyping effort even more efficient. References [1] Mark van Harmelen (editor), Object Modeling and User Interface Design, AddisonWesley, March 2001, ISBN 0-201-65789-9 [2] Virtual Reality Modeling Language, VRML, Web3D Consortium, http://www.web3d.org/ [3] Aaron E. Walsh, Understanding Scene Graphs, Dr. Dobb's Journal, July 2002 [4] Improving the Cockpit Application Development Process, IMCAD project website, http://www.nlr.nl/public/hosted-sites/imcad [5] Aeronautical Radio Inc, 2003, Arinc specification 661-1, Cockpit Display System Interfaces to User Systems [6] Three Large LCD Cockpit, NEWSCREEN project website, http://www.nlr.nl/public/hosted-sites/newscreen E-mail addresses Ronald Verhoeven, Training, Human Factors and Cockpit Operations Department, Air Transport Division, NLR, The Netherlands, verhoeve@nlr.nl Antoine de Reus, Training, Human Factors and Cockpit Operations Department, Air Transport Division, NLR, The Netherlands, dereus@nlr.nl