Early Validation and Verification of System Behaviour in Model-based Systems Engineering: A Systematic Literature Review

Although all the analysed papers regard the concept of V&V at an early stage of development, few papers explicitly state what that means in their context. Instead, most of the papers implicitly infer that the targeted V&V takes place at some point of system design or requirements elicitation, often referring to the INCOSE definition that states that verification and validation begin in the conceptual design phase [88]. Of the papers explicitly describing the phase or context of early V&V, a majority refers to the design phase and, as previously stated, often refers to the INCOSE definition of the “Conceptual design phase.” Apart from the design phase, some papers argue that early V&V targets the “requirements phase,” and some authors report that their solution targets both the requirements and design phases. Figure 5 visualises the target phase of the solutions among the papers.

In more detail, 107 of the 149 publications (71.8%) report that the proposed solution applies for the design phase, while 29 (19.5%) report that the solution exists in the requirements phase. Thirteen (8.7%) of the papers report that their solution covers both requirements and design and tend to be large in scope. As an example, Lemazurier et al. [P28] use requirement boiler-plates as a starting point to leverage five unique domain-specific languages and several views to generate a functional architecture. Bouffaron et al. [P44] instead define an iterative process of system refinement that spans several stages of development. In both cases, there is a large emphasis on the process, and the solutions play a supporting technical role.

From the extracted data, we can see that authors consider system requirements and system design as both targets for early system behaviour V&V. While few authors explicitly define early V&V, it is clear that the majority of them considers the topic of early validation to regard system design rather than system requirements. Moreover, the problem of “will a particular design meet the requirements?” prevails over “will a particular set of requirements capture the system of interest adequately?” Eventually, when papers deal with both requirements and design phases they use the available information to perform cross-checks, notably for better understanding the system and reason about constraints [P4, P48, P149]; to raise the quality of models and reduce “bad smells” [P15, P28, P53, P95]; to improve traceability and understand artefact dependencies [P28]; to validate non-functional requirements [P44, P60, P96]; and to reason about tradeoffs or to reduce the design space [P51, P63].

To understand the motivations for doing early V&V, we extracted from each paper the reason(s) the authors use to motivate their activities. The exact extractions are found in the replication package, while Figure 6 summarises the findings. In particular, it displays all the motivations mentioned at least twice in the papers, while those mentioned only once are valued as “Other” in the figure.

The reasons listed in Figure 6 can partially overlap, and many papers report multiple reasons for performing V&V activities. The most common motivation for performing early V&V is to ensure a desired level of quality for the design before proceeding with the implementation. This is followed by reducing risks with late flaw detection, reducing time to market, reducing risk for incomplete requirements, and exploring system behaviour before implementation. In this respect, the predominant motivations seem to directly target the reduction of risks associated with introducing errors or creating incomplete specifications in the early phases, which should lead to a more streamlined process. This is also confirmed by the fact that reducing time to market is an often quoted motivation for the activities performed by authors. Other potential risks to be prevented are those caused by sub-optimal (or even wrong) design decisions that could impact critical quality attributes of the system. In this respect, several of the more frequently mentioned reasons are related to performances and dependability (safety in particular).

A set of papers does not clearly motivate the reasons for employing their solutions targeting early V&V; we note that many of these papers describe more theoretical works without any real target case study, which is perhaps why there is a lack of motivation for the activities. For example, Kahani and Cordy discuss bounded verification for state machines in a train system controller [P40]. However, they do not discuss their solution motivation at length, as they deem the concepts already well established. Similarly, Liu et al. [P138] discuss Assume-Guarantee Reasoning in the context of scheduled components and present a general theory instead without a concrete motivation. In other words, these research works deal with techniques that might support early V&V solutions but do not explicitly mention concrete usage scenarios.

RQ1 discussion: To summarise, while there is some spread in the extracted motivating reasons, there seems to be a consensus about V&V in the earlier phases of development as being key to minimising development costs and reducing risks with flaws later in the development lifecycle. Although the main category of early V&V motivation can be formulated as “anticipating V&V before implementation” and is perhaps what is to be expected, it does not even map to half the papers.¹³ Apart from the more expected results, commonly reported motivations include “reducing risks associated with faulty design or requirements,” “performing tradeoff studies,” and “improving communication and integration between system aspects.” Most of the reported motivations for V&V are well in line with SE state of the art and practices [88], in which MBSE is seen as means of going earlier with the involved activities while maintaining the necessary rigour. Additionally, the authors mostly regard their solutions as situated in the design phase, considering the requirements phase to a smaller extent. Moreover, phases past design are not represented to any significant extent in the paper solutions. The extracted data indicate that MBSE is reaching maturity for using methods of analysis in system design with significant benefits, and it may also signify that the requirements phase has a lower need for added capabilities to current methods. However, the lower amount of papers targeting the requirements phase could also highlight that solutions need to be more mature, something reflected in the lower representation of industrial cases in the selected papers targeting the requirements phase. We also observe that few papers seem to target both the requirements and design phases, probably due to the issues related to connecting different artefacts (and corresponding tools), often reported as a weak point for MBSE [41].

Perhaps, more interesting is what is not reported by the authors to any significant extent, notably aspects such as re-use (of the V&V artefacts for other phases of the development process as well as for other projects) and improved traceability between development stages. This is surprising, since the mentioned aspects are often argued to be strong points for MBSE adoption [41]. In the same fashion, there is little discussion on the broader integration of the V&V activities in the SE landscape, which is argued as a benefit of MBSE. Notably, there are few mentions of the digital thread [80], which can be enabled with model-based methods. In conjunction with this, few papers discuss cross-domain integration, which also is something to be expected when leveraging abstraction. In essence, the solutions seem to overall lack a holistic motivation; that is how early V&V supports parallel and down-stream activities.

Figure 7 shows the languages utilised in the solutions to describe system behaviour. The category “Other” refers to entries only represented once in the extraction that are not created specifically for the solution, which instead is represented by “Custom language.” When we consider the implementation of the solutions, we differentiate between tools, languages, and formalisms (where possible) based on existing classifications as presented in Section 4.

The data in Figure 7 are based on the reported languages/formalisms from the analysed papers. Here it is worth noting that the input varied a lot in detail. Notably, some papers state “SysML” without specifying or showing what sub-set is utilised. Similarly, different types of diagrams or means of representing behaviour are shared among the various languages, such as state-based formalisms. Moreover, the selected papers reported languages or formalisms in an inconsistent way. These issues do not allow us to perform meaningful clustering of the results, and therefore we limit the reporting of the extraction results to the languages, which all analysed papers mention.

While there is a large spread of languages to represent system behaviour in the early phases of development, it is clear that SysML is by far the most common language, and much of the language is utilised, especially the behavioural diagrams and block diagrams. Figure 8 illustrates the sub-sets of SysML utilised in the papers adopting SysML as part of the proposed solution. We note that the distribution of diagram types is found in a similar way for UML and other UML profiles.

When considering the sub-parts of SysML utilised in the selected papers, activity diagrams and state machine diagrams are the most popular means of using SysML for describing system behaviour; particularly, it is utilised in many papers aiming for automated translations. For example, Staskal et al. [P140] map SysML activity diagrams to the symbolic model checker nuXmv, and Mahani et al. [P36] similarly map SysML state machine diagrams to the NuSMV model checker. Many solutions use more than one diagram of the language to describe the behaviour, and often it is not entirely clear what is used and what is not by simply reading the paper. Nonetheless, of the four diagrams classified as behavioural diagrams in the SysML standard, activity and state machine diagrams are preferred over use case diagrams and sequence diagrams. Besides, one paper argues that the solution utilises the entire SysML specification [P114], and four others that all of the behavioural diagrams are utilised [P12, P46, P91, P122].

SysML, as a general-purpose language, aims to be a solution for all types of systems. Further, it is evident that UML and languages related to UML, such as MARTE, are among the most commonly used languages and formalisms apart from SysML (which indeed has been typically implemented as a UML profile until the recent version SysML v2). This might indicate that general-purpose languages such as SysML and UML are suitable for describing early systems specifications in general and behaviour in particular. Nonetheless, a large number of papers utilise a means of description that is only observed once in the selected papers, indicating a large variety of domains and possibly the need for domain-specific support. For example, Supremica is used by Markovski [P26], which extends finite automata to handle large complex industrial systems, KARMA used by Ding et al. [P78] to unify formalisms across several MBSE models and simulations, or SWRL used by Chen et al. [P95] in combination by OWL to leverage ontology reasoning for verification.

A few papers propose custom languages or formalisms in their solutions. Miao et al. [P1] present a Python-like custom informal language for their system descriptions. Lemazurier et al. [P28] utilise a domain-specific language for nuclear power plants, along with several other domain-specific languages, to manage their system descriptions. Similarly, Deng et al. [P48] utilise a custom language and environment for mechanical product design, and Stachtiari et al. [P60] use their custom language for describing satellites. Singh and Muller [P80] use a tool-based approach to describe their system of interest based on needs observed in their industrial context of manufacturing systems. Bernaerts et al. [P126] use a specific type of model for describing safety-related aspects in the automotive domain. Miyazawa et al. [P99] use their custom state machine formalism for describing robotic systems. Zhang et al. [P141] propose an integrated intelligent modelling and simulation language to cope with the combination of domain specific models in System of Systems.

To present the results from this data extraction, we differentiate between the reported languages and the reported formalisms. Analogously to the behaviour description, authors rarely report in a precise way how a language is implemented or what parts are employed for what purpose. Further, some papers report on the particular formalisms employed while others do not. In particular, Figure 9 shows the reported languages and their frequency for analysing behaviour, where “Other” groups the entries reported only once that are not custom made for a particular solution; Figure 10 depicts the reported formalisms for analysing the behaviour.

Figures 9 and 10 clearly show that languages and formalisms used for analysis are typically case-dependent, since a large majority of the entries is represented only once or twice in the papers. In this respect, many reported languages and formalisms serve narrow/domain-specific purposes that are not easily portable to a more general case. Examples include languages such as Event-B [P24], OWL [P104], LabVIEW [P97], or Sabotage [P137]. In addition, despite the widespread use of SysML- and UML-based languages for the system description, there is no evidence about potential “standard” analysis approaches for early V&V of specific properties.

Apart from the case dependent solutions, some general-purpose languages and formalisms are frequently used, namely SysML, MATLAB/Simulink, Modelica, Petri nets, and various state-based formalisms. Moreover, it is interesting to notice that there exists a group of solutions presented as implementation agnostic with respect to the particular type of formalism and language used for the analysis [P77, P94, P95, P125]. Friedl et al. [P77] argue that their solution can be adapted to cover various case-dependent languages. Similarly, the solution from Castet et al. [P94] involve ontologies and is presented as being adaptable in terms of languages, where an implementation example is given in the Modelica language. Chen et al. [P95] also utilise a method of automatically creating specific design ontologies and rules from requirements and argue that the target implementation depends on the context considered for the system under study. Damm et al. [P125] showcase a method for contract-based virtual integration testing and consider it to be language and tool agnostic. Similarly, as for languages used for behaviour description, there are a few custom languages and formalisms, of which two overlapping with custom languages for describing behaviour [P1, P28]. Moreover, Kang et al. [P4] extend previous work to analyse EAST-ADL models, and Brandstetter et al. [P59] use validation rules for automation process software requirements validation that is not tooling dependent.

Although SysML is the primary language used to represent system behaviour, it is seldom used for analysis. More in general, only in 24 cases of all the papers is the language or formalism for representation and analysis are the same, where SysML is used 13 times [P22, P31, P50, P51, P55, P58, P63, P70, P101, P104, P107, P114, P130, P147]. As a consequence, in all the other cases a transformation (which can be composed of several sub-transformations) is required between the different languages and formalisms, at least to translate concepts from the representation language toward the analysis one. In more complex scenarios, such transformation would be used to extract and synthesise the necessary information to perform the analysis due to early V&V.

We note that 94 of the papers (63.0%) utilise fully automated transformations, while 19 (12.7%) solutions utilise semi-automated transformations, and 12 papers (8%) use some form of manual transformation. We note that rarely do authors provide proof of transformation correctness in case of automated solutions, possibly highlighting a gap in the works. Moreover, as previously mentioned, 24 (16.1%) use no transformation. The results from the extraction indicate that if a transformation is performed, then it is mostly done via automatic means; otherwise, either no transformation or semi-automatic means are employed. Singh and Muller use Dynamic A3 Architectures as an approach for validation leveraging tool support without the need for transformations [P80], particularly by promoting view management and cross-communication between teams via integrated tool capabilities, early validation is achieved by increased communication, collaboration, and integration between engineering teams. Farooq models directly in Simulink and leverages the tool capabilities for simulation and verification [P84]. Since some of the proposed solutions utilise several languages and formalisms in both the description and analysis, semi-automated transformations also include cases where some of the multiple translations are automatic while some are not. For example, González et al. [P62] leverage co-simulation mechanisms in conjunction with SysML models and only transforms a sub-set of the model information to MATLAB equations. Friedl et al. [P77] discuss the use of SysML architecture models in a process for model integration between system architects and model experts and highlights a semi-automated guided approach for integration. Only few papers rely on a completely manual approach for mapping different languages and formalisms [P1, P30, P43, P44, P48, P59, P65, P85, P120, P123, P127]. It is interesting to note that these solutions tend to be reported as not validated in an industrial context by the publication authors [P30, P43, P44, P48, P65, P120, P123, P127] and targeting the requirements phase; vice versa, the solutions deploying automated transformations are often validated in industry and do not target requirements.

RQ2 discussion: When describing a system’s behaviour, there is an apparent representation of SysML and UML or other types of UML profiles, such as MARTE. The prominence of these languages is not surprising and has been reported previously in other types of reviews [21, 60, 66]. Of the different types of behavioural diagrams in SysML, activity diagrams and state machine diagrams are the most commonly used. However both use case diagrams, and sequence diagrams are used significantly less, seemingly not good candidates for describing the systems at this stage to enable V&V. However, as opposed to the description being uniform in the language, analysis mechanisms vary a lot. Indeed most papers utilise languages or formalisms only found once or twice in the set of selected papers. The most-often-reported languages are SysML, Simulink, and Modelica. Petri net diagrams (with various formalisms), UPPAAL, and other similar languages are also used often. The mappings between the languages in the cases where the behaviour representation and analysis use different means (which is mostly the case) show a clear tendency toward automated or semi-automated solutions. This is consistent with the pragmatics of model-based development (and consequently of MBSE) [78], since handling such discontinuities by hand would introduce accidental complexity to the solution, making its adoption difficult and even not possible at all.

In a broader perspective, the wide adoption of SysML- and UML-based languages for behaviour description might confirm their status of de-facto standards for early systems’ modelling. In fact, keeping those languages for early behaviour description eases the technology transfer for the developed analysis mechanisms by avoiding learning new languages (and also adopting new tools). However, as SysML- and UML-based languages are general purpose, they often do not convey enough support for domain-specific analysis, thus requiring information extraction/translation toward semantic domains in which the analysis can be performed. Relying heavily on such technologies introduces further complexity into the process, which might make industrial adoption a larger challenge.

While we see some expected results from the analysis, there is at the same time a lack of many important topics. Relying on model-transformations to create analytical models creates a relevant dependency on these transformations, but there is little discussion on the viability of transformations. Similarly, the notion of consistency management across different languages in addition to interoperability and scalability is mostly missing. Implementing MBSE in industrial contexts will often be reliant on large tool-chains, relying on a set of model transformations in a landscape of changing tools, standards, and users is a considerable risk. For languages not using transformations, many are implemented in advanced tooling like Simulink or integrated MBSE toolkits like Cameo Systems Modeller or MagicDraw, hinting at these solutions being more closely tied with industry needs. Overall, the large range of analytical languages and notations proves powerful flexibility but, however, introduces complexity in the process, which is a considerable tradeoff.

Figure 11 reports the V&V analysis methods and their frequency as extracted from the papers. Similarly to what done previously, we do not display entries only represented once in the papers and group them as “Other” in the figure.

The most frequently reported method is simulation, which for some papers is also referred to as model execution, virtual testing, and co-simulation. The papers not utilising simulation employ many different means of analysing models, often through static analysis methods. More common types of analysis are model checking and manual inspection/review. There exists some overlap between the reported techniques, partly due to some solutions employing several means of analysing system behaviour or because of solutions being broad in the sense that several aspects are analysed either in parallel or through different steps. As an example, Quadri et al. [P81] discuss a framework with several interconnected components for V&V on SysML/MARTE models, including simulation, compile time visualisation, and temporal property checks. Similarly, Liu et al. [P16] introduce a methodology to leverage a sub-language of AADL for simulation, schedulability analysis, syntax analysis, lexical analysis, and more.

Methods with fewer entries are usually devoted to domain-specific analysis, and many of the reported entries relate to techniques for reasoning about or analysing the models directly without any need for execution. Examples include program analysis [P13], compositional verification [P138], model reasoning [P116], and correctness by construction [P60].

This section illustrates the results collected due to the data extraction related to question 8 in Table 1. To further understand early V&V from the authors’ perspective, we map the results of interest in Figure 12. Again, we do not include entries only represented once, which are instead classified as “Other.”

Similarly to previous results, there is a large spread of interest in the analysis feedback and outcomes. Even more, the results for this question are influenced by the lack of harmony and consistency in the terms used and the presentation of data from the authors, which is likely due to the broad range of domains involved in the selected papers. Execution traces refer to works using the traces of execution themselves for analysis, for example, Delvi et al. [P74] plot execution traces to compare and evaluate energy consumption, while Kotronis et al. [P100] visualise and compare traces for railway system configuration performance. Safety features can include, for example, guarantees of execution as in the case of Dragomir et al. [P148], safety assessments from Fault Trees or Failure Mode and Effect Analysis in the case of Krishnan et al. [P139], or more comprehensive V&V safety analysis through a set of techniques as in the case of Bozzano et al. [P67]. Inconsistency can refer to inconsistency between different system behavioural views, as in the case of Duhil et al. [P58], or between different tools and notations belonging to traditionally separated teams as in the case of Gregory et al. [P117]. Intended functional behaviour can refer to simulation to validate the execution of generated code like for Bocciarelli et al. [P10] or to detect design and integration errors with model-checking as in the case of Braspenning et al. [P88]. Functional requirements verification, however, can refer to the explicit requirement satisfaction checks as in Kang et al. [P4] or in Anwar et al. [P86], where SystemVerilog assertion code is generated for a particular SoI based on the design requirements defined using OCL, which can be leveraged during the eventual detailed system design for requirements verification.

Domains such as power plants [P44], canal/waterway systems [P85], web applications [P132], and medical systems [P140] have varied vocabularies compared to the more represented domains such as aerospace or automotive. Nevertheless, it is possible to identify some clear trends, such as execution traces and intended functional behaviour being of high interest. Similarly, results commonly obtained through static analysis, such as liveness, inconsistency, deadlock-freeness, reachability, and so on, are also reported often.

A few papers argue that their analysis approaches are adaptable to the application of interest for a specific context, and as such the results of interest might vary depending on the utilisation scenario. Notably, Anwar et al. [P41] present a meta-model for modelling FPGA-related concepts and argue that their solution can be used to generate test benches for various purposes. Votintseva et al. [P47] reason about multi-domain systems and propose solutions for managing cross-domain analysis, suggesting that their approach depends on the chosen domains and applications. Kaslow et al. [P55] use various inspection forms to perform their analysis, which is argued to be adapted for the stakeholder needs. González et al. [P62] propose a means of using co-simulation for early V&V and argue that the results of interest depend on the evaluation context. Hecht and Chen [P130] utilise various query-based operations on SysML models and argue that this analysis should be adapted for the particular system of interest and corresponding models. Zhang et al. [P141] use their custom language X with a prototype tooling that covers an extensive array of potential analysis, and as such, the results vary accordingly.

This section discusses the outcomes of the data extraction related to question 9 in Table 1. We present a categorisation of tools used in Figure 13 and refer to the appendix for an explicit mapping of tools to solutions.

Consistently with previous results on languages for system description and V&V analysis, the number of tools used is extensive, and most of the tools are found only once or twice among the analysed papers. The more common categories are integrated MBSE toolkits, graphical programming/simulation environments, model-checkers, modelling frameworks, and simulation toolkits. The most commonly used individual tools are MATLAB/Simulink with corresponding libraries and EMF-based solutions. EMF is one of the leading free/open source platforms for modelling, which makes it an attractive tool/platform for academic investigations [82]. MATLAB/Simulink, in a similar fashion, is suitable for both academics and industry and has a rich history of applications in the model-based community [4]. Apart from these tools, many proprietary SysML editors are commonly used for their integrated analysis capabilities, eliminating, at least partially, the need for different tools for describing and analysing behaviour. In some cases, the tools are not presented [P12, P31, P37, P38, P43, P54, P66, P92, P98, P104], and other times it is argued that the solution is not tool dependent [P47, P64, P76, P95, P101, P109, P125, P134]. Moreover, some authors have proposed custom tools or environments for their analysis [P27, P48, P68, P82, P87, P141, P144].

RQ3 discussion: Simulation is by far the most commonly used means of analysis. This perhaps indicates that for meaningful analysis at early stages, it is required to have advanced means of analysis, particularly for dynamic behaviour. Apart from simulation, there is a broader spread of different methods that can be reduced to model checking or automated reviews/inspections.

The results of interest are typically tightly coupled with the domain, the adopted languages, the tools, and so forth, hence showing a large variety of target properties. More explicit and shared set of results of interest can be observed when model checking is adopted, namely freedom of deadlock, liveness, safety, and reachability. In general, the embedded systems domain seems to have a clearer view of the process and scope of early V&V, which is reflected in the more compact set of tools, languages, and results compared to the other types of domains. Moreover, the embedded systems domain shows some maturity regarding early validation practices, with earlier publications in the observed timeline for early V&V.

Another interesting aspect is that models describing the behaviour in a somewhat uniform way, often in SysML, can lead to a wide array of analyses; this shows the powerful flexibility of the semi-formal nature of general-purpose languages like SysML and UML profiles. Of course, such semi-formal descriptions often entail the need of transformations to more structured representations for analysis, which increases the complexity and reduces the freedom of modelling, as automatic transformations require structured formatting.

The tooling reported for analysis in the papers is the category with the most significant spread observed. Very few tools are seen as good enough for the general audience of MBSE. Of the more frequent entries, Eclipse/EMF, MATLAB/Simulink, Papyrus, OpenModelica, and UPPAAL are the more commonly reported tools. Some tools that do not perform transformations between languages are also represented in the papers, like MagicDraw and Cameo Systems Modeler. These latter two are tools used in solutions where early analysis is performed directly on SysML diagrams thanks to functionalities embedded in the tooling. Moreover, as expected, these solutions are often observed in industrial applications, where it can also happen that the tooling is customised on-demand to perform specific tasks.

While tooling is tightly connected with MBSE, it is surprising that so few solutions claim to be tool agnostic. In fact, if tooling is central, then MBSE likewise considers methods, methodology, and languages. However many of the solutions rely on tool-specific analysis, regardless if the solution is extensive or not. Academic tools tend to be more compact and openly accessible for the best impact, while industrial tooling should often integrate into larger processes and there is a reluctance on openly available tooling as it rarely is scalable and maintainable for large enterprises in addition to intellectual property protection concerns. Therefore, the fact that much of the tooling is very case-specific feeds into the known problems of interoperability and maintainability.

Figure 14 displays the domains identified in the selected papers: There is a multitude of domains observed in the papers, of which the most prominent are Aerospace and Avionics.¹⁴ Other domains with a significant presence are embedded systems, cyber-physical systems, safety critical systems, automotive, and railway. Finally, there is also a large category of domains only reported once in the papers, such as nuclear power plants [P28], canal systems [P85], web applications [P132], and cloud computing [P134].

Aerospace is somewhat expected to be one of the most commonly reported and investigated domain given the complexity of the developed systems and the historical relevance of this field. Indeed, there are prominent promoters for MBSE, such as NASA, which produce quality research on the topic [42, 70]. Similarly, embedded systems are quite mature regarding formal methods and model checking [54, 85], which are often reported in the papers. Moreover, the automotive and railway industry have strong foundations on model-based practices [7, 24], and standards such as AUTOSAR exist for automotive [32]. Cyber-physical and safety critical systems are broader categories than previously mentioned ones, but we observe a strong presence regardless. In this case, CPSs typically benefit from the unified view models provide (e.g., for integration analysis purposes) while safety critical systems require early analysis for providing the necessary evidence that systems will meet the imposed requirements.

Of the analysed papers, 117 contain domain-specific solutions (78.5%), 16 (10.7%) papers argue to be domain independent, and 16 (10.5%) illustrate a domain-specific solution but make a case for it being extensible for more domains. Examples include Bocciarelli et al. [P10], which describes an approach based on the IEEE 1516-2010 - HLA standard and apply it concretely in the SoS domain but argue for wide applicability. Similarly, Stewart et al. [P17] describe a process for safety analysis in the aerospace domain but argue that it can be extended for any safety critical domain such as automotive or nuclear power plants. In the same way, Herzig et al. [P30] apply a SysML-based simulation analysis method on a telescope use case but argue that the approach can be extended for other domains. The fact that such a large majority of papers are domain-specific and many different domains are reported is consistent with the broad range of solutions proposed in the analysed papers.

Of the non-domain-specific papers, only three are validated in an industrial setting [P1, P34, P105], indicating that such solutions might not have the necessary maturity required for industry. The three papers discuss their applications in at least two domains. For example, Cheng et al. [P34] discuss an approach for automatic analysis of multi-view architectures and present both an automotive and micro-service cloud application.

This section answers question 13 from Table 1 by extracting whether the solution was validated in an industrial setting or not and what kind of validation is discussed in the selected papers.

Fifty-one papers (34.2%) evaluate their results in an industrial setting, which showcases a strong industrial presence. We re-emphasise here that the validation of the approach is extracted from the publication itself, and hence we rely on the definition adopted by the authors.

One hundred fourteen (76.5%) of the analysed papers use some form of running example as defined by Shaw [79] for the validation of the solution; in particular, here we group toy examples with slice of life examples. A sub-set of contributions, 18 (12%), present empirical observations of the solutions [P3, P4, P5, P11, P14, P40, P49, P60, P62, P66, P67, P80, P87, P103, P110, P134, P149]. For example, Mens et al. [P3] introduce a new method for testing and validating state-charts, and part of their evaluation is a controlled user study to measure the tools’ effectiveness and usability. Andrade et al. [P5] introduce a method to map SysML activity diagrams to Petri Nets for the evaluation of real-time systems with energy constraints and measure the obtained analysis results with hardware measurements, which confirm the adequacy of analysis results. Anwar et al. [P11] introduce a model-based method for design of FPGAs and measure the time taken for baseline methods compared to their approach in man-hours. Eventually, 17 (11.4%) papers have no example or empirical observation for their solution [P15, P25, P31, P32, P42, P53, P58, P59, P70, P98, P100, P104, P119, P124, P128]. For example, Lima et al. [P25] describe a semantic for reasoning about SysML diagrams via refinement and support their approach through abstract high-level disjoint diagrammatic examples; instead, Gauthier et al. [P119] explain a process to transform SysML models to Modelica code without any example in the publication apart from the transformation rules.

The outcome about users of early validation is consistent with the other categories in the sense that there is a large and significant spread. The outliers in this regard are aerospace (with a related domain referring to avionics), embedded systems, CPS, safety-critical systems, railway, and automotive. Interestingly, solutions in the embedded systems domain tend to be consistent in terms of solutions proposed by the analysed papers, while the same cannot be said for the other domains. In fact, in most of the larger reported domains there is no common/shared view of how early V&V is expected to be performed. For example, in aerospace there are examples of requirement analysis [P7], simulation [P10], schedulability analysis [P16], model checking [P17], inspection of diagrams [P55], and so on.

RQ4 discussion: As it might be expected, most of the solutions are domain specific, and only around 11% claim to be applicable to any domain. As a matter of fact, since performing analysis early in the SE process requires assumptions to be made or uncertainties to be managed, the solutions tend to be domain specific due to more precise and reliable information on which to build the analysis. Further, as many target languages or tools are often coupled strongly to a domain, profiles and constraints need to be in place to enable SysML and similar semi-formal (general-purpose) languages to be applied with enough rigour.

Around 34% of the analysed papers describe research performed in an industrial setting, seemingly indicating the industrial applicability of many solutions. Indeed MBSE is a paradigm with a solid industrial perspective, which puts many requirements on the types of solutions. Related to this substantial industrial scope we note also that most of the solutions are based on examples to various degrees, with less focus on empirical measurements. Indeed, only few papers discuss empirical evidence to support the solutions that instead largely rely on arguing for the perceived benefits. These observations are consistent with the previously reported weakness of MBSE [41], that is, the lack of empirical evidence. In this respect, we argue that empirical evidence could be hard to produce for solutions of this nature, as the measurements are intricate to define and might be challenging to perform in industrial settings without a high risk of introducing bias and other validity threats. Nevertheless, this is a reoccurring problem regarding model-based practices that hinders evidence-based discussions on a broader scale [16].

The main artefacts of MBSE, the models themselves, need to fulfil certain criteria to be useful. For example, the models must have adequate abstraction and address different stakeholder concerns depending on the context while keeping enough rigour to facilitate analysis. Finding the correct balance between abstraction and fidelity is a re-occurring challenge in the papers and is often a pre-requisite for enabling mature early V&V in the context of MBSE. Overcoming the gap of abstraction and fidelity often results in model transformations from one notation for description to one or more other notations for analysis, seen in the large heterogeneity in RQ2 and RQ3. While these are valid techniques, the introduced complexity makes the MBSE adoption more difficult for new practitioners. Particularly, the need to maintain and support model transformations in changing environments regarding tools, languages, users, and the systems themselves introduces technical complexity and usability concerns, seen in the literature [60, 83] and reconfirmed by RQ5. Little attention is as well paid to the holistic nature of MBSE. Most solutions are situated in very narrow applications that do not consider the surrounding development or overall system development, noticeable in the extraction from RQ1. In the same fashion, solutions often follow a particular working process. Many solutions enforce a particular process on the users, which can often clash with engineers and workflows already in place, again seen in RQ3 in the techniques and tools. Indeed, the practitioners themselves are mostly ignored from the papers, and the overall usability of solutions is not a priority. There is seemingly no clear view of how this can be addressed, perhaps due to the wide application in area in many domains seen in RQ4. Additionally, model-based practices are largely heavily tool dependent, and many traditional issues such as interoperability, automation, and scalability are encountered in the extraction, which reconfirms previous work in the field [11, 25]. In addition to the explicit challenges (still) existing, we also notice a distinct lack of discussions on academic tool integration with more significant tooling landscapes, observable in RQ3. As expected the challenges overlap, but many of the academic solutions observed suffer from a very narrow target with unique tools and languages when the industry is increasingly moving toward standards such as OSLC¹⁵ to promote more standardised data exchanges. Additionally, there seems to be a gap between the industry practitioners and the academic researchers, and there is a risk that the tools developed for academia are aiming to solve other problems compared to industry needs.

As discussed in Section 2.1, a large part of the industry utilising SE is undergoing a transformation toward at least partial MBSE adoption. However, the legacy is often substantial and cannot simply be discarded in favour of new methods or working methods, yet there is little discussion on how to facilitate such integration in the RQ3 data extraction. Meaningful integration of legacy formats, such as office tools, drawings, and text, must be incorporated in solutions to be viable on a larger scale. Similarly, the audience of SE is heterogeneous, and the discussed context of early is not strictly defined. A wide audience is expected to leverage early V&V. For example, a common solution pattern in the RQ2 and RQ3 analysis is engineers leveraging SysML to orchestrate simulations without the need for simulation knowledge. However, the solutions are often designed with an expert audience in mind, which limits the wider applicability of solutions (particularly since many engineers are complete novices regarding Model-Based methods). Furthermore, the extracted data hints at a limited integration in workflows to promote the integration of models across development, seen through the authors properties of interest and their usage in RQ3 and limitations in RQ5. Many solutions are limited in extension to surrounding activities, e.g., a solution lacks up and downstream linkage. Similarly, there is little discussion on how analysis models should be managed to facilitate re-use and efficient future decision-making, noticeable in the intersection of RQ1 and RQ3. Without a means of classifying models or results for future use, the value of approaches becomes heavily reduced. Similarly, traceability, and more recently, the digital thread, are largely omitted from any discussions thus reducing their applicability. Indeed, there needs to be propagation of results and their impact across the development to maximise the benefits of the analysis. As mentioned previously in the section, a significant barrier to adoption is the difficulty of providing measurable benefits of MBSE approaches. Often, the solutions’ value is argued based on anecdotal examples or rough estimates, which hinders the rigorous evaluation of solutions, visible in the RQ4 extraction. In the same fashion, when measurements are performed in some way, no proper benchmarks can be used to position papers in the wider context. A more common view of benchmarks and metrics for SE at large would help reduce some issues related to the measurement and evaluation of solutions.

A large gap in the solutions is the discussion of uncertainty and analysis validity, also recognised among limitations in RQ5. In the early stages of development, uncertainty is present in all models and requires considerable efforts to mitigate and reason about so that results can be correctly positioned and used for decision-making. However, there is rarely any discussion on how uncertainty can be managed in a solution and what implications the analysis abstraction introduces, which is one of the more prominent limitations reported by authors in RQ5. The same is true for analysis validity. If re-use is of interest, then there is a need to catalogue when a method or model can be used for correct analysis and how a user might be able to determine on a case basis when the considered models are rich enough to yield valid analysis results. In addition, the methods themselves are often complex and can introduce a burden on the user to incorporate effectively the methods “as is.” Furthermore, the back-propagation of analytical results to the engineers, or descriptive models, is generally missing from the solutions, and results are often simply presented in the analytical tools presented in RQ3. In particular, how analytical results should affect decision-making, how the analytical results should be translated back to the system design directly, and how these results can be leveraged in industrial processes is rarely part of the methods of authors. For example, the notion of standards should be included in articles that guide practitioners during development and how the proposed solutions can support standard processes or certification activities and are inline with the major extraction for RQ1. While research might not be expected to produce technology with industrial readiness, it still needs to consider the context, which is mostly standard based and standard driven. The analysis methods are generally domain specific and cater to specific needs as per RQ4 extraction, and often implicit domain knowledge is a prerequisite for useful analytical results. Nonetheless, the general lack of domain-agnostic and more general solutions inhibits the adoption of existing solutions due to the difficulty of transferring those solutions to different domains as the implicit knowledge is hidden or not valid across different domains. In the same fashion, many solutions utilise custom tools for the implementation, limiting integration in SE processes. Particularly, relying on several specific tools increases the process complexity and can lead to increased vendor lock-in.