Cognitive sensor systems for NDE 4.0: Technology, AI embedding, validation and qualification

2.2.1 Key values of OPC UA

OPC UA is an open standard for cross-manufacturer and cross-platform communication across different levels and degrees of automation [52]. This standard is already used by various industrial companies and by individual associations such as the Mechanical Engineering Industry Association (VDMA) to implement the Industry 4.0 (I4.0) aspects. Also, the communication between different machines of a production network is realized via OPC UA [87]. Within the framework of the committee work of the VDMA, manufacturer-independent information models for the areas of robotics and industrial image processing (machine vision) have already been developed and published [84], [85]. The specifications of such information models are called companion specification. With the OPC UA Companion Specification for Machinery [86] a specification also exists that can be used as a universal fundament for further companion specifications.

To understand the importance of OPC UA for I4.0 in general and NDE 4.0 in particular, it is necessary to take a brief look at the current state of the art for communication interfaces. Previous communication structures of the third industrial revolution are based on the model of the automation pyramid [45]. Depending on the communication level, different communication interfaces have been used to meet the requirements in terms of data volume and transmission speed. This resulted in the structure of the data and the choice of technology being manufacturer-dependent [26].

OPC UA builds on these existing technologies and extends them in a useful way. The technology can be used at all levels of the automation pyramid [9]. To realize deterministic communication requirements on the actuator/sensor level, the hardware standard Time-Sensitive-Network is additionally required [10]. Furthermore, the structure of an OPC UA server is explicit, since its description is integrated on the server and does not have to be interpreted manually by the end user. This is also the case without the use of companion specifications, whereby their uniform structure allows additional facilitation with regard to the integration and exchange of components into existing network structures.

2.2.2 OPC UA for NDE 4.0

For the use of NDE methods in IIoT networks, OPC UA represents an essential building block for the above reasons. Methods can communicate directly with MES (Manufacturing Execution System) and ERP systems, receive inspection instructions and provide their results in real time. However, in order to be able to use this technology effectively, a NDE-specific companion specification is required that fulfills the general requirements for the inspection methods and takes into account the respective degree of automation of each method. This includes, for example, the inspection instructions and the results of the inspection report. Such a companion specification is currently under development within the OPC UA working group of the ZFP4.0 expert group of the German Society for Nondestructive Testing (DGZfP). Independent of the development of this information model, OPC UA is already used for the control of inspection tasks. The authors in [38] describe a robotic inspection of adhesively bonded joints using active thermography and OPC UA as a communication interface. The implementation of a digital twin for quality assurance of welding processes was also realized using OPC UA [51]. OPC UA thus enables the digital provision of information, which was previously only available in printed form or as a PDF file, as well as the use of the results as relevant data for other IIoT approaches, such as AI aspects or the digital twin.

2.3.1 The standardized data format DICONDE

In order to archive and use relevant information in a generally usable way for years, uniform data formats are an essential requirement. One appropriate data format that meets the requirements of NDE 4.0 is DICONDE. The technology DICONDE is derived from Association for Computing Machinery (ACM) and National Electrical Manufacturers Association (NEMA). DICONDE describes both a generic data format and a communication interface. It has been known in medicine since 1985 and was introduced in 1993 as Digital Imaging and Communications in Medicine (DICOM). The medical variant has already been implemented for storage of images procedures by e. g. ultrasound or X-ray. There are a number of extensions for DICONDE, which are based on medicine and are adapted in various fields, extend them or add new possibilities. Instead of the patient, the focus is now on the specific test object with its proper-ties. Standards specify which values must be filled in by the user and which are optional or conditionally optional for example, for eddy current, ultrasound and X-ray inspection [1], [3], [4]. In parallel with the linking of test results with component parts, relevant metadata are also stored in the DICONDE format. Beside UIDs (Unique Identifier) and date of creation, this also includes, amongst others, component information such as material properties or, with regard to inspection procedures, serial numbers of the systems and device settings. This makes an inspection process completely traceable, even for further users, and enables further evaluations and analyses.

The implementation of a widely used protocol results in various advantages. Inspection systems can be embedded in larger industrial plants and offer new possibilities for cross-modal data evaluation and audit-proof storage via standardized paths. This reduces both integration costs and the susceptibility to errors, since all participants use a structured unique generic data format. Thus, DICONDE enables that future evaluations or analyses do not necessarily have to be performed with the software with which the data were acquired. Since evaluated data are stored in DICONDE, the recording software can directly calculate all necessary images (e. g. cross-sectional images or C-scans) and store them directly into the DICONDE archive.

2.3.2 DICONDE+

Under the keyword DICONDE+, research is currently taking place that addresses the integration of CSS in DICONDE, but also pursues the intelligent use of data sets for NDE 4.0 aspects, for example by integrating also the raw data sets into DICONDE+ in general. On the one hand, the clear structure of DICONDE means that data relating to components and inspection results can be clearly interpreted, while at the same time this structure must be strictly applied in order to generate DICONDE-compliant data sets. This makes it difficult, for example, to integrate inspection data obtained with unconventional multi-sensor systems or newly developed hybrid inspection methods [12], [82]. By creating suitable standards for such methods, new inspection and evaluation methods and thus data from innovative novel sensor systems or fused data sets could also be integrated into the data format in the future. Furthermore, this standard could generally be extended to include destructive testing methods.

With regard to the second aspect, DICONDE is also suited for intelligent data processing using approaches that go beyond those of curated data hosting. This allows DICONDE data sets to be used not only for archiving but also as an example in AI applications by means of a semantic search. To enable such a semantic search, the metadata of the DICONDE data must be mapped in a suited database using an adequate ontology. Such an implementation could be realized, for example, by a SPARQL server (SPARQL Protocol and RDF Query Language) [73]. In this way, complex search queries are possible, which allow the user to search specifically for data sets for his current application. An example of this would be the search for additional data sets as training data for the implementation, qualification and validation of an AI algorithm. This algorithm is more robust with additional data from a DICONDE archive and allows more extensive qualification and validation of the approaches developed. At the same time, new data sets, which were only possible using the semantic search, can be added to the DICONDE. In this way, generic data formats are not only an effective means for archiving data, but also pave the way for the implementation, qualification and validation of AI applications and they allow new data-driven business models.

Figure 3

NDE 4.0 demonstrator consisting of a dual robot unit (left: KUKA LBR iiwa, right: ABB YuMi). The B-pillar base is located on the worktable of the ABB YuMi. The three displays show raw data of a sensor (right), the common display of all current sensor results (middle) and the documentation in DICONDE (left).

3.1.1 Computing in sensor systems

Cognition in terms of sensor systems can be understood as the ability of the system itself for learning and/or inferencing. In simplified terms, each sensor system consists of the individual components shown in Fig. 4. It consists of the sensor itself and a converter to the electrical domain. Both elements are referred to as the transducer. Together with a measurement amplification and/or normalization, the so-called transmitter is created. After the signal normalization, the conversion from the analog to the digital domain can take place [78]. Before analog-to-digital conversion, the sensor system does not yet have any cognitive capabilities. The discretization of the analog signal and digital signal processing improve the robustness of signal processing and transmission. In addition, microprocessors can be used to solve more complex tasks with the existing sensor signals. The data volume when processing sensor signals is mainly dependent on resolution, sampling rate as well as the number of signals to be digitized. With the increasing use of data-driven signal processing approaches such as machine-learning algorithms to build complex models for decision-making and classification, logically the amount of data is also increasing. With the growing demand for sensor systems that can understand complex processes and make conclusions based on incoming sensor data, as well as the increase in miniaturization and energy efficiency, it is necessary to develop new processor structures. A well-known limitation in processing large amounts of data from microprocessors is the so-called memory-wall. This problem arises from embedded systems up to high performance systems (e. g. servers, supercomputers) [80]. One approach is to follow a biologically inspired data processing, as it takes place in the brain of living beings, known as neuromorphic computing. Among this, especially the approach to process data within the memory is very promising. The creation of complex models or processing of data in memory is also referred as In-Memory Computing (IMC) [36]. It has been shown that this technique can improve the energy efficiency up to multiple hundreds of Tera Operations per Second per Watt (TOPS/W) [105] and up to 10000 TOPS/W [53]. New technologies such as resistive RAMs (Random-Access Memory) can be used for data processing, but also familiar memory technologies such as SRAM (Static RAM), DRAM (Dynamic RAM) and flash memory. Each of these technologies has its technical and economic advantages and disadvantages [36]. Not only the systematic structure and the way of data to be transported within the processor changes, but also the way of signal processing. Using IMC units as an example, analog data processing is implemented. This can lead to advantages in terms of energy efficiency and data throughput. On the other hand, the analog nature can have the disadvantage that the accuracy can be reduced considering classification algorithms.

3.1.2 Energy efficient sensor nodes

The selection of a suitable hardware platform for a specific application is an essential part in the development of sensor systems. Energy efficiency, miniaturization, cost, as well as usability are only a few parameters that can play a role. While an Application-Specific-Integrated-Circuit (ASIC) implementation is probably the most efficient solution, microcontroller as well as Field-Programmable-Gate-Array (FPGA) solutions show a much higher flexibility. Therefore, when designing a sensor system, it is important to know all parameters and requirements for the target application. In the case of applications with machine-learning algorithms on sensor nodes, the efficient use of available resources plays an increasingly important role, as also briefly explained in Section 3.1.1. The design of dedicated circuits for the execution of previously trained algorithms are an important part of achieving an optimal solution.

3.2.1 The role of advanced signal processing for NDE 4.0

There is unanimous agreement that realizing the vision of NDE 4.0 requires the deployment of large quantities of sensor nodes to deliver the required digital data [88]. In terms of the scalability, the development of low-cost IIoT sensor devices is particularly attractive, as it allows their disposition in large quantities [89]. If we manage to solve the challenges attached to making such systems interoperable (cf. Section 2), we can aggregate large streams of data about our processes and products. On the one hand, this allows sophisticated analyses and a robust training of data driven methods (cf. Section 4). On the other hand, collecting, maintaining, and accessing these data sets may turn into a severe challenge on its own [104]. It requires not only continuous significant investment in IT resources but also personnel (i. e. data scientists), able to retrieve the relevant information from the collected data. In many cases, this approach may be wasteful as an unfiltered collection of raw sensor data leads to a high degree of redundancy and it would be much more efficient to focus on the relevant data in the first place.

This is the promise of the Compressed Sensing (CS) paradigm [25]. In its core, it states that we can sample signals significantly below the Shannon-Nyquist rate without loss of relevant information, provided that prior knowledge on the signals is available [17]. This is very often the case in industrial applications. Hence, if we manage to encode this information into the sensors themselves, their recorded signals contain significantly less redundancy and deliver much more relevant data. CS has been successfully applied in a number of research fields, including fields like magnetic resonance imaging [50], radar [15], imaging [19], and many others. Recently, we have started to explore the intimate connections between CS and the field of NDE 4.0 [68]. There are at least two aspects of CS that can be applied beneficially: novel sub-Nyquist sampling architectures for reducing data redundancy on the one hand and an advanced algorithmic framework for data processing on the other hand. Since we do not aim at a comprehensive review of all NDE applications of CS in this section, let us instead focus on a few relevant examples.

As a first example, let us consider industrial X-ray Computed Tomography (CT). X-ray CT is used in a large and increasing number of industrial applications [13]. It delivers representations of the interior structure of a wide range of materials, allows identification and measurement of geometrical features (including defects), and delivers material information through the X-ray absorption [96]. A bottleneck of 3D X-ray CT systems is their inspection speed, since the spatial Nyquist theorem requires a certain number of projections, each of which needs a certain minimum exposure time [62]. CS offers a potential solution since it admits accurate reconstructions from sub-sampled data, provided that prior knowledge about the objects is available. In industrial settings, such knowledge comes in multiple forms: manufactured pieces typically consist of a limited number of known materials, rendering the reconstructed images piece-wise constant. Also, in some cases (such as inline inspection), the geometry of the piece is known up to deviations due to manufacturing and positioning tolerances as well as the defects that we aim to find. Based on this prior knowledge, we can carry out 3D scans with significantly fewer projections without compromising image quality, provided that the algorithmic framework for the reconstruction is suitably adjusted. Adapting methods first proposed in a medical context [18], [76], we were able to show that in inline X-ray CT applications, a ten-fold reduction of the projections is possible without compromising the capabilities of finding even small defects [77]. As a second example, let us consider ultrasound. Ultrasonic signals possess high degrees of redundancy in many industrial settings for two reasons: firstly, their time signatures often consist of isolated echoes with (partially) known pulse shapes. Secondly, the wavefield typically does not change abruptly over space, rendering scans taking at adjacent points highly correlated. Both factors have been used to devise systems that incorporate temporal CS (e. g., through the modulated wideband converter architecture [14] or through (scrambled) Fourier sensing [49]), combined with spatial subsampling [63]. In [46] it is shown how spatial resolution can be obtained from a single transducer based on CS principles by placing pseudo-random masks into the acoustic path. For lamb waves, jittered subsampling is considered in [28]. Subsampling is usually combined with advanced reconstruction algorithms (which can also be applied on Nyquist sampled data). A prominent example is the use of sparse recovery algorithms such as FISTA (Fast Iterative Shrinkage-Thresholding Algorithm) [11] or STELA (Soft-Thresholding with Exact Line search Algorithm) [103], which yield superior image quality, even when applied to Nyquist-sampled data [48], but also facilitate reconstruction from CS measurements with very high compression rates [49]. As a final example, let us consider Terahertz imaging, i. e., the usage of electromagnetic waves in frequency ranges above 100 GHz. The interaction of such waves with matter permits scans with high sensitivity and high spatial resolution in a number of specific applications, such as the inspection of polymers, thermoplastics, as well as certain composite materials, alongside the measurement of coating thickness, the evaluation of adhesions and welds, and others [27]. While hazard-free and flexible to apply, the main drawback of Terahertz scans is the fact that they are typically time-consuming due to the required raster scans [43]. This is an interesting application of CS where the combination of Fourier optics [32] and the principle of the single pixel camera [19] have enabled the conception of systems capable of spatially resolved Terahertz imaging without the necessity to carry out mechanical movements [16]. These examples show that CS-based acquisition devices can lead to quite unconventional designs.

Overall, CS allows significantly increased flexibility on the design of our sampling architectures, to take into account prior knowledge and thereby focus more on sampling relevant data directly. Such innovations can lead to reduced power consumption and data rate, which is particularly relevant for wireless sensors, but may also be a key enabler for high-rate applications (such as scaling up the number of channels in a Multi-Input-Multi-Output setup). At the same time, the algorithmic framework of the CS field can be beneficially applied for enhanced image reconstruction, giving rise to families of algorithms with adjustable complexity vs. performance trade-off.

3.2.2 Quality assessment of advanced reconstruction schemes

As CS introduces new concepts into the acquisition and processing of sensor signals, including concepts of randomness, it is not always easy to characterize the achievable quality deterministically. Yet, for NDE applications, a reliable quality assessment is of utmost importance. Hence, this section discusses potential performance metrics along with their advantages and drawbacks. There are two key ingredients to a successful reconstruction method: its achievable quality and the computational complexity to obtain this result. More often than not, these two are conflicting goals, and as argued above, the algorithmic framework of the CS field provides methods to control the trade-off flexibly. Hence, we need to evaluate both: the quality as well as the complexity. In this section, we want to discuss metrics for these two aspects, using CS in ultrasound as an example.

Regarding quality metrics for ultrasonic images, the first and most obvious choice is a subjective comparison of the image quality. As state of the art NDE is still often manual, practitioners are quite used to assessing unprocessed and processed data (such as A-Scans or B-Scans) by visual inspection. It has been shown that parametric reconstruction algorithms can lead to better visual image quality by virtue of a more precise image of the defect geometry [48], even when combined with severe sub-Nyquist sampling through CS [49]. However, the evaluations of the visual image quality is subjective and may be misleading. As a more objective alternative, it is quite common to apply the Contrast to Noise Ratio (CNR) for imaging features in regions of interest, such as lesions in medical settings. However, for modern reconstruction algorithms, the CNR may not be appropriate since their amplitudes do not necessarily represent backscattering intensities anymore and the CNR is susceptible to amplitude variations such as a dynamic range compression that do not improve detectability, as argued in [65]. The paper also suggests a more stable metric given by the generalized CNR (gCNR), which is directly related to the probabilities of false/missed detections and always normalized to the interval [0,1]. The gCNR has proven to be a useful tool when analyzing imperfections in reconstruction methods in NDE, e. g. due to positioning uncertainties in manual ultrasonic testing [44]. Likewise, to study the degradation of image quality due to changes in the measurement setup, such as sub-Nyquist sampling, image quality metrics such as the structural similarity index measure (SSIM) [70] have been applied.

Finally, a common approach in the field of NDE is to study the abilities of a given imaging approach to detect and accurately size defects [95]. The corresponding metric is the POD, which measures the chance of a given detector to find defects as a function of their size. While it is common to apply POD analysis to data acquired from reference specimen, it can also be used in a simulative manner by the following procedure: (1) using a forward model that can generate realistic ultrasound signals from defects with varying sizes; (2) emulating measurement effects (such as compressed sensing and additive noise) in software; (3) feeding the synthetically generated measurement data through the reconstruction framework; (4) extracting relevant features from the obtained images (such as a defect amplitude); (5) performing a regression between defect size and the chosen feature from a set of noisy realizations; (6) using the empirical statistics to compute a POD curve, e. g. described in [7]. While the absolute numbers obtained with this approach may not always be entirely realistic (since simulations can be too ideal), it provides a way to compare the influence of system parameters (such as compression rates and strategies) and the used algorithmic framework on the POD curve. An example of this procedure is shown in Fig. 5 where we compare the POD in a synthetic 16 × 16 Full Matrix Capture data set (FMC) with a single defect of varying size. We compare the Total Focusing Method (TFM, red curve) with parametric reconstructions through FISTA. For FISTA we employ temporal compression [49] (factor 5, orange curve) and spatio-temporal compression [63] (5× temporal, 10× spatial, blue curve). Figure 5 shows that in the simulation FISTA always outperforms the TFM, even when significant data reduction (factor 50) is applied. These results should be verified based on a real specimen with varying defect sizes in future investigations.

Figure 5

Sample POD result for a synthetic 16 × 16 FMC data set with a single defect of varying size, comparing classical reconstruction via the TFM (red curve) with FISTA using 5× temporal compression (orange curve) and 50× spatio-temporal compression (5× temporal and 10× spatial, blue curve).

Alongside the performance, we also need to quantify the complexity of given algorithms. While in the past, it was common to simply count the number required of floating point operations, this is no longer sufficient since computations are not necessarily linear any longer. With a variety of new computing paradigms, allowing for an increasing amount of parallelization and optimized processing on specific platforms such as GPUs, the complexity analysis is not an easy task anymore. Often, we resort to studying asymptotic complexities that give an indication of the expected scaling of the run time with a change in the input dimension. As an example, while algorithms such as Synthetic Aperture Focusing Technique (SAFT) possess a linear scaling of the complexity with respect to its input dimensions, iterative algorithms such as FISTA can scale quadratically, which may render them infeasible for larger input sizes. At the same time, as the available computing power keeps increasing due to Moore’s law and the advent of quantum computing promises the availability of unprecedented computing resources [94], interest in more complex processing algorithms is now bigger than it ever was before.

3.3.1 IZFP generic AI-accelerator

Currently, Fraunhofer IZFP is developing solutions for CSS that are more energy-efficient and more ecological friendly and at the same time scalable in terms of data volume, and data processing. The first approach here is an all-digital FPGA solution that is currently in development. The system is developed to process eddy current data and consists of a RISC-V (Reduced Instruction Set Computers – five) co-processor as well as dedicated AI accelerators. The key idea of AI accelerators is to take advantage of the nature of neural networks. These consist of several layers, where it is possible to execute every operation of the neurons of a single layer simultaneously. Since each FPGA has different resources, accelerators have been developed that are not only adaptable in bit width and thus adaptable in the accuracy of the algorithm to be executed, but also configurable in terms of parallelism of data processing and number of accelerator stages. With this approach, it is possible to easily adapt the hardware with software hardware co-design is facilitated. In addition, RISC-V processors allow the underlying instruction set architecture to be extended, which in future works should lead to even greater flexibility in processing data.

3.3.2 Assistance in NDE by augmented reality

In the context of NDE 4.0, an active area of research is the development of assistance systems in order to support the operating personal during inspection tasks. The main goals are the automated documentation, the visualization of intermediate inspection results, the automatic verification of the correct scanning and data acquisition process [22], [57]. In addition, they facilitate novel digital workflows for manual inspection tasks. Generally, all systems rely on real-time combination of measurement data with the tracked position of the probe [89]. An additional aspect is the improvement of the training of inspection personnel, e. g. through mixed reality concepts as discussed in [58]. The recorded data can also be integrated into the framework of digital building information modeling (BIM), as described in [75]. A survey of the recent state of the art for these technologies can be found in [99].

To give a specific example, we briefly describe the features of the smart inspection systems from [89]. The presented laboratory prototype system acts as a complete platform to gain R&D experience for future NDE 4.0 sensor systems. Further aspects consider the impact of usability on human-machine interaction and the improvement on acquiring proficiency and the practical use of such technologies. A picture of this system is shown in Fig. 6.

Figure 6

Assistance system with NDE 4.0 features and Augmented Reality visualization (AR), image has been edited for illustration purposes [89].

The smart assistance system shown in Fig. 6 is suitable for various sensor modalities (e. g., ultrasound or eddy current testing). It implements a camera-based tracking of the probe and supports the inspector during the scanning procedure (scanned area, non-scanned area, signal validity, etc.). The inspection data and results are visualized via AR in real-time.

It combines several features relevant to future NDE 4.0 systems, including:

1. advanced signal processing and progressive image reconstruction techniques (inspired by CS concepts, cf. Section 3.2) as well as AI-based advanced pattern recognition techniques (cf. Section 4) for generating high quality feedback in real-time,

2. unified interfaces and data formats to facilitate the IIoT integration (cf. Section 2),

3. autonomous signal control and self-monitoring to ensure the safety and the validity of the recorded data,

4. augmented reality-based real-time visualization,

5. remote control and cloud-based operation.

5.2.1 Qualification of ML-based NDE systems

The performance of an NDE system is proven within the framework of the ENIQ qualification methodology by means of a technical justification and practical trials on suitable test specimens. For systems that use ML-based methods for data analysis, the proof of performance can be performed in the same manner. In the following aspects, the ML components of an NDE 4.0 system must be included in the qualification process or require new activities that do not have to be considered for conventional NDE systems [31].

1. When designing specimens with known and unknown defects for procedure validation within the TJ and practical trials, the design of training data sets and test data sets must be considered for ML components.

2. Selection and training of ML algorithms

3. Generation of qualification data sets for ML

4. Consideration of ML during procedure validation

5. Qualification of the inspection procedure and the ML algorithms (cf. Section 5.4)

5.2.2 Requirements for technical justification when using ML concepts

The contents and structure of a technical justification are documented in the guide Strategy and Recommended Contents for Technical Justifications [29]. As part of the approach described in this document, ML components must be considered in various sections of the TJ [31]. A summary of the ML system, including the justification for its use of the selected algorithms and the expected results, should be provided. Furthermore, it should be presented how the system will be put into operation and how version traceability can be ensured. Finally, a summary should be provided detailing the capabilities of the ML system and indicating any recommendations for improvement if the full specification could not be achieved. The explanation of influencing factors as well as the experimental proof of the performance in terms of reliability of the ML components as well as the overall system will be discussed in more detail here. When considering influencing factors, it is assumed that the performance of ML algorithms is based more on experimental results than on clearly comprehensible physical relationships. At this point, transparency about the algorithms used plays an essential role (cf. Section 4.2). The decision to select the algorithms used must be presented as part of the qualification process. The selected ML algorithms should be documented in sufficient detail to assess the potential for error and the required justification. It is expected that the quality of the data sets used for training as well as for validation of the ML algorithms will have a significant impact on the performance of the overall system. Therefore, the data sets used should cover the range of defects necessary to achieve the test objective, as well as contain representations that lead to possible false calls. In addition, it must be demonstrated that the data sets do not contain bias that negatively influences the test results. Another influencing factor is the distribution of the data sets used for the validation and qualification of the ML algorithms. Here, care must be taken to ensure a clear separation of training data sets that are used directly for training the ML model. Test data sets used for monitoring and evaluation of the training and qualification data sets used for validation of the final performance of the model must be clearly separated. Another important point is the validation of the performance of the ML components and the overall system by determining the reliability. Known statistical performance indicators can be used here. It is important to select the statistical methods so that they are suitable for the NDE system to be qualified [31], [42]. Here a subdivision can be made. On the one hand, into procedures for the evaluation of faults such as the probability of detection (POD) as a function of fault characteristics or the receiver operating characteristic (ROC) described in more detail in Section 5.3 [5], [33] and into procedures which determine the suitability of a procedure for the determination of fault characteristics with the aid of a modular model where, in addition to the intrinsic capabilities, application parameters, human factors and the organizational context are taken into account [54], [67]. Another issue when considering the reliability of an NDE 4.0 system is the model variability. Estimation of this variability can be done either by using probabilistic ML models or by validation of different ML models for a given training data set (e. g. by a cross-validation procedure) [31]. During the qualification process of the ML components, care should be taken not only to consider the target variable such as defect size/defect class, but also to consider the false calls that occur. The false call rates can be used to draw conclusions about performance at an early stage during the development and qualification process. Possible overfitting, i. e. the ML model delivers good results for training and validation data used during development, but fails when generalized to unknown defects, must be excluded by testing on unknown data sets.

5.2.3 Qualification in practical trails

For traditional NDE systems, practical trials are performed as part of the qualification process. These requirements are described in detail in [30]. Essentially, the tests are divided into open tests and blind tests. The open trials are usually used to validate the inspection technique and the process procedures. In the blind tests, the performance of the operators in interaction with the inspection technique is considered.

For NDE 4.0 systems that perform data evaluation based on ML components, the operator’s tasks change. The operator may only be involved in the data evaluation in a supervisory role. However, the shift of data evaluation to the test system now also requires blind tests on unknown real components or specimens for the ML components. This means that separating the evidence of process performance and operator performance becomes less important. A recommendation for the procedure is described in [31].

5.2.4 Essentials for guidelines and standards of NDE 4.0

ML is important for NDE 4.0 systems. They offer the potential to improve the performance as well as the reliability of currently used inspection techniques. NDE 4.0 systems allow to reduce human factors that lead to high variation in inspection results. In contrast, however, new challenges arise that must be taken into account in qualification processes for such systems. These include the very reduced focus and experience compared to humans with regard to the influencing factors to be considered during an inspection. Due to this, as well as the low level of experience and the resulting lack of trust, ML systems are expected to be highly repeatable and consistent in their application. This means that qualification processes such as those described in [30] must be extended for NDE 4.0 systems. This includes e. g. the development and maintenance of the corresponding qualification specimen and data sets in order to be able to prove reliability in a resilient way.

The ENIQ document [31] provides a clear recommendation for the extension of existing qualification processes in the field of nuclear facilities. The recommendations set out in this document can also be transferred to applications beyond nuclear technology and can be used as a basis for a generalized procedure.

5.4.1 Global context and the necessity for trusted microelectronics

The worldwide drive for digitalization in civil and industrial environments, as well as the desire for a networked world, are causing a global increase in the demand for electronic components. Solutions for current trends such as smart mobility/city as well as IIoT and 5G are technology drivers here. However, the increased use of microelectronic components and their data traffic to cloud systems is significantly increasing global energy consumption. To counteract this development, an increasing shift of decision-making from the cloud level towards the end device is being strived for. In this way, decisions can be made directly at the point of action and energy-intensive data traffic can be saved [37]. The consequence is the use of high-performance microelectronic circuits in the most diverse areas of daily life and opens up a wide range of possibilities for malicious attacks. In principle, malicious attacks can pursue different goals such as spying on personal data or company secrets, manipulating or deleting files or sabotaging production processes and security-relevant infrastructure. In addition to the software technologies that have been established for many years, such as computer viruses, Trojan horses, or computer worms, the use of so-called hardware Trojans is becoming increasingly relevant.

A hardware Trojan is the manipulation of hardware or firmware of integrated circuits and electronic modules. The manipulation can take place at different points of the value chain and remain unnoticed for a long time [35]. In addition to being considered in the development phase of microelectronic circuits, hardware Trojans can also be implemented unnoticed within chip or module production. Another point of attack is the transport of electronic components. FPGA devices offer special attack possibilities here by manipulating the programming process [74]. The potential risk of attack is further exacerbated by the globally interwoven supply and value chains. While products from the consumer sector, with the exception of communication devices such as mobile phones, routers, etc. [79], are less in focus, components in security-relevant areas such as power plants, internet nodes or communication systems in intelligence agencies must be reliable and trustworthy.

Sensor systems in NDE 4.0 often provide safety-relevant data, are used to optimize processes or determine the quality of products. Ensuring the trustworthiness of NDE sensors is therefore a particular challenge.

5.4.2 Technological solutions for security of microelectronics in NDE 4.0

Security on low-level hardware is crucial as a solution for NDE 4.0 security. This implies among others, a transparency and openness of the underlying processor structure. RISC-V as an open-source instruction set architecture is believed to offer a level of security that is not yet available with other solutions. There are groups working at security extensions considering cryptographic extensions as well as trusted execution environment. Because of the open-source nature, everyone can get involved to fill security vulnerabilities.

In addition to the described technologies for a precautionary defense against tampering, microelectronic circuits and assemblies can also be examined for the presence of hardware Trojans. For the classification of hardware Trojans, the characterization of the physical implementation, the type of activation as well as the action during the active phase can be used [71]. Different destructive and nondestructive methods can be used for detection. The detection of manipulated circuit parts on chip level can be done with the help of microscopic methods (e. g. scanning optical microscopy, scanning electron microscopy, and others) after opening the component package [71]. With the scanning acoustic microscopy method, chip structures can be imaged without package manipulation [34]. Functional tests, e. g. applying defined test patterns to the input pins, can indicate manufacturing faults or hardware Trojans by comparing the output signals. Side channel analysis is able to analyze different signals caused by electrical activity in electronic circuits. Side-channel signals include electromagnetic radiation, timing information, thermal radiation, and power consumption [71]. The use of imaging thermographic cameras is considered to be promising [69]. To increase the detection probability of different classes of hardware Trojans, the combination of several detection methods can be used.