Semi-automatic assessment of lack of control code documentation in automated production systems

2.1 Development of aPS

aPS is developed by engineers from multiple disciplines [8]. The development often starts in the mechanical engineering discipline, which creates the construction plan of the mechanical parts. Based on this construction plan and component lists (e.g., sensors or actuators), the electrical engineers design the electrical system. Documents from both disciplines are then forwarded to the software engineers, who use them to develop the software running in Programmable Logic Controllers (PLC) (i.e. control software or aPS software), the hardware platforms used to manage the automation for machines and plants via sensor inputs and actuator outputs.

There are differences in the languages for PLCs compared to those used in classical software engineering. The aPS software is often developed following the IEC 61131-3 standard [5]. IEC 61131-3 defines three types of Program Organization Units (POU): (1) Function (FC), (2) Function Block (FB), and (3) Program (PRG). Each POU includes a comment header, a variable declaration section, and an implementation section. IEC 61131-3 compliant languages include three graphical languages (i.e., Ladder Diagram LD, Function Block Diagram FBD, and Sequential Function Chart SFC) and two textual languages (i.e., Structured Text ST and Instruction List IL). In the field of PLC programming, there are guidelines from the communities of PLCopen or MISRA [9]. PLCopen is a worldwide initiative of different platform suppliers to reduce engineering effort and enhance software quality by providing standards, guidelines, and education for platform users in the community of industrial automation. The PLCopen guidelines, e.g., naming conventions or structuring with SFC, support in ensuring, e.g., consistency of automation software. MISRA provides rules and directives for coding and documentation that improve software maintainability. The MISRA guidelines, e.g., using start and end comment markers, support in, e.g., portability of automation software.

The aPS development is often validated with GAMP (Good Automated Manufacturing Practice) [10], which is a risk-based approach widely used in industry to regulate computerised systems. Risk Priority in the GAMP method (i.e., criticality estimation in risk analysis) is based on three factors Severity, Probability, and Detectability (cf. Figure 1). For each risk entry, ratings are given to each of the factors. Risk Priority is calculated with a multiplicative approach: factors are multiplied by each other in two stages. The factors are allocated to two levels: Detectability and Risk class are level 1 factors; Severity and Probability are level 2 factors. In the first stage, the Severity is offset against the Probability. This calculation determines the Risk class (1–3). In the second stage, the Risk class is then offset against the Detectability and results in a Risk Priority (low, medium, high).

Figure 1:

Factor hierarchy in the GAMP method [10].

2.2 Quality analysis of aPS software

In classical software engineering, various approaches are available to measure software characteristics, such as complexity or code documentation (i.e., code comments). It is due to high complexity or poor code documentation that might hinder software maintainability, which may introduce undesired additional costs [11]. Analogously, characteristics of aPS software need to be analysed to measure the criticality of documentation for the software. Vogel-Heuser et al. [12] proposed the MICOSE4aPS approach to measure the maturity of software considering different change types (e.g., bug fixing or new development). Wilch et al. [13] presented a semi-automatic concept to identify POU functionality based on characteristics of implementation or description. Fischer et al. [5] proposed an approach to compare the complexity of POUs using a metric for the Overall Complexity based on six sub-metrics, M ₁–M _6, as follows.

1. ProgramLength (M ₁) refers to Halstead’s Program Length, i.e., the sum of the tokens (operators and operands).

2. Cyclomatic Complexity (M ₂) refers to McCabe’s Cyclomatic Complexity determined by the number of decision statements.

3. FanIn_FanOut (M ₃) is determined by incoming (e.g., input variables) and outgoing (e.g., output variables) data flows.

4. Vocabulary Size (M ₄) and Difficulty (M ₅) follow the ideas of Halstead’s Vocabulary Size and Difficulty, which are also based on the number of (unique) operators and operands.

5. Data Structure (M ₆) targets the complexity of the data processed. For instance, the program interface variables add more complexity than local variables and sub-variables.

The calculation includes three steps. First, six metrics M ₁–M ₆ (M _i in (1)) are applied to each POU, resulting in six values with different scales per unit. The median values M ̃ 1 to M ̃ 6 are then determined. Second, the metrics values are scaled using equation (1) to the corresponding median.

Third, the Overall Complexity OC _rel of a POU is calculated based on the sum of the six metrics values scaled in the second step (cf. equation (2)). The weights w _i are used to adjust the influences of individual metrics on the overall complexity. The sum of w _i is 1 to scale Overall Complexity values to a predefined value range (e.g., (0, 1]).

2.3 Lack of control code documentation in the aPS domain

In classical software engineering, the phenomenon of insufficient documentation is referred to as technical debt (TD), more specifically, documentation debt [14]. The TD concept describes a context where a technical compromise is selected against better knowledge [15]. For instance, low-quality code is delivered to meet an urgent deadline. The technical compromise can enable a short-term benefit (e.g., quicker delivery) but may introduce some undesired effects (e.g., high maintenance cost). The survey of Li et al. [15] indicates that although documentation debt received significant attention, studies on documentation debt management are scarce. Avgeriou et al. [16] reported that TD always relates to cost. Inadequate monitoring of TD may introduce additional long-term effort in software projects. TD might cause project delivery delays, as well as a decrease in organisational profitability. Detofeno et al. [17] suggested that to prioritise TD in software, one should not only consider source code alone but also consider external factors such as project characteristics or test coverage.

Near the aPS domain, Martini et al. [18] analysed the accumulation of TD in software in embedded systems and additional activities (e.g., refactoring) required at five software companies. They reported that insufficient documentation “causes the misinterpretation by the developers implementing code” as well as hinders refactoring activities. However, code comments were not considered as the work focused on the level of design and architecture. A survey with six software companies from Besker et al. [11] indicated that time pressure commonly triggers documentation debt, such as insufficient updating related documentation for code implementation. The study reported a significant productivity loss of software developers due to TD. However, the code comment was not focused.

In the aPS domain, Besker et al. [14] investigated the work of software engineers at one company working with aPS in Scandinavia and reported that there were “quite a lot of resources in paying the interest on Technical Debt, on average 32 % of the development time”. Biffl et al. [19] studied the risks of TD in engineering artefacts related to data exchange (e.g., process documentation and configuration management). Waltersdorfer et al. [20] reported two types of documentation debt: Insufficient Data Model and Insufficient Product Process Resources Model. However, the code comment was not considered in the above studies.

Due to the nature of aPS, on-site changes are often performed to adapt the manufacturing systems to unanticipated raw materials or environmental conditions. Under pressure to start up the system, proper control code documentation for on-site adaptions might be neglected, resulting in documentation debt (cf. Section 1, e.g.). TD is causing additional development and maintenance costs over time [14] or developer productivity loss [11]. Due to the long lifetime of aPS, the impact from TD might be large. Unfortunately, awareness of TD in the aPS domain is still low in general, according to a survey of 48 German companies supplying aPS [21].

2.4 Research gap

Research addressing aPS documentation quality is still scarce. Among the publications, Neumann et al. [22] introduced templates to document design decisions at the architectural level of aPS. Neumann et al. [23] proposed some simple metrics such as HeaderCommentLinesOfCode (size of comment header), MultiLineComments (amount of comments wrapping over multiple lines), and SourceCodeCommentedRatio (density of comments) to assess the software maintainability. The authors’ previous work [7] mainly focused on methods for document exchange or on how code/configuration is generated from documents.

As aforementioned, to assist the engineers in maintaining, enhancing, or reusing the control code at a later stage, the comment header often provides an overview description of the POU, and the comments in the variable declaration and implementation section explain the control code in detail. A well-documented software supports its reusability in future projects; thus, the development costs could be optimised. Not only code comments, but a broad view should also be considered due to interdisciplinary constraints. Thus, the availability of documentation in different disciplines, the use of internal guidelines, or the automatic generation of engineering documents needs to be studied. Another factor in enabling reusability is standardisation, i.e., whether standardised guidelines provided by the community are followed. To the best of the authors’ knowledge, there is no method to determine the criticality or quality of documentation in the aPS domain. By analysing results from an industrial expert survey in machine and plant manufacturing, this work firstly studies the state of the practice regarding documentation. Secondly, factors influencing the documentation activities are derived considering the documentation debt concept to propose a risk priority indicator for identifying the documentation debt. An overview of findings from related work and expert survey corresponding to the sections is illustrated in Figure 2.

Figure 2:

Steps of the study and findings correspond to the sections.

4.1 Derived requirements of indicators for documentation debt

From the state of the art, requirements of indicators for documentation debt are derived in the following.

R1. RPI4DD should be extensible with new factors identified. The requirement for extensibility is due to TD research in the aPS domain being an ongoing work. The aspects of identifying TD, in particular documentation debt, are not fully explored yet. Thus, the list of factors is not yet fixed. When new factors are identified in future work, their calculation rules should be seamlessly integrated into RPI4DD.

R2. RPI4DD should be flexible with calculation rules of individual factors. Different calculation rules might be introduced to assess a factor due to the high diversity of applications in the context of the aPS domain. Therefore, flexibility is a requirement for RPI4DD.

R3. RPI4DD should be automatable. The program size of industrial projects is often large; thus, an automatic method is required to enable applicability in industrial practice.

4.2 Concept of a risk-based approach to indicate documentation debt

The concept of determining indicators for documentation debt is based on the GAMP method, which allows extensibility for additional factors that could be added to an appropriate level. Therefore, since the GAMP approach satisfies R1 (extensible) , the approach is used as a basis to derive RPI4DD. To satisfy R2 (flexible) , weights could be added to reflect the different importance of each factor. In addition, besides the multiplication operator, other operators (e.g., addition) could also be applied to the calculation of the factors.

In the following, based on the GAMP method, the six main steps to establish a new risk assessment method for documentation debt (cf. Figure 3) are described. This work focuses as a starting point on the control code documentation, in particular code comments and naming conventions. However, the design of RPI4DD shall enable the integration of further documentation artefacts, such as manuals or architect documents, in future work (cf. R1).

1. Step 1: Identify an initial set of factors influencing TD. In documentation in general and control code documentation in particular, a successful document often meets two criteria: (1) the document covers the necessary information of the object being described (i.e. Document Coherence) and (2) the document is available on time (i.e., Document Urgency).

2. Step 2: For each identified factor, identify the sub-factors influencing the factor. GAMP method stops at level 2 factors (cf. Figure 1); however, this step can be further applied for sub-factors if necessary. It should be noted that the ratings in the GAMP method may cause confusion. For example, a Risk class rating of 3 and low Detectability result in low Risk Priority, while a Risk class rating as 1 and low Detectability result in high Risk Priority. This paper does not aim to address the above issue but just proposes some modifications regarding the ratings. Still, the offset mechanism of the GAMP method is followed by grouping related (sub-)factors into classes, but the classes’ rating is based on Roman numbers. Besides the three-level rating of the GAMP method, the rating of (sub-)factors is based on Arabic numerals. Thus, all low Arabic number ratings indicate low risk and vice versa. In addition, the ranges of the (sub-)factors’ rating are more flexible since some (sub-)factors might need only two or more than three levels.

3. Step 3: Snowballing to extend the factor classification. Related aspects of identified sub-factor are explored. For example, different Change Type of control code results in different change of comment which has different quality (i.e. Comment Quality). Thus, standards of comment (i.e. Comment Compliance) are included in the factor classification.

4. Step 4: Propose calculation rules for (sub-)factors. Once sub-factors are selected, the calculation rules are proposed.

5. Step 5: Conduct multi-stage assessments. The GAMP method starts with two sub-factors at level 2 (i.e. Severity and Probability) to assess their corresponding factor at level 1 (i.e. Risk class). Thus, the assessment might start from the lowest-level sub-factors and gradually move up the factor hierarchy. However, as mentioned, to enable flexibility, the calculation process and operators are free of choice.

6. Step 6: Visualise and interpret the results for further action. The value of RPI4DD is reviewed to indicate the areas if the document is insufficient, incomplete, or outdated, which indicates documentation debt.

Figure 3:

Steps to establish risk prioritisation indicator of documentation debt RPI4DD.

In the next section, the results of the approach are presented.

4.3 Factor classification of RPI4DD formulation

From the approach presented in Section 4.2, a basic structure to calculate RPI4DD is derived with two main influencing factors (cf. equation (3)). The Document Urgency factor (represented by RPI _Urgency ) assesses how acute the need for control code documentation is. RPI _Urgency is based on the assumption that the urgency can vary depending on the Functionality (RPI _{Functionality} ) implemented by the respective control code part, the Required Change On-site (RPI _On-site ), as well as the Grade of Test/Quality Of Test (RPI _Test ). The Document Coherence factor (RPI _Coherence ) assesses how coherent the existing documentation is. The expert responses reported a sub-optimal standardisation of programming and documentation guidelines, i.e., company-specific guidelines are more frequently used than the guidelines provided by the community (cf. Section 3). Thus, a conformity check of documentation with standards shall take place. A summary of the groups of sub-factors influencing the control code documentation is presented in Figure 4.

Figure 4:

Selected factors influencing control code documentation (i.e. factor classification of RPI4DD formulation) following the systematic classification of Li et al. [15] and GAMP procedure [10].

In the following, the specification of sub-factors and corresponding rationale are described in the order presented in Figure 4.

4.3.1 Document urgency (RPI _Urgency)

RPI _Urgency is calculated based on three sub-factors: (1) Functionality, (2) Required Change On-site, and (3) Grade Of Test/Quality Of Test.

4.3.1.1 Functionality (RPI _{Functionality})

Depending on the complexity of an implemented functionality, different amounts of explanatory documentation are required to enable maintainability by different persons [26]. Poor functional description availability (#72) poses a high risk since comprehensibility (readability) received the highest rank among the software properties influenced by complexity [5]. Therefore, Complexity Of Software is identified as a sub-factor of Functionality. The Overall Complexity metric proposed in [5] has proven as a reliable means to quantify different aspects of software complexity and, thus, could be used as a calculation rule for the sub-factor Complexity Of Software. As identified in [5], the list of characteristics includes:

1. ProgramLength M ₁

2. Cyclomatic Complexity M ₂

3. FanIn_FanOut M ₃

4. Vocabulary Size M ₄

5. Difficulty M ₅

6. Data Structure M ₆

It is worth noting that Cyclomatic Complexity or Data Structure may be weighted more strongly as they might have larger effects on the program complexity.

Second, the difficulty of software functionalities might vary [5]. As documentation activity is quite laborious and tedious, the documentation effort needs to be prioritised for the most difficult functionality (i.e., goal-oriented behavior [13]) of software. Therefore, Difficulty Of Functionality is included as another sub-factor of RPI _{Functionality}. As identified in [5, 13, 27], the list of functionalities is as follows.

1. Organizational (low weight = 1).

2. Messages (low weight = 1).

3. Status (low weight = 1).

4. Error/generic/homing (medium weight = 2)

5. Motion Control (high weight = 3)

6. Safety-related (high weight = 3).

Motion control in aPS is highly challenging in case a production step (e.g., work piece handling) requires the synchronisation of multiple drives performing different motion tasks (e.g., rotatory or linear movements [28]) by simultaneously fulfilling hard real-time requirements [27]. According to [1], positioning tasks of the robot-like-systems components are rated as one of the most challenging tasks to implement in software. Changes in production (e.g., introduction of a new product) might require additional or more complex movement; thus, the motion control must be adapted accordingly (e.g., changing the number of involved cooperating motion axes of robot arms). The difficulty of motion control is therefore rated as high (2). Safety-related tasks are even more critical since errors or malfunctions in these software parts may cause severe damage to aPS, or – in the worst case – even to humans; thus, they are rated with 3. Here, poor automatic control code generation (#51 and #52) might pose a high risk, e.g., safety-related control code might differ from the predefined model or requirements document.

Following the calculation rules presented in [5, 28] for POU complexity, a calculation for RPI _{Functionality} is proposed as follows. As the medians M ̃ 1 to M ̃ 6 are used (cf. Section 2.2), they are “stable against outliers and provide reliable results” [5]. Therefore, the distribution of Overall Complexity OC values would be mostly symmetrical without outliers. Thus, RPI _{Functionality} is based on the mean Overall Complexity O C ̄ .

(4) R P I Functionality = O C Organizational ̄ + O C Messages ̄ + O C Status ̄ + 2 ⋅ O C Error ̄ + 2 ⋅ O C Motion ̄ + 3 ⋅ O C Safety ̄

4.3.1.2 Required change on-site (RPI _On-site)

The sub-factors of Required Change On-site include Manufacturing Type, Change Frequency and Change Source. First, while MM companies can usually develop machines entirely in-house, PM companies often have to integrate and start up the system on-site [8]. Thus, the Manufacturing Type is identified as a sub-factor for Required Change On-site. Second, customers have different schedules (e.g., new product introductions); therefore, the remote software update intervals (i.e., Change Frequency) vary. Third, while in-house changes often undergo a rigorous review process by the software development department and management, on-site changes are often conducted without feedback from the development department (e.g., due to time pressure). Therefore, the qualification of the person conducting a change or the origin of changes (Change Source) is identified as another sub-factor. Other potential sub-factors include code-sharing strategies, acquiring software status, or company sizes (documentation availability might vary in sizes of companies, according to #72).

Regarding the Manufacturing Type, the highest document urgency is rated at PM (rating = 3) as PM has the most required change on site. As on-site changes must be quickly performed (e.g., to meet hard deadlines or to reduce the cost of suspending production), it is urgent to provide exact instructions. The rating is lower for series machinery manufacturing (rating = 1) and special-purpose machinery manufacturing (rating = 2).

Regarding Change Frequency, the rating scale is based on the frequency of software updates reported in [1]:

1. <3 months = 1

2. 3–6 months = 2

3. 6–12 months = 3

4. > 12 months = 4

5. No update = 5

Regarding Change Source, a rating of 1 is assigned to Changes conducted in-house (less risky due to rigorous review), and On-site changes are given a rating of 2.

The assessment on Required Change On-site is described in Figure 5. Following the idea in the GAMP method, this assessment includes two steps with three aspects. Firstly, the Manufacturing Type (severity of change) is plotted against the Change Frequency (probability that a change will occur), giving a Required Change Class (I to III). Secondly, the Required Change Class is plotted against Change Source, thus resulting in giving a degree of Required Change On-site (low, medium, high). There is an adaption in this assessment that the detectability aspect introduced in the GAMP method is replaced by Change Source. Nevertheless, the detectability concept is still included since the detectability of inadequate documentation may vary with different Change Sources, which might undergo different review processes.

Figure 5:

Required Change On-site assessment according to the GAMP method [10] (a: Required Change class calculation; b: Required Change On-site calculation).

Finally, the value RPI _On-site is determined using equation (5).

(5) R P I O n ‐ s i t e = 1 5 10 i f i f i f l o w u r g e n c y o n ‐ s i t e m e d i u m u r g e n c y o n ‐ s i t e h i g h u r g e n c y o n ‐ s i t e

4.3.1.3 Grade Of Test/Quality Of Test (RPI _Test)

In the aPS domain, the method proposed by Ulewicz et al. [29] can be followed to identify which control code snippets are not well tested yet. The process employed the control code coverage method to determine which control code snippets are covered by which test cases. Among the proposed metrics in [29], there is a test coverage measure on the POU hierarchy (i.e., TestDepth) and three control code coverage measures (i.e., BranchCoverage, StatementCoverage, and PathCoverage). With StatementCoverage, it is aimed to traverse all the statements (or lines) at least once. BranchCoverage aims to traverse all the control flow paths (e.g., if statements) at least once. The main goal of PathCoverage is to execute all possible control flow paths (full paths from input to output). A brief comparison of these control code coverage metrics is illustrated in Figure 6. With test cases TC1 and TC2 (cf. Figure 6a), StatementCoverage shows 100 %; however, only 75 % of the branches are covered (BranchCoverage). Thus, BranchCoverage is a better indicator as it shows that more test cases are required. Regarding PathCoverage, there is a case that the number of paths through the loop could be significantly large (cf. Figure 6b), infinite, or not predictable (e.g., number of loops is determined at run time). Thus, covering all possible paths from input to output is mostly not realisable in practice. Therefore, among these control code coverage metrics, BranchCoverage is recommended as it delivers more accurate results.

Figure 6:

Comparison on control code coverage metrics proposed in [29] (a: Branch and Statement; b: Path).

In practice, 80 % coverage is generally accepted as good (following the Pareto principle in testing: given that 80 % of code is covered, the remaining code part may require a significant effort but is not worth it). To distinguish between a poor and a moderate test quality, additional information needs to be taken into account. More precisely, the expectation for test quality depends on various company-specific boundary conditions, such as software modularity or reusability. For instance, the boundary conditions at the industrial partner in the previous study [25] are as follows: the company has a high-modularised structure of its control code and a high degree of reuse by applying library modules and templates (about 75 % of a machine’s PLC project consist of reused control code). The other 25 % of the machine’s PLC project is newly developed machine- or customer-specific control code, e.g., data exchange between the reused library modules or implementation of the machine-specific process logic. These newly developed, machine-specific parts are the error-prone parts of the PLC project, which require thorough testing. Generally, the company’s engineers know which requirements target these new code parts, and due to the company’s mature code structure, they can locate the code parts in the PLC project. Consequently, the defined tests usually aim at targeting specifically the requirements linked to these critical code parts. Again, following the Pareto principle, at least 80 % of the 25 % new machine-specific code needs to be covered by tests. Since the 80 % of the 25 % new machine-specific code is about 20 % of the total code, the company-specific threshold for moderate test quality is set to 20 % of the entire control code in the considered PLC project. The threshold of 20 % for “moderate” is set given the assumption that test cases for the machine-specific part are selected and that the remaining code is already well-tested and thus assumed to be less error-prone. Therefore, a proposal to assess the test quality with two metrics, TestDepth and BranchCoverage, is shown in equation (6).

(6) T e s t Q u a l i t y = g o o d i f T e s t D e p t h ≥ 80 % ∧ B r a n c h C o v e r a g e ≥ 80 % m o d e r a t e i f 20 % ≤ T e s t D e p t h , B r a n c h C o v e r a g e < 80 % p o o r i f T e s t D e p t h , B r a n c h C o v e r a g e < 20 %

Finally, the value RPI _Test is determined using equation (7).

(7) R P I Test = 1 5 10 i f i f i f g o o d t e s t q u a l i t y m o d e r a t e t e s t q u a l i t y p o o r t e s t q u a l i t y

As this paper’s scope focuses on control code documentation, there is no claim for completeness regarding test metrics. Nevertheless, many test metrics are available in the literature that can be integrated into the RPI _Test calculation.

4.3.2 Document Coherence (RPI _Coherence)

The Document Coherence factor is assessed based on the three sub-factors Comment Compliance, Comment Quality and Change Type. Firstly, practice shows that implementation of the standards could reduce errors. Violation of coding standards (e.g., MISRA) may result in TD [30]. A lack of exact instructions (#77) or a lack of coding guidelines in general (#34) might pose a risk, e.g., poor quality or standardised code. Therefore, Comment Compliance is identified as a sub-factor, which is categorised into the Document Coherence factor. Secondly, the coherence of control code documentation and corresponding implementation may influence the software’s maintainability. Insufficient Comment Quality (e.g., code comments do not describe the implementation properly) may hinder on comprehensibility (readability) of the control code (e.g., causing confusion for the maintenance staff). Thirdly, different change categories (i.e., Enhancements, Bug Fixes, New Features, and New Developments) could influence the software maturity value, according to [12]. Thus, Change Type is identified as a factor within the Document Coherence factor.

Regarding Comment Compliance, the rating is based on results from checking the control code documentation with MISRA guidelines. MISRA rules or directives for comments include:

1. Rule 3.1 The character sequences /* and // shall not be used within a comment.

2. Rule 3.2 Line-splicing shall not be used in // comments.

3. Rule 20.1 #include directives should only be preceded by preprocessor directives or comments.

4. Dir 4.4 Sections of code should not be “commented out”

The details of the rating scale for Comment Compliance are presented in Table 5. Among the available guidelines, MISRA is proposed as its scope focuses on coding standards. As an alternative or a broader scope, one could refer to other guidelines or standards such as PLCopen, ISO 26262 or ISO 17961.

Table 5:

Rating scale for Comment Compliance with guidelines or standards.

Rating	Description	Comment Compliance from checking with guidelines (e.g., MISRA)
1	Noncritical	Fully compliance (i.e., no error or warning messages)
2	Minor	Partially compliance (i.e., violations only include warning messages)
3	Critical	Noncompliance (i.e., violations include error messages)

Regarding Comment Quality (CQ) rating, it is assumed that a high amount of comments is beneficial to enhance the software’s understandability, and, thus, reduces the risk of documentation debt. The metrics HeaderCommentLinesOfCode (size of comment header), MultiLineComments (amount of comments wrapping over multiple lines), and SourceCodeCommentedRatio (density of comments) proposed in [23] could be employed to quantify the amount of control code comments (cf. equation (8)).

(8) C Q = 1 2 3 i f i f i f S o u r c e C o d e C o m m e n t e d R a t i o > 0 S o u r c e C o d e C o m m e n t e d R a t i o > 0 S o u r c e C o d e C o m m e n t e d R a t i o = 0 ∧ M u l t i L i n e C o m m e n t s > 0 ∧ H e a d e r C o m m e n t L i n e s O f C o d e > 0 ∧ ( M u l t i L i n e C o m m e n t s = 0 ∨ H e a d e r C o m m e n t L i n e s O f C o d e = 0 )

Regarding Change Type, the scale refers to the change categories reported in [12]. The details of the rating scale are presented in Table 6.

Table 6:

Rating scale for Change Type based on scope and criticality.

Rating	Description	Rationale according to [12]
1	Enhancements	[…] represent the least critical category […]
2	Bug fixes	[…] represent a more critical […] than enhancements […]
3	New features	[…] significantly more critical , […]
4	New developments	[…] the most extensive change category […]

1. The bold terms highlight the exact phases extracted from ref. [12] explaining the rationale for the corresponding ratings in column 1.

The Document Coherence assessment method is described in Figure 7. Following the idea of the GAMP method, firstly, the Comment Quality (determined using equation (8)) is offset against the Comment Compliance (determined using the rating scale in Table 5). This calculation determines the degree of excellence of comment, i.e., Comment Class (I to III). Secondly, the Comment Class is offset against the degree of excellence of control code (represented by Change Type, which influences the software maturity value). It results in Document Coherence (low, medium, high). The Change Type is determined using the rating scale in Table 6. It could be observed that the three aspects defined in GAMP (i.e., severity, probability, and detectability) are not used but replaced by the three new aspects in the context of coherent assessment.

Figure 7:

Document Coherence assessment according to the GAMP method [10] (a: Comment Class calculation; b: Document Coherence calculation).

Finally, the RPI _Coherence is determined with equation (9).

(9) R P I Coherence = 1 5 10 i f i f i f h i g h d o c u m e n t c o h e r e n c e m e d i u m d o c u m e n t c o h e r e n c e l o w d o c u m e n t c o h e r e n c e

4.4 Proof of concept of risk prioritisation indicator of documentation debt

To illustrate the applicability, RPI4DD is in the following determined for two scenarios of the extended Pick and Place Unit (xPPU) [8], i.e., a lab-sized demonstrator that stamps, transports, and sorts work pieces with different color, weight, and material (cf. Figure 8). The work pieces arrive at the stack, where they are picked up by the crane. Depending on the work piece type, they are either further processed to the stamp or directly transported with the conveyor to be sorted into the respective ramp.

Figure 8:

Extended Pick and Place Unit [8] to apply the risk priority indicator RPI4DD.

In the following, RPI4DD is calculated for two typical functionalities in aPS emulated for the xPPU, i.e.:

1. F_Stamp: Stamping work pieces: This scenario refers to the code parts controlling a typical functionality of the system during regular production, i.e., in the case of the xPPU, the stamping of work pieces at the crane before they are sorted.

2. F_Restart: Restart operation after emergency stop: The recovery of the system after an emergency stop to take up regular production is a highly challenging task in industrial practice, e.g., due to complex resynchronisation of drives, which is why the scenario is considered on a small scale for the xPPU.

Table 7 follows a practical risk assessment template used to evaluate risks during the development of industrial aPS. The table template can serve different views to different stakeholders. The individual sub-factors are intended to be initially determined by technicians on a fine-grained level resulting in precise numbers for the different factors (e.g., results from complexity metrics). Next, the fine-grained values are clustered and categorised into a coarse-grained system (e.g., following a three-level categorisation into low, medium, and high) to make the risk priority number intuitively understandable at first glance, e.g., for customers or management positions.

Table 7:

A (fictive) example on RPI4DD calculation on two xPPU functionalities Stamping work pieces (F_Stamp) and Restart operation after emergency stop (F_Restart).

1. (^a) cf. Figure 7 and equation (9) for calculation rules of Document Coherence. (^b) Risk Priority Risk of Documentation is assumed. Low if RPI4DD ≤ 625 (5 × 5 × 5 × 5). Medium if 625< RPI4DD < 1250 (10 × 5 × 5 × 5). High if RPI4DD ≥ 1250.

In the following, the sub-factors are determined for both scenarios to derive RPI4DD for the software parts implementing the respective functionalities.

4.4.2 Calculation of Document Coherence (RPI _Coherence)

Regarding the Comment Compliance, the scopes of both functionalities are not trivial; therefore, it is assumed that there are error messages associated with the Comment Compliance rules listed in Section 4.3. For the fictive examples focused for the RPI calculation for the xPPU, it is assumed that both contain comments that are non-compliant with the MISRA comment guidelines, e.g., in the implementation of the crane (cf. Figure 9). These non-compliant violations are critical for both functionalities F_Stamp and F_Restart, thus are rated with 3 for Comment Compliance. It should be noted that there are various configurable tools are available to check compliance with MISRA rules, enabling automated assessment for this criterion (cf. requirement R3).

Figure 9:

A (fictive) example of non-compliant comment based on MISRA rule 3.1 example [9] and xPPU [8].

Regarding Comment Quality, as aforementioned, F_Restart scope is larger and relates to plant manufacturing which mostly requires on-site changes, while F_Stamp scope mostly relates to series machine manufacturing which could be mainly developed in-house. Therefore, it is assumed that the three metrics HeaderCommentLinesOfCode, MultiLineComments, and SourceCodeCommentedRatio deliver better results at F_Stamp than F_Restart. For instance, some header comments in F_Restart on-site changes might be neglected due to high time pressure and short deadline, resulting in HeaderCommentLinesOfCode = 0. Thus, the Comment Quality of F_Restart appears to be larger than F_Stamp, according to calculation rules in equation (8). Hence, the Comment Quality of F_Stamp and F_Restart are rated as 1 and 2, respectively.

The Comment Class is determined by offsetting the Comment Compliance against Comment Quality (cf. detailed calculation rules in Figure 7a).

Regarding the Change Type, a change at F_Restart often requires a configuration of multiple tasks at multiple components; therefore, it is assumed that a change at F_Restart is generally larger than a change at F_Stamp (e.g., mainly New Features at F_Restart and mainly Enhancements at F_Stamp). Thus, according to the rating scale in Table 6, Change Type of F_Stamp and F_Restart are rated with 1 and 3, respectively.

Altogether, the RPI4DD is determined by multiplying the RPI _Urgency and RPI _Coherence, according to equation (3). The RPI4DD results indicate that the risk of documentation debt at F_Restart (2500) is higher than at F_Stamp (50). Thus, document improvements or actions to prevent damage from documentation TD related to F_Restart should be planned for corresponding module developers or application engineers. When transferring the results received for the xPPU to real-world production systems, companies may benefit by RPI4DD as a quantitative indicator for different stakeholders (e.g., from management or software development) to prioritize starting points for reducing the amount of documentation debt and, thus, avoid high long-term cost.

4.5 Summary of RPI4DD calculation and requirement fulfilment

The calculation for RPI4DD is summarised as follows (cf. equation (10)). The range of each factor RPI _{Functionality}, RPI _On-site, RPI _Test and RPI _Coherence is (0, 10] with an assumption that all operands are equally critical for TD. This proposal employs the widely-used scale up to 10 for these factors to simplify the calculations. To scale the RPI to another range, one can use weights (cf. RPI _{Functionality} calculation) or offset the factors (cf. RPI _On-site calculation). It could be observed that the range of RPI _Urgency (multiplication of RPI _{Functionality}, RPI _On-site and RPI _Test) (0, 1000] is larger than RPI _Coherence range (0, 10]. It is due to the criticality of documentation being considered more important than the coherence of documentation. For example, if a commissioner has to make changes at the construction site at the customer’s site, it is crucial that documentation is available (urgent), but it is not a priority whether the comments fulfil the MISRA guidelines (coherent). It is worth noting that the factor Grade Of Test/Quality Of Test could be optional. RPI _Test is only relevant in case changes are conducted in-house (e.g., by module developer or application engineer), and test results are available for assessment. RPI _Test acts as a considered issue to tweak RPI4DD. Square brackets denote the optionality of RPI _Test.

The RPI4DD approach is achieved with a set of selected factors to indicate control code documentation debt in aPS (cf. Figure 4), which is extensible (R1), flexible (R2), and automatable (R3), therefore, feasible for industrial large-scale software projects, which often involve hundreds of POUs. The extensibility and flexibility characteristics presented in the Required Change On-site assessment (cf. Section 4.4) can generally be applied to assessments of other aspects (cf. Facility Availability and Manufacturing Type examples in Required Change On-site in Section 4.4). New sub-factors can be systematically included in the proposed hierarchy (cf. Figure 4). All proposed rating scales or calculation rules are adjustable. Thus, R1 (extensible) and R2 (flexible) are satisfied. R3 (automatable) is evaluated as partially fulfilled since all presented calculation rules can be automated except Change Source, which might require a manual assessment. Nevertheless, with trivial effort, semi-automatic assessment methods can be developed to classify Change Source (e.g., by a configuration file).

To summarise, the integration of RPI4DD into the risk management process of machine and plant manufacturing companies yields high potential to support practitioners in identifying and analysing the risks associated with documentation debt in technical systems by providing an intuitive quantitative indicator based on established risk assessment approaches. Therefore, earlier reaction or risk-reduction measures can be developed to enhance the system quality and drastically reduce costs by preventing errors and mal-functions caused by a lack of documentation in early phases.