Keywords

1 Introduction

The introduction of the new Industry 4.0 paradigm has recently allowed the increasingly high dissemination of advanced technologies for the improvement of the industrial processes [1]. However, the technological advancements that are taking place should go hand in hand with a rapid effective training of the industrial workers so that this change can be supported by a proper alignment of the skills to the new needs of the companies [2]. In this context, the issue of the evaluation of the training provided to the workers is increasingly critical, since more and more complex data have to be collected and properly analyzed in order to define the skill level of the workers and introduce in case the necessary corrective actions [3]. For this reason, the identification and measurement of data directly from the shop floor becomes extremely important. In fact, this data can provide real-time indications about the workers’ performances following training interventions implemented by the company. The problem is hence twofold. On one hand, it is necessary to understand the approach that is needed to collect and manage a potentially huge amount of data from the field, in order to make their elaboration automatic. On the other hand, the development of models that can provide a summary evaluation of the evolution over time of the worker skill levels on the basis of the available data is certainly a non-trivial task.

An answer to this problem can be represented by the integration of the so-called semantic technologies with training evaluation models. The former are in their turn one of the most relevant enabler of the Industry 4.0 paradigm and can provide the structure that is necessary to model, enrich and make interoperable the data collected from the shop floor, while the latter can address the concurrent creation and elaboration of summary indications on the skill levels that can be easily interpreted and used by the managers of the company. In the light of this final objective, the paper aims at presenting the Training Data Evaluation Tool (TDET), i.e. an extension of the previously developed SatisFactory Ontology (SFO) with an analytical model for the definition of the skill levels of the workers. In particular, in Sect. 2 the SFO is briefly introduced, as well as a literature review about semantic technologies and the models for training evaluation. In Sect. 3.1, the extension of the SFO, i.e. the Training Evaluation Ontology (TEO), is presented. In Sect. 3.2, the Training Analytics Model (TAM) for the evaluation of training and skills is introduced. Finally, in Sect. 4, the conclusions and the next steps of the research are reported.

2 Relation to Existing Theories and Work

2.1 Semantic Technologies and the SFO

Nowadays, the areas of knowledge representation (KR) and knowledge management (KM) are gaining from the use of semantic technologies over conventional approaches. Semantic models, such as ontologies, play an important role for many knowledge-intensive applications since they provide a formal representations of domain knowledge [4]. Organizations that use the languages and standards of the semantic Web, i.e. RDF, RDFS, OWL, SPARQL/RIF, aim to integrate existing information assets, using the best practices of linked data and the open world assumption, aiming to enhance their knowledge management system. However, despite the common belief that semantic technologies might be limited to cloud computing and big data, they are equivalently useful to private or proprietary data. Ontologies provide, indeed, a formal and ubiquitous information artefact aiming to make all the elements of a domain and their relations explicit [5].

The SatisFactory Ontology (SFO) presented in this work has been developed in the framework of the H2020 funded SatisFactory project. The SFO has an upper structure developed to gather and manage manufacturing knowledge, mainly focused on processes and assets at shop floor level. Therefore, the model is specialized in two different directions (clepsydra-like shape, see Fig. 1): (i) the data structure-oriented level that enhances with semantics the xml schemas used to exchange data within the SatisFactory ecosystem; (ii) the shop floor-oriented level that models the elements and terms characterizing the specific manufacturing environment.

Fig. 1.
figure 1

SatisFactory network of ontologies

Full size image

Lastly, the use of semantic technologies, and in particular the exploitation of the presented model, is not meant to be a replacement for existing information technologies, but rather an added layer that can leverage those assets for semantic interoperability.

2.2 Models for Training Evaluation

Different models for training evaluation have been developed and used so far. The most widespread and commonly accepted is the Kirkpatrick’s [6] four-levels model [7]. The four levels of the model are Reaction, Learning, Employee Behaviour and Organizational Results. The first level evaluates the reaction of the trainees on the training activity. The second level evaluates the knowledge and skills acquired by the trainees. The third level evaluates the transfer of the knowledge and skills on the job. The fourth level evaluates the overall impact on the company/business unit in terms of economic and/or organizational performances. In particular, the two last levels should be considered as the most important for a company/business unit [8].

Over time, various changes to the Kirkpatrick’s model have been proposed. For example, Hamblin [9] proposed five levels, by splitting the two last levels of Kirkpatrick’s model in Organization and Ultimate value. Guskey [10] also presented a five levels model where the first two levels correspond to Kirkpatrick’s ones, while the other three are Organizational support and Learning, Participant use of knowledge and skills, and Participant learning outcomes. The Nine outcomes model [11] presented a list of nine items (Reaction, Satisfaction, Knowledge, Skills, Attitude, Behaviour, Results, Return on investment, Psychological capital) to be checked in order to evaluate the overall impact of training activities, where all the concepts of Kirkpatrick’s model are included.

The Kirkpatrick’s model, together with the variations described above, represent conceptual approaches to be followed in a general training evaluation. From an analytical perspective, the few contributions available are focused on the estimation of the ROI as in the case of Phillips [12], therefore leaving to the single intervention/company the specific measurement of the impact, which is usually unstructured and targeting only the first two levels of Kirkpatrick’s model. For this reason, the TDET described in the following section aims at providing a mechanism to collect and reuse the data from the shop floor in order to provide analytical indications about the impact of the training and the skill level reached by the trainees involved in the program.

3 Training Data Evaluation Tool (TDET)

The TEO should be perceived as a further enrichment of the actual SatisFactory ontology. In particular, the lowest level (or data structure-oriented) semantic model is extended with concepts that support the management and analysis of training data. This is due to the need of enriching with semantics training data coming from the shop floor, which are collected from heterogeneous sources (Fig. 2)

Fig. 2.
figure 2

Overall architecture

Full size image

The TDET aims at producing a semantics-driven classification and evaluation of the trainees’ expertise. As a first step, the so-called Training Evaluation Ontology (TEO) drives the semantic enrichment of the training activities and their evaluation data coming from the shop floor. Then, the model underpinning the quantitative evaluation is addressed by the Training Analytics Model (TAM) that exploits such semantically enriched data.

3.1 Training Evaluation Ontology (TEO)

The data structure-oriented ontology model, as conceived in the SFO, has been extended with the following concepts: (i) Training activity; (ii) KPI, (iii) KPI_Category, (iv) KPI_Score; (v) KPI Type; (vi) KPI_Focus. These, together with the specification of their inter-links, represent the pillars of the trainee expertise evaluation (see Fig. 3).

Fig. 3.
figure 3

Training Evaluation Ontology

Full size image

The evaluation of the worker’s expertise level exploits the semantic structure presented in Fig. 4, In particular each instance of the KIP_Score should be perceived according to the following statement: One trainer may have several KPI scores. The latter ubiquitously describes (refers to) one kind of KPI and one specific activity. The KPI score has a (xsd:double) value.

Fig. 4.
figure 4

Semantic structure of the KPI score

Full size image

3.2 Training Analytics Model (TAM)

The quantitative evaluation of the training and the skill classification is addressed by the TAM, which partially stems from the work of Kiritsis et al. [13], who proposed the ActionPlanT Industrial Learning (IL) methodology for the implementation and evaluation of IL actions specifically addressing the last developments of ICT for manufacturing. It is in this framework that an evaluation approach based on Kirkpatrick’s training evaluation model was presented. In particular, the first three levels were targeted, namely Reaction, Learning and Behaviour, and a weighted sum model (WSM) to quantify the impact of the training was proposed. However, the ActionPlanT IL WSM had some limitations:

  • The performance indicators (KPIs) identified for each level were based only on answers to questionnaires and not on data retrieved from the shop floor

  • The KPIs were only at the level of the single trainee, with no KPIs to evaluate the performances of the team (i.e. Organizational level)

  • No mechanism to evaluate the skill level of the trainee according to a pre-defined target and the performances of the other trainees was provided

Therefore, on this basis a set of KPIs based on data to be retrieved automatically from the shop floor was developed for each level of the Kirkpatrick’s training evaluation model (i.e. Reaction, Learning, Behaviour, and Organization). Furthermore, a mechanism able to assign each trainee to a given skill level (Low, Medium, and High) based on the comparison with both a target level and the performances of the other trainees was designed.

In particular, on the basis of the KPIs identified, a summary indicator summarizing the overall performance of the trainee i at time j for the KPI k was defined:

$$ {\text{P}}_{\text{ijk}} = \left[ {\left( {\frac{{{\text{V}}_{\text{ijk}} }}{{{\text{AV}}_{\text{jk}} }}} \right) * {\text{WR}}_{\text{jk}} + \left( {\frac{{{\text{V}}_{\text{ijk}} }}{{{\text{TV}}_{\text{jk}} }}} \right) * {\text{WA}}_{\text{jk}} } \right] *100 . $$
(1)

Vijk/AVjk is the ratio between the value of KPI k for employee i at time j (Vijk) and the average of the values of KPI k of the n employees at time j (AVjk). If this ratio is higher than 1 it means that the trainee i at time j for the KPI k is performing better than the average, if it is lower than 1 that is performing worse. Vijk/TVjk is the ratio between the value of KPI k for employee i at time j (Vijk) and the target value of KPI k at time j (TVjk) established by the management. If this ratio is higher than 1 it means that the trainee i at time j for the KPI k is performing better than the target value, if it is lower than 1 that is performing worse. WRjk is the weight related to the Value/Average ratio while WAjk is the weight related to the Value/Target ratio. These two weights can be balanced in order to give each time more importance to the performance of the single trainee compared with the average performance of the group or to the same performance compared with the pre-defined target value.

According to the value of Pijk, each trainee i can be classified according to three different skill levels, namely Low, Medium and High. Considering as a reference all the values of KPI k of the n employees at time j, the trainee’s skill level will be classified as Low if included between the zero and first quartile, as Medium if included between the first and third quartile and as High if included between the third and fourth quartile.

In order to aggregate the values of more KPIs for a given trainee i, standardized values of the single Pijk should be used, in order to take into account different average and target values. As a consequence, for each Pijk the following formula should be used:

$$ {\text{Pstd}}_{\text{ijk}} = ({\text{P}}_{\text{ijk}} - {\text{AV}}_{\text{jk}} )/{\text{SD}}_{\text{jk}} . $$
(2)

where SDjk is the standard deviation of the values of KPI k of the n employees at time j. On this basis, the following aggregated performance of the trainee i at time j can be formulated:

$$ {\text{AP}}_{\text{ij}} = \sum\nolimits_{\text{kj}} {\upalpha_{\text{kj}} * {\text{Pstd}}_{\text{ijk}} } . $$
(3)

where αk is the weight related to the KPI k at time j. This way it is possible to evaluate the overall skill level of the trainee i at time j, by using the same approach described above. Finally, the overall performances of the single team/group can be easily obtained by means of the averages previously calculated for the different evaluation and temporal levels.

4 Conclusions and Next Steps

With the introduction of the Industry 4.0 paradigm and of the related technologies, a huge effort is nowadays requested to manufacturing companies in order to continuously update the skill of the workers. For this reason, the relevance of training is increasing as well as the need to properly evaluate its effectiveness. In particular, thanks to the huge amount of data that can be retrieved from the shop floor through smart devices, sensors and HMIs, the evaluation of the training and skill levels can be now potentially made automatic with the provision of summary performance indicators to the management as an output. In order to pursue this final objective, the paper has presented the results of a preliminary work aiming at the integration of semantic technologies with training evaluation models in the Training Data Evaluation Tool (TDET), which put together the Training Evaluation Ontology (TEO) with the Training Analytics Model (TAM) for the summary evaluation of training and skills.

The TDET is suitable to the automatic elaboration of data retrieved from the shop floor and to the provision of a general evaluation of the given training activity by means of the aggregation of different KPIs developed ad hoc, which are related to all the four levels of the well-known Kirkpatrick’s model. In addition, the overall performance of a trainee can be computed, as well as his skill level (Low, Medium, High) according to a comparison with the performances of the other trainees. The overall performance and the skill level can be obtained for both a given time j or as a summary elaboration of the results over a longer time span.

The next steps of the research will include the deployment of the TDET through existing tools for ontologies and smart data management, and the evaluation of this framework on different industrial use cases in order to provide empirical evidence of the effectiveness of the presented approach.