Symbolic Knowledge Extraction and Injection with Sub-symbolic Predictors: A Systematic Literature Review

ML methods, and sub-symbolic approaches in general, represent data as (possibly multi-dimensional) arrays (e.g., vectors, matrices, or tensors) of real numbers and knowledge as functions over data. This is particularly relevant as opposed to symbolic knowledge representation approaches, which represent data via logic formulæ (see Section 2.2).

In spite of the fact that numbers are technically symbols as well, we cannot consider arrays and their functions as means for symbolic knowledge representation (KR). Indeed, according to [50], to be considered as symbolic, KR approaches should (a) involve a set of symbols (b) that can be combined (e.g., concatenated) in possibly infinite ways following precise grammatical rules and (c) where both elementary symbols and any admissible combination of them can be assigned with meaning—i.e., each symbol can be mapped into some entity from the domain at hand. Below, we discuss how sub-symbolic approaches most typically do not satisfy requirements 2.1.1 and 2.1.1.

Vectors, matrices, tensors. Multi-dimensional arrays are the basic brick of sub-symbolic data representation. More formally, a D-order array consists of an ordered container of real numbers, where D denotes the amount of indices required to locate each single item into the array. We may refer to 1-order arrays as vectors, 2-order arrays as matrices, and higher-order arrays as tensors.

In any given sub-symbolic data-representation task leveraging upon arrays, information may be carried by both (i) the actual numbers contained into the array, and (ii) their location into the array itself. In practice, the actual dimensions \((d_1 \times \ldots \times d_D)\) of the array play a central role as well. Indeed, sub-symbolic data processing is commonly tailored on arrays of fixed sizes—meaning that the actual values of \(d_1, \ldots , d_D\) are chosen at design time and never changed after that. This violates requirement 2.1.1 above, hence, we define sub-symbolic KR as the task of expressing data in the form of rigid arrays of numbers.

Local vs. distributed. When data is represented in the form of numeric arrays, the whole representation may be local or distributed [50]. In local representations, each single number into the array is characterised by a well-delimited meaning—i.e., it is measuring or describing a clearly identifiable concept from a given domain. Conversely, in distributed representations, each single item of the array is nearly meaningless, unless it is considered along with its neighbourhood—i.e., any other item that is ‘close’ in the indexing space of the array according to some given notion of closeness. Thus, while in local representations the location of each number in the array is mostly negligible, in distributed representations it is of paramount importance. Notably, distributed representations violate the aforementioned requirement 2.1.1. In recent literature, authors call ‘sub-symbolic’ those predictors who rely on distributed representations of data.

Depending on the predictor family of choice, the nature of the admissible hypothesis spaces and learning algorithms may vary dramatically as well as the predictive performance of the target predictor, and the whole efficiency of learning.

In the literature of machine learning, statistical learning, and data mining, a plethora of learning algorithms have been proposed through the years. Because of the ‘no free lunch’ (NFL) theorem [55], however, no algorithm is guaranteed to outperform the others in all possible scenarios. For this reason, the literature and the practice of data science keeps leveraging on algorithms and methods whose first proposal was published decades ago. The most notable algorithms include, among the many others, (deep) neural networks (NNs), decision trees (DTs), (generalised) linear models, nearest neighbours, support vector machines (SVMs), and random forests.

These algorithms can be categorised in several ways, for instance, depending (i) on the supervised learning task they support (classification vs. regression) or (ii) on the underlying strategy adopted for learning (e.g., gradient descent, least square optimisation).

Some learning algorithms (e.g., NNs) naturally target regression problems despite being adaptable to classification as well, whereas others (e.g., SVMs) target classification problems while being adaptable to regression as well. Similarly, some target multi-dimensional outputs ( \(\mathbf {y} \in \mathbb {R}^m\) and \(m\gt 1\) ), whereas others target mono-dimensional outputs ( \(m = 1\) ). Regressors are considered as the most general case, as other learning tasks can usually be defined in terms of mono-dimensional regression.

The learning strategy is inherently bound to the predictor family of choice. NNs, for instance, are trained via back-propagation [46] and stochastic gradient descent (SGD), generalised linear models via Gauss’s least squares method, decision trees via methods described in [9], and so forth. Even though all the aforementioned algorithms may appear interchangeable in principle because of the NFL theorem, their malleability is very different in practice. For instance, the least square method involves inverting matrices of order N, where N is the amount of available examples in the training set, making the computational complexity of learning more than quadratic in time. Furthermore, in practice, convergence of the method is not guaranteed in the general case; instead, it is guaranteed for generalised linear models, hence it is not adopted elsewhere. Thus, learning by least square optimisation may become impractical for big datasets or for predictor families outside the scope of generalised linear models. Conversely, the SGD method involves arbitrarily sized subsets of the dataset (i.e., batches) to be processed a finite (i.e., controllable) amount of times. Hence, the complexity of SGD can be finely controlled and adapted to the computational resources at hand, e.g., by making the learning process incremental and by avoiding all data to be loaded in memory. Moreover, SGD can be applied to several sorts of predictor families (including NNs and generalised linear models), as it only requires the target function to be differentiable with regard to its parameters. For all these reasons, despite the lack of optimality guarantees, SGD is considered to be very effective, scalable, and malleable in practice. Hence, it is extensively exploited in modern data science applications.

In the remainder of this subsection, we focus on two families of predictors, DTs and NNs, and their respective learning methods. We focus precisely on them because they are related to many surveyed SKE/SKI methods. DTs are noteworthy because of their user friendliness, whereas NNs are mostly popular because of their predictive performance and flexibility.

Decision trees. Decision trees are particular sorts of predictors supporting both classification and regression tasks. In their learning phase, the input space is recursively partitioned through a number of splits (i.e., decisions) based on the input data X in such a way that the prediction in each partition is constant and the error with regard to the expected outputs Y is minimal while keeping the total amount of partitions low as well. The whole procedure then synthesises a number of hierarchical decision rules to be followed whenever the prediction corresponding to any \(x \in \mathcal {X}\) must be computed. In the inference phase, decision rules are orderly evaluated from the root to a leaf, to select the portion of the input space \(\mathcal {X}\) containing x. As each leaf corresponds to a single portion of the input space, the whole procedure results in a single prediction for each x.

Unlike other families of predictors, the peculiarity of DTs lies in the particular outcome of the learning process—that is, the tree of decision rules—which is straightforwardly intelligible for humans and graphically representable in 2D charts. As further discussed in the remainder of the article, this property is of paramount importance whenever the inner operation of an automatic predictor must be interpreted and understood by a human agent.

Neural networks. Neural networks are biologically inspired computational models made of several elementary units (neurons) commonly interconnected into a directed acyclic graph (DAG) via weighted synapses. Accordingly, the most relevant aspects of NNs concern the inner operation of neurons and the particular architecture of their interconnection.

Neurons are very simple numeric computational units. They accept n scalar inputs \((x_1, \ldots , x_n) = \mathbf {x} \in \mathbb {R}^n\) weighted by as many scalar weights \((w_1, \ldots , w_n) = \mathbf {w} \in \mathbb {R}^n\) and they process the linear combination \(\mathbf {x} \cdot \mathbf {w}\) via an activation function \(\sigma : \mathbb {R} \mapsto \mathbb {R}\) , producing a scalar output \(y = \sigma (\mathbf {x} \cdot \mathbf {w})\) . The output of a neuron may become the input of many others, possibly forming networks of neurons having arbitrary topologies. These networks may be fed with any numeric information encoded as vectors of real numbers by simply letting a number of neurons produce constant outputs.

While virtually all topologies are admissible for NNs, not all are convenient. Many convenient architectures—roughly, patterns of well-studied topologies—have been proposed in the literature [51] to serve disparate purposes far beyond the scope of supervised machine learning. However, identification of the most appropriate architecture for any given task is non-trivial: recent efforts propose to learn their construction automatically [2, 33].

Most common NN architectures are feed-forward, meaning that neurons are organised in layers, where neurons from layer i can only accept ingoing synapses from neurons of layers \(j \lt i\) . The first layer is considered the input layer, which is used to feed the whole network. The last one is the output layer, where predictions are drawn. In NN architectures, inference lets information flow from the input to the output layer assuming the weights of synapses are fixed, whereas training lets information flow from the output to the input layer, causing the variation of weights to minimise the prediction error of the overall network.

The recent success of deep learning [20] has proved the flexibility and the predictive performance of deepneural networks (DNNs). ‘Deep’ here refers to the large amount of (possibly convolutional) layers. In other words, DNNs can learn how to apply cascades of convolutional operations to the input data. Convolutions let the network spot relevant features in the input data, at possibly different scales. This is why DNNs are good at solving complex pattern-recognition tasks—e.g., computer vision or speech recognition. However, unprecedented predictive performances of DNNs come at the cost of their increased internal complexity, non-inspectability, and greater data greediness.

Briefly speaking, an ML workflow is the process of producing a suitable predictor for the available data and the learning task at hand with the purpose of exploiting the predictor later to draw analyses or to drive decisions. Hence, any ML workflow is commonly described as composed of two major phases: training, in which predictors are fitted on data, and inference, in which predictors are exploited. However, in practice, further phases are included, such as data provisioning and pre-processing as well as model selection and assessment.

In other words, before using a sub-symbolic predictor in a real-world scenario, data scientists must ensure that it has been sufficiently trained and its predictive performance is sufficiently high. In turn, training requires (i) an adequate amount of data to be available; (ii) a family of predictors to be chosen (e.g., NNs, K-nearest neighbours, linear models); (iii) any structural hyper-parameter to be defined (e.g., amount, type, size of layers, K, maximum order of the polynomials); (iv) and any other learning parameter to be fixed (e.g., learning rate, momentum, batch size, epoch limit). Data must therefore be provisioned before training and possibly pre-processed to ease training itself, for example, by normalising data or by encoding non-numeric features into numeric form. The structure of the network must be defined in terms of (roughly) input, hidden, and output layers as well as their activation functions. Finally, hyper-parameters must be carefully tuned according to the data scientist’s experience and the time constraints and computational resources at hand.

Thus, from a coarse-grained perspective, an ML workflow can be conceived as composed of six major phases, enumerated as follows.

Many kinds of logic-based KR systems have been proposed over the years, mostly relying on first-order logic(FOL) either by restricting or extending it, e.g., on description logics and modal logics, which have been used to represent, for instance, terminological knowledge and time-dependent or subjective knowledge. Here, we briefly recall the state-of-the-art of FOL and its most relevant subsets.

First-order logic. FOL is a general-purpose logic that can be used to represent knowledge symbolically, in a very flexible way. More precisely, it allows both human and computational agents to express (i.e., write) the properties of, and the relations among, a set of entities constituting the domain of the discourse via one or more formulæ and, possibly, to reason over such formulæ by drawing inferences. Here, the domain of the discourse \(\mathbb {D}\) is the set of all relevant entities that should be represented in FOL to be amenable of formal treatment in a particular scenario.

Informally, the syntax for the general FOL formula is defined over the assumption that there exist: (i) a set of constant or function symbols, (ii) a set of predicate symbols, and (iii) a set of variables. Under this assumption, a FOL formula is any expression composed of a list of quantified variables, followed by a number of literals, i.e., predicates that may or may not be prefixed by the negation operator ( \(\lnot\) ). Literals are commonly combined into expressions via logic connectives, such as conjunction ( \(\wedge\) ), disjunction ( \(\vee\) ), implication ( \(\rightarrow\) ), or equivalence ( \(\leftrightarrow\) ).

Each predicate consists of a predicate symbol, possibly applied to one or more terms. Terms may be of three sorts: constants, functions, or variables. Constants represent entities from the domain of the discourse. In particular, each constant references a different entity. Functions are combinations of one or more entities via a function symbol. Similar to predicates, functions may carry one or more terms. Being containers of terms, functions enable the creation of arbitrarily complex data structures combining several elementary terms into composite ones. This kind of composability by recursion is what makes the aforementioned definition of ‘symbolic’ valid for FOL. Finally, variables are placeholders for unknown terms, i.e., for either individual entities or groups of entities.

Predicates and terms are very flexible tools to represent knowledge. While terms can be used to represent or reference either entities or groups of entities from the domain of the discourse, predicates can be used to represent relations among entities or the properties of each single entity.

Intensional vs. extensional. In logic, one may define concepts—i.e., describe data—either extensionally or intensionally. Extensional definitions are direct representations of data. In the particular case of FOL, this implies defining a relation or set by explicitly mentioning the entities it involves. Conversely, intensional definitions are indirect representations of data. In the particular case of FOL, this implies defining a relation or set by describing its elements via other relations or sets. Recursive intensional predicates are very expressive and powerful, as they enable the description of infinite sets via a finite (and commonly small) amount of formulæ. This is one of the key benefits of FOL as a means for KR.

Tractability deals with the theoretical question: Can a logic reasoner compute whether a logic formula is true (or not) in reasonable time? Such aspects are deeply entangled with the particular reasoner of choice. Depending on which and how many features a logic includes, it may be more or less expressive. The higher the expressiveness, the more the complexity of the problems that may be represented via logic and processed via inference increases. This opens the possibility for the solver to meet queries that cannot be answered in practical time or by relying upon a limited amount of memory—or just cannot get an answer at all. Roughly speaking, more expressive logic languages make it easier for human beings to describe a particular domain, usually requiring them to write less and more concise clauses at the expense of higher difficulty for software agents to draw inferences autonomously, because of computational tractability. This is a well-understood phenomenon in both computer science and computational logic [8, 31], often referred to as the expressiveness/tractability trade-off.

FOL, in particular, is considered very expressive. Indeed, it comes with many undecidable, semi-decidable, or simply intractable properties. Hence, several relevant subsets of FOL have been identified in the literature, often sacrificing expressiveness for tractability. Major notions concerning these logics are recalled below.

Horn logic. Horn logic is a notable subset of FOL, characterised by a good trade-off among theoretical expressiveness and practical tractability [36].

Horn logic is designed around the notion of the Horn clause [26]. Horn clauses are FOL formulæ having no quantifiers and consisting of a disjunction of predicates, where only at most one literal is non-negated—or, equivalently, an implication having a single predicate as post-condition and a conjunction of predicates as pre-condition: \(h \leftarrow b_1,\ \ldots ,\ b_n\) . Here, \(\leftarrow\) denotes logic implication from right to left, commas denote logic conjunction, and all \(b_i\) , as well as h, are predicates of arbitrary arity, possibly carrying FOL terms of any sort—i.e., variables, constants, or functions. Horn clauses are thus if–then rules written in reverse order and only supporting conjunctions of predicates as pre-conditions.

Essentially, Horn logic is a very restricted subset of FOL where (i) formulæ are reduced to clauses, as they can only contain predicates, conjunctions, and a single implication operator; therefore, (ii) operators such as \(\vee\) , \(\leftrightarrow\) , or \(\lnot\) cannot be used; (iii) variables are implicitly quantified; and (iv) terms work as in FOL.

Datalog. Datalog is a restricted subset of FOL [3] representing knowledge via function-free Horn clauses, defined above. Thus, essentially, Datalog is a subset of Horn logic where structured terms (i.e., recursive data structures) are forbidden. This is a direct consequence of the lack of function symbols.

Similar to Horn logic, Datalog’s knowledge bases consist of sets of function-free Horn clauses.

Description logics. Description logics (DL) are a family of subsets of FOL, generally involving some or no quantifiers, no structured terms, and no n-ary predicates such that \(n \ge 3\) . In other words, description logics represent knowledge by only leveraging on constants and variables other than atomic, unary, and binary predicates.

Differences among specific variants of DL lay in which and how many logic connectives are supported other than, of course, whether negation is supported or not. The wide variety of DL is due to the well-known expressiveness/tractability trade-off. However, depending on the particular situation at hand, one may either prefer a more expressive ( \(\approx\) feature-rich) DL variant at the price of a reduced tractability (or even decidability) of the algorithms aimed at manipulating knowledge represented through that DL or vice versa.

Regardless of the particular DL variant of choice, it is common practice in the scope of DL to call (i) constant terms ‘individuals’ as each constant references a single entity from a given domain, (ii) unary predicates, e.g., either ‘classes’ or ‘concepts’ as each predicate groups a set of individuals, i.e., all those individuals for which the predicate is true, and (iii) binary predicates, e.g., either ‘properties’ or ‘roles’ as each predicate relates two sets of individuals. Following such a nomenclature, any piece of knowledge can be represented in DL by tagging each relevant entity with some constant (e.g., a URL) and by defining concepts and properties accordingly.

Notably, binary predicates are of particular interest as they support connecting couples of entities altogether. This is commonly achieved via subject–predicate–object triplets, i.e., ground binary predicates of the form \(\langle \mathtt {a}\ \mathit {f}\ \mathtt {b} \rangle\) or \(\mathit {f}(\mathtt {a}, \mathtt {b})\) , where \(\mathtt {a}\) is the subject, \(\mathit {f}\) is the predicate, and \(\mathtt {b}\) is the object. Such triplets allow users to extensionally describe knowledge in a readable, machine-interpretable, and tractable way.

Collections of triplets constitute the so-called knowledge graphs(KGs), i.e., directed graphs where vertices represent individuals, while arcs represent the binary properties connecting these individuals. These may explicitly or implicitly instantiate a particular ontology, i.e., a formal description of classes characterising a given domain and description of their relations (inclusion, exclusion, intersection, equivalence, etc.) as well as the properties they must (or must not) include.

Propositional logic. Propositional logic is a very restricted subset of FOL, where quantifiers, terms, and non-atomic predicates are missing. Hence, propositional formulæ simply consist of expressions involving one or many 0-ary predicates—i.e., propositions—possibly interconnected by ordinary logic connectives. Here, each proposition may be interpreted as a Boolean variable that can either be true or false and the truth of formulæ can be computed as in the Boolean algebra. Thus, for instance, a notable example of a propositional formula could be as follows: \(p \wedge \lnot q \rightarrow r,\) where p may be the proposition ‘it is raining’, q may be the proposition ‘there is a roof’, and r may be the proposition ‘the floor is wet’.

The expressiveness of propositional logic is far lower than the one of FOL. For instance, because of the lack of quantifiers, each relevant aspect/event should be explicitly modelled as a proposition. Furthermore, because of the lack of terms, entities from a given domain cannot be explicitly referenced. Such a lack of expressiveness, however, implies that computing the satisfiability of a propositional formula is a decidable problem, which may be a desirable property in some application scenarios.

Despite the fact that propositional logic may appear too trivial to handle common decision tasks where non-binary data is involved, it turns out that a number of apparently complex situations can indeed be reduced to a propositional setting. This is the case, for instance, of any expression involving numeric variables or constants, arithmetical comparison operators, logic connectives, and nothing more than that. In fact, formulæ containing comparisons among variables or constants (or among each others) can be reduced to propositional logic by mapping each comparison into a proposition.

According to the main impact surveys in the XAI area [6, 12, 24], two major approaches exist to bring explainability or interpretability features to intelligent systems: by design or post-hoc.

XAI by design. This approach to XAI aims at making intelligent systems interpretable or explainable ex-ante since they are designed to keep these features as first-class goals. Methods adhering to this approach can be further classified according to two sub-categories.

In the remainder of this article, we focus on methods from the latter category as it is deeply entangled with symbolic knowledge injection.

Post-hoc explainability. This approach to XAI aims at making intelligent systems interpretable or explainable ex-post, i.e., by somehow manipulating poorly interpretable pre-existing systems. Methods adhering to this approach can be further classified according to the following sub-categories.

In the remainder of this article, we focus on methods from the ‘model simplification’ category, as it is deeply entangled with symbolic knowledge extraction.

Generally speaking, SKE serves the purpose of generating intelligible representations for the sub-symbolic knowledge that an ML predictor has grasped from data during learning. Here, we provide a general definition of SKE and discuss its purpose as well as the major benefits it brings against the XAI landscape.

Definition. We define SKE as

This definition emphasises a number of key aspects of SKE that are worth describing in further detail.

First, SKE is modelled as a class of algorithms—hence, finite-step recipes—characterised by what they accept as input and what they produce as output.

As far as the inputs of SKE procedures are concerned, the only explicit requirement is on trained ML predictors. There is no constraint with regard to the nature of the predictor itself. Hence, SKE procedures may be designed for any possible predictor family in principle. Yet, this requirement implies that the predictor’s training has already occurred and has reached some satisfying performance with regard to the task it has been trained for. Hence, in an ML workflow, SKE should occur after training and validation are concluded.

As far as the outputs of SKE procedures are concerned, the only explicit requirement is about the production of symbolic knowledge. ‘Symbolic’ is intended here, in a broader sense, as a synonym of ‘intelligible’ (for the human being). Hence, admissible outcomes are logic formulæ as well as decision trees or bare human-readable text.

In any case, for an algorithm to be considered a valid SKE procedure, the output knowledge should mirror as much as possible the behaviour of the original predictor with regard to the domain it was trained for. This involves a fidelity score aimed at measuring how well the extracted knowledge mimics the predictor it was extracted by with regard to the domain and the task that predictor was trained for. This, in turn, implies that the extracted knowledge should act in principle as a predictor as well, thus being as queryable as the original predictor would be. Thus, for instance, if the original predictor is an image classifier, the extracted knowledge should let an intelligent agent classify images of the same sort, expecting the same result. The agent may then be either computational (i.e., a software program) or human depending on whether the extracted knowledge is machine- or human-interpretable. The exploitation of logic knowledge as the target of SKE is of particular interest as it would enable both options.

Purpose and benefits. Generally speaking, one may be interested in performing SKE to inspect the inner operation of an opaque predictor, which should be considered a black box otherwise. However, one may also perform SKE to automatise and speed up the process of acquiring symbolic knowledge instead of crafting knowledge bases manually.

Inspecting a black-box predictor through SKE, in turn, is an interesting capability within the scope of XAI. Given a black-box predictor and a knowledge-extraction procedure applicable to it, any extracted knowledge can be adopted as a basis to construct explanations for that particular predictor. The extracted knowledge may act as an interpretable replacement (i.e., surrogate model) for the original predictor, provided that the two have a high-fidelity score [15].

Accordingly, the application of SKE to XAI brings a number of relevant opportunities, e.g., by letting human users (i) study the internal operation of an opaque predictor to find mispredicted input patterns or correctly predicted input patterns leveraging some unethical decision process; (ii) highlight the differences or the common behaviours between two or more black-box predictors performing the same task; and (iii) merge the knowledge acquired by various predictors, possibly of different kinds, on the same domain provided that the same representation format is used for extraction procedures [14].

Generally speaking, SKI serves a dual purpose with regard to SKE. SKI aims at letting an ML predictor take some symbolic knowledge into account when drawing predictions. Here, we provide a general definition of SKI and discuss its purpose and the major benefits it brings with regard to the XAI panorama.

Definition. We define SKI as

This definition emphasises a number of key aspects of SKI that are worth describing in further detail. Similar to SKE, it is modelled as a class of algorithms. Yet, dually with regard to extraction, SKI algorithms are procedures accepting symbolic knowledge as input and producing ML predictors as output.

In terms of the inputs of SKI procedures, the only explicit requirement is that knowledge should be symbolic and user provided—hence, human interpretable. However, since any input knowledge should be algorithmically manipulated by the SKI procedure, we elicit an implicit requirement here, constraining the input knowledge to be machine interpretable as well. This implies that some formal language—e.g., some formal logic or decision tree—should be employed for knowledge representation, whereas free text or natural language should be avoided.

Along this line, another implicit requirement is that the input knowledge should be functionally analogous with regard to the predictors undergoing injection. In other words, if a predictor aims at classifying customer profiles as either worthy or unworthy for credit, then the symbolic knowledge should encode decision procedures to serve the exact same purpose and observe the exact same information.

In terms of the outcomes of SKI procedures, our definition identifies two relevant situations that are not necessarily mutually exclusive. On the one hand, SKI procedures may enable sub-symbolic predictors to accept symbolic knowledge as input. SKI procedures of this sort essentially consist of a pre-processing algorithm aimed at encoding symbolic knowledge in sub-symbolic form, enabling sub-symbolic predictors to accept them as input. In this sense, SKI procedures of this sort enable sub-symbolic predictors to (learn how to) compute predictions as functions of the symbolic knowledge they were fed with assuming that it has been conveniently converted into sub-symbolic form. On the other hand, SKI procedures may alter sub-symbolic predictors so that they draw predictions that are consistent with the symbolic knowledge according to some notion of consistency. SKI procedures of this sort essentially affect either the structure or the training process of the sub-symbolic predictors they are applied to in such a way that the predictor must then take the symbolic knowledge into account when drawing predictions. In this sense, SKI procedures of this sort force sub-symbolic predictors to learn not only from data but from symbolic knowledge as well.

In any case, regardless of their outcomes, SKI procedures fit the ML workflow in its early phases, as they may affect both pre-processing and training.

Notably, consistency plays a pivotal role in SKI, dually with regard to what fidelity does for SKE. Along this line, our definition involves a consistency score aimed at measuring how well the predictor undergoing injection can take advantage of the injected knowledge with regard to the domain and the task that the predictor was trained for. Thus, for instance, if a knowledge base states that loans should be guaranteed to people from a given minority as long as annual income overcomes a given threshold, then any predictor undergoing injection of that knowledge base should output predictions respecting that statement or at least minimise violations with regard to it.

Purpose and benefits. One may be interested in performing SKI to reach a higher degree of control on what a sub-symbolic predictor is learning. In fact, SKI may either incentivise the predictor to learn some desirable behaviour or discourage it from learning some undesired behaviour. However, one may also exploit SKI to perform sub-symbolic or fuzzy manipulations of symbolic knowledge that would be otherwise unfeasible or hard to formalise via crisp symbols. While the latter option is further analysed by a number of authors (e.g.,[1, 30]), in the remainder of this section we focus on the former use case as it is better suited to serve the purposes of XAI.

Within the scope of XAI, SKI is a remarkable capability as it provides a workaround for the issues arising from the opacity of ML predictors. While SKE aims at reducing the opacity of a predictor by letting users understand its behaviour, SKI aims at bypassing the need for transparency. Indeed, predictors undergoing the injection of trusted symbolic knowledge provide higher guarantees about their behaviour, which will be more predictable and comprehensible.

Accordingly, the application of SKI to XAI brings a number of relevant opportunities, e.g., by letting human designers (i) endow sub-symbolic predictors with their common sense and, therefore, (ii) allowing them to finely control what predictors are learning, in particular, (iii) letting predictors learn about relevant situations despite poor data being available to describe them. Provided that adequate SKI procedures exist, all such use cases come at the price of handcrafting ad hoc knowledge bases reifying the designers’ common sense in symbols and then injecting it into ordinary ML predictors.

Answers for questions RQ1 and RQ5 are deeply entangled, as they are both related to SKE methods’ translucency. Translucency deals with the need for SKE methods to inspect the internal structure of the underlying black-box model while producing the extracted rules.

SKE methods provide for translucency in two ways [4] and can be labelled accordingly as

Along this line, we observe that surveyed SKE methods can be grouped into as many big clusters depending on how they treat the predictor undergoing extraction.

With regard to RQ1, it is worth highlighting that pedagogical methods can be applied to any sort of supervised ML predictor, in principle despite the fact that the literature may only report particular cases of application to specific predictors. Conversely, each decompositional method focuses on a specific sort of supervised ML predictor. Hence, decompositional SKE methods can be further categorised with regard to which sort of supervised ML predictors they are tailored to. As detailed in Figure 2, the translucency is far from uniform for SKE methods. Indeed, nearly half of the surveyed methods are pedagogical, whereas the rest are tailored to feed-forward NNs (possibly with fixed amounts of layers), SVM, linear classifiers, or decision tree ensembles.

With regard to RQ5, it is worth highlighting that pedagogical methods treat the underlying predictor as an oracle to be queried for predictions the symbolic knowledge shall emulate. Conversely, decompositional methods must look into the internal structure of predictors, hoping to detect meaningful patterns. For instance, SKE methods focusing on NNs may try to interpret inner neurons as meaningful expressions combining their ingoing synapses.

This question can be answered by looking at the accepted input data type of the surveyed SKE methods. In most cases, data is structured, i.e., it consists of tables of numberswith three different types of features:

Alternatively, data may consist of the following.

In Figure 3, we report absolute occurrence of the types of input features accepted by the surveyed SKE methods, as described by their authors.

As the reader may notice, the vast majority of surveyed methods are tailored to structured data with continuous and/or discrete features.

Broadly speaking, any extracted SK should mirror (i.e., mimic) the operation of the ML predictor it has been extracted from. For supervised ML, this means that the extracted knowledge should express a function, mapping input features into output features (e.g., classes for classification tasks). Functions can be represented in symbols in several ways. Indeed, the SK extracted by the surveyed methods comes in various forms.

These forms can be categorised under both syntactic or semantic perspective. Here, syntax refers to the shape of the extracted SK, whereas semantic refers to what kind of logic formalism the extracted knowledge may leverage—which is a matter of expressiveness.

Shape of the extracted knowledge. As far as syntax is concerned, decision rules [19, 28, 40] and trees [9, 43] are the most widespread human-comprehensible formats for the output knowledge. Thus, the vast majority of surveyed methods adopt one of these. However, other solutions have been exploited as well—e.g., decision tables. In all cases, however, a common trait is that functions of real numbers are expressed by using symbols to denote the same input and output features the underlying ML predictor was trained on.

With regard to surveyed SKE methods, we identify four major admissible shapes:

Figure 4 sums up the occurrence of the different shapes of output rules required for SKE algorithms. As the reader may notice, the majority of the surveyed methods target rule lists. Arguably, this trend may be motivated by the great simplicity of rule lists in terms of readability and their algorithmic tractability.

Expressiveness of the extracted knowledge. Despite the fact that the extracted knowledge may contain statements of different shapes (e.g., rules, trees, tables), the readability, conciseness, and tractability of the extracted rules heavily depend on what those statements can contain—which, in turn, dictates what can (or cannot) be expressed. Generally, statements may contain predicates or relations among the symbols representing input or output features. These may (or may not) contain logic connectives as well as arithmetic or logic comparators. SKE methods can be categorised with regard to which and how many ways of combining symbols are admissible within statements.

Along this line, we identify five major formats for statements in the surveyed SKE methods.

Figure 5 summarises the occurrence of the different SK formats produced by the surveyed SKE algorithms. As the reader may notice, the vast majority of surveyed SKE methods produce predicative rules, i.e., rules composed of several Boolean statements about individual input features possibly interconnected via logic connectives. Arguably, this trend may be motivated by the great tractability of propositional rules and by their simplicity. In fact, to construct propositional rules, SKE algorithms may follow a divide-et-impera approach by focusing on one single input feature at a time—hence, enabling the simplification of the extraction process itself.

ML methods are commonly exploited in AI to serve specific purposes, e.g., classification, regression, and clustering. Regardless of the particular means by which SKE is attained, extraction aids the human users willing to inspect how those methods work. However, the particular AI tasks that ML predictors have been designed for play a pivotal role in determining what outputs users may expect from those predictors. A similar argument holds for extraction procedures, as the extracted knowledge should reflect the inner behaviour of the original predictor. Along this line, it is interesting to categorise SKE methods with regard to the AI task they assume for the ML predictors they are applied to.

Figure 6 summarises the occurrence of tasks among the surveyed SKE methods. Notably, most of them can be applied uniquely to classifiers, whereas a small portion of them is explicitly designed for regressors. Only a few methods can handle both categories.

In general, we observe how the surveyed methods are tailored to either classification or regression tasks—when not both. In either case, surveyed methods focus on supervised ML tasks. To the best of our knowledge, currently, there are no SKE procedures tailored on unsupervised or reinforcement learning tasks.

Among the 132 surveyed methods for SKE, we found runnable software implementations for 27 (20.5%). Of these, 10 consist of reusable software libraries, whereas the others are just experimental code. Figure 7 summarises this situation. In the supplementary materials, we provide details about these implementations-including the algorithm that they implement and the link to the repository hosting the source code.

Generally speaking, SKI methods support the injection of knowledge expressed by various formalisms despite each surveyed method focusing on some particular formalism. A key discriminating factor is whether the chosen formalism is machine interpretable or not other than human interpretable.

With regard to the formalism the input knowledge should adopt to support SKI, we may cluster the surveyed methods into two major groups:

In Figure 8, we categorise the surveyed SKI methods with regard to their formalism of choice. Here, KGs are the most prominent cluster (including almost half of the surveyed methods), whereas model logic is the smallest. Methods tailored to FOL or its subsets (apart from KGs) form another relevant cluster. Among the FOL subsets, propositional logic plays a pivotal role, as it involves the relative majority of methods.

As long as the logic formalism is concerned, we consider and report the actual logic used in the papers. This is rarely explicitly stated by the authors in their papers. Thus, we deduce the actual logic used by each SKI method from the constraints that its logic is subject to according to its authors.

By analysing the surveyed SKI methods, we acknowledge great variety in the way that injection is performed. Arguably, however, such variety can be tackled by focusing on three major strategies, depicted in Figure 9 and summarised below:

Figure 10 summarises the frequency of these strategies among the surveyed SKI algorithms. Notably, the distribution of surveyed SKI methods among the three categories above is quite balanced.

Virtually all surveyed SKI methods are designed to inject knowledge into NNs. However, as this survey spans over 2 decades, the kinds of NNs supported by SKI methods are manifold despite the fact that each method is tailored to specific kinds of NNs.

Accordingly, surveyed SKI methods can be classified with regard to the particular kind of NN they support. As detailed in Figure 11, admissible choices along this line fit the many kinds of NN discussed in Section 2.1.2, as follows.

Notable exceptions are as follows.

The reason why the vast majority of methods rely on (some sort of) NN is straightforward: methods tailored to GNNs (resp., CNNs) assume the networks to accept specific kinds of data as input, e.g., graphs (resp., images), while ordinary feed-forward NNs accept raw vectors of real numbers.

Unlike SKE methods, which uniquely serve the purpose of inspecting black-box predictors by mimicking the way they address supervised learning tasks, SKI methods from the literature may serve multiple purposes. In Figure 12, we identify the following two major purposes that SKI methods may pursue by targeting symbolic or sub-symbolic AI tasks.

As the reader may note from the picture, surveyed SKI methods are quite balanced with regard to the categories above, with a slight preference for SK manipulation.

Among the 117 surveyed methods for SKI, we found runnable software implementations for 60 (51.3%). Of these, 11 consist of reusable software libraries, whereas the others are just experimental code. Figure 13 summarises this situation. In the supplementary materials, we provide details about these implementations, including the algorithm they implement and the link to the repository hosting the source code.

Debugging is a common activity for software programmers: it aims at spotting and fixing bugs in computer programs under production/maintenance. A bug is some unknown error contained in the program that leads to an unexpected or undesired observable behaviour of the computer(s) running that program. The whole procedure relies on the underlying assumption that computer programs are intelligible to the programmer debugging them and that the program can be precisely edited to fix the bug.

One may consider XAI techniques as means of debugging sub-symbolic predictors. In this metaphor, sub-symbolic predictors correspond to computer programs despite the fact that they are not manually written by programmers but rather learned from data, whereas data scientists correspond to programmers. However, debugging sub-symbolic predictors is hard because of their opacity, which makes their inner behaviour poorly intelligible for data scientists, and because they cannot be precisely edited after training and should be retrained from scratch instead. We discuss here the role of SKE and SKI in overcoming these issues, hence, allowing data scientists to debug sub-symbolic predictors.

Figure 15 provides an overview of how SKI and SKE fit the generic ML workflow. The figure stresses the relative position of both SKI and SKE with regard to the other phases of the ML workflow. Notably, SKI should occur before (or during) training, whereas SKE should occur after it. However, Figure 15 also stresses the addition of a loop into an otherwise linear workflow (right-hand side of the figure). We call it the ‘train–extract–fix–inject’ (TEFI) loop, which we argue is a possible way to debug sub-symbolic predictors.

In the TEFI loop, SKE is the basic mechanism by which the inner operation of a sub-symbolic predictor (i.e., ‘the program’ in the metaphor) is made intelligible to data scientists. The extracted knowledge may then be understood by data scientists and debugged—looking for pieces of knowledge that are wrong with regard to data scientist expectations. Then, symbolic knowledge may be precisely edited and fixed. SKI is the basic mechanism by which a trained predictor is precisely edited to adhere to the fixed symbolic knowledge.

Intelligent systems can be suitably modelled and described as composed of several intelligent, heterogeneous, and hybrid computational agents interoperating, and possibly communicating, among each other. Here, a computational agent is any software or robotic entity capable of computing other than perceiving and affecting some given environment—be it the Web, the physical world, or anything in between. To make the overall systems intelligent, these agents should be capable of a number of intelligent behaviours, ranging from image, speech, or text recognition to autonomous decision-making, planning, or deliberation. Behind the scenes, these agents may (also) leverage sub-symbolic predictors possibly trained on locally available data as well as symbolic reasoners, solvers, or planners to support these kinds of intelligent behaviours. Such agents are hybrid, meaning that they involve both symbolic and sub-symbolic AI facilities. However, interoperability may easily be a mirage because of (i) the wide variety of algorithms, libraries, and platforms supporting sub-symbolic ML other than (ii) the possibly different data items each agent may locally collect and later train predictors on. Each agent may learn (slightly) different behaviours due to the differences in the training data and in the actual ML workflow it adopts locally. When this is the case, exchange of behavioural knowledge may become cumbersome or infeasible.

In such scenarios, SKI and SKE may be enablers of a higher degree of interoperability, by supporting the exploitation of symbolic knowledge as the lingua franca for heterogeneous agents. Hybrid agents may exploit SKE to extract symbolic knowledge out of their local sub-symbolic predictors and exchange (and possibly improve) that symbolic knowledge with other agents. Then, any possible improvement of the symbolic knowledge attained via interaction may be back-propagated into local sub-symbolic predictors via SKI, enabling agents’ behaviour to improve as well.