Categorical Foundation of Explainable AI: A Unifying Theory

Abstract

Explainable AI (XAI) aims to address the human need for safe and reliable AI systems. However, numerous surveys emphasize the absence of a sound mathematical formalization of key XAI notions—remarkably including the term “explanation”, which still lacks a precise definition. To bridge this gap, this paper introduces a unifying mathematical framework allowing the rigorous definition of key XAI notions and processes, using the well-funded formalism of Category theory. In particular, we show that the introduced framework allows us to: (i) model existing learning schemes and architectures in both XAI and AI in general, (ii) formally define the term “explanation”, (iii) establish a theoretical basis for XAI taxonomies, and (iv) analyze commonly overlooked aspects of explaining methods. As a consequence, the proposed categorical framework represents a significant step towards a sound theoretical foundation of explainable AI by providing an unambiguous language to describe and model concepts, algorithms, and systems, thus also promoting research in XAI and collaboration between researchers from diverse fields, such as computer science, cognitive science, and abstract mathematics.

F. Giannini, S. Fioravanti and P. Barbiero—Equal contribution.

A Elements of Category Theory

1.1 A.1 Monoidal Categories

The process interpretation of monoidal categories [8, 27] sees morphisms in monoidal categories as modelling processes with multiple inputs and multiple outputs. Monoidal categories also provide an intuitive syntax for them through string diagrams [40]. The coherence theorem for monoidal categories [53] ensures that string diagrams are a sound and complete syntax for them and thus all coherence equations for monoidal categories correspond to continuous deformations of string diagrams. One of the main advantages of string diagrams is that they make reasoning with equational theories more intuitive.

Definition 1

([21]). A category $\textsf{C}$ is given by a class of objects $\textsf{C}^o$ and, for every two objects $X,Y \in \textsf{C}^o$, a set of morphisms $\hom (X,Y)$ with input type $X$ and output type $Y$. A morphism $f \in \hom (X,Y)$ is written $f :X \rightarrow Y$. For all morphisms $f :X \rightarrow Y$ and morphisms $g :Y \rightarrow Z$ there is a composite morphisms $f \mathbin {;}g :X \rightarrow Z$. For each object $X \in \textsf{C}^o$ there is an identity morphism $\mathbb {1}_{X} \in \hom (X,X)$, which represents the process that “does nothing” to the input and just returns it as it is. Composition needs to be associative, i.e. there is no ambiguity in writing $f \mathbin {;}g \mathbin {;}h$, and unital, i.e. $f \mathbin {;}\mathbb {1}_{Y} = f = \mathbb {1}_{X} \mathbin {;}f$.

Monoidal categories [53] are categories endowed with extra structure, a monoidal product and a monoidal unit, that allows morphisms to be composed in parallel. The monoidal product is a functor $\times :\textsf{C} \times \textsf{C} \rightarrow \textsf{C}$ that associates to two processes, $f_1 :X_1 \rightarrow Y_1$ and $f_2 :X_2 \rightarrow Y_2$, their parallel composition $f_1 \times f_2 :X_1 \times X_2 \rightarrow Y_1 \times Y_2$. The monoidal unit is an object $U \in \textsf{C}^o$, which represents the “absence of inputs or outputs” and needs to satisfy $X \times U \cong X \cong U \times X$, for each $X\in \textsf{C}^o$. For this reason, this object is often not drawn in string diagrams and a morphism $s :U \rightarrow Y$, or $t :X \rightarrow U$, is represented as a box with no inputs, or no outputs.

1.2 A.2 Cartesian and Symmetric Monoidal Categories

A symmetric monoidal structure on a category is required to satisfy some coherence conditions [53], which ensure that string diagrams are a sound and complete syntax for symmetric monoidal categories [40]. Like functors are mappings between categories that preserve their structure, symmetric monoidal functors are mappings between symmetric monoidal categories that preserve the structure and axioms of symmetric monoidal categories.

Some symmetric monoidal categories have additional structure that allows resources to be copied and discarded [25]. These are called Cartesian categories.

1.3 A.3 Feedback Monoidal Categories

Feedback monoidal functors are mappings between feedback monoidal categories that preserve the structure and axioms of feedback monoidal categories.

Feedback monoidal categories are the syntax for processes with feedback loops. When the monoidal structure of a feedback monoidal category is cartesian, we call it feedback cartesian category. Their semantics can be given by monoidal streams [15]. In cartesian categories, these have an explicit description. We refer to them as cartesian streams, but they have appeared in the literature multiple times under the name of “stateful morphism sequences” [76] and “causal stream functions” [84].

1.4 A.4 Free Categories

We generate “abstract” categories using the notion of free category [53]. Intuitively, a free category serves as a template for a class of categories (e.g., feedback monoidals). To generate a free category, we just need to specify a set of objects and morphisms generators. Then we can realize “concrete” instances of a free category $\textsf{F}$ using a functor from $\textsf{F}$ to another category $\textsf{C}$ that preserves the axioms of $\textsf{F}$. If such a functor exists then $\textsf{C}$ is of the same type of $\textsf{F}$ (e.g., the image of a free feedback monoidal category via a feedback functor is a feedback monoidal category).

1.5 A.5 Institutions

An institution I is constituted by:

• (i) a category $\textsf{Sign}_I$ whose objects are signatures (i.e. vocabularies of symbols);

• (ii) a functor $Sen: \textsf{Sign}_I \mapsto \textsf{Set}$ providing sets of well-formed expressions ($\varSigma$-sentences) for each signature $\varSigma \in \textsf{Sign}_I^o$;

• (iii) a functor $Mod: \textsf{Sign}_I^{op} \mapsto \textsf{Set}$ providing semantic interpretations, i.e. worlds.

Furthermore, Satisfaction is then a parametrized relation $\models _{\varSigma }$ between $Mod(\varSigma )$ and $Sen(\varSigma )$, such that for all signature morphism $\rho : \varSigma \mapsto \varSigma '$, $\varSigma '$-model $M'$, and any $\varSigma$-sentence e,

$$\begin{aligned} M' \models _{\varSigma } \rho (e) \text { iff } \rho (M') \models _{\varSigma } e \end{aligned}$$

where $\rho (e)$ abbreviates $Sen(\rho )(e)$ and $\rho (M')$ stands for $Mod(\rho )(e)$.