A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations

Symbolic representations. Symbolic representations [42, 124] are natural for humans and widely used in computers. In symbolic representations, each item or object is represented by a symbol. Here, by objects we refer to items of various nature and complexity, such as features, relations, physical objects, scenes, their classes, and so on. More complex symbolic representations can be composed from the simpler ones.

Symbolic representations naturally possess a combinatorial structure that allows producing indefinitely many propositions/symbolic expressions (see Section 2.1.2 below). This is achieved by composition using rules or programs as demonstrated by, e.g., Turing Machines. This process is, of course, limited by memory size that can, however, be expanded without changing the computational structure of a system. A vivid example of a symbolic representation/system is a natural language, where a plethora of words is composed from a small alphabet of letters. In turn, a finite number of words is used to compose an infinite number of sentences and so on.

Symbolic representations have all-or-none explicit similarity: the same symbols have maximal similarity, whereas different symbols have zero similarity and are, therefore, called dissimilar. To process symbolic structures, one needs to follow edges and/or match vertices of underlying graphs to, e.g., reveal the whole structure or calculate the similarity between composite symbolic structures. Therefore, symbolic models usually have problems with scaling because similarity search and reasoning in such models require complex and sequential operations, which quickly become intractable.

In the context of brain-like computations, the downside of the conventional implementation of symbolic computations is that they require reliable hardware [189], since any error in computation might result in a fatal fault. In general, it is also unclear how symbolic representations and computations with them could be implemented in a biological tissue, especially when taking into account its unreliable nature.

Connectionist representations: localist and distributed. In this article, connectionism is used as an umbrella term for approaches related to neural networks and brain-like computations. Two main types of connectionist representations are distinguished: localist and distributed [184, 186].

Localist representations are akin to symbolic representations in that for each object there exists a single corresponding element in the implementation of the representation. Examples of localist representations are a single neuron (node) or a single vector component.

There is some evidence that localist representations might be used in the brain (so-called “grandmother cells”) [143]. In order to link localist representations, connections between elements can be created, corresponding to pointers in symbolic representations. However, constructing compositional structures that include combinations of already represented objects requires the allocation of (a potentially infinite number of) new additional elements and connections, which is neurobiologically questionable. For example, representing “abc”, “abd”, “acd”, and so on requires introducing new elements for them, as well as connections to “a”, “b”, “c”, “d” and so on. Also, localist representations share symbolic representations’ drawbacks of lacking enough semantic basis, that may be considered as a lack of immediate explicit similarity between the representations. In other words, different neurons representing different objects are dissimilar and estimating object similarity requires additional processes.

Distributed representations⁴ were inspired by the idea of a “holographic” representation as an alternative to the localist representation [48, 140, 184, 186]. They are attributed to a connectionist approach based on modeling the representation of information in the brain as “distributed” over many neurons. In distributed representations, the state of a set of neurons (of finite size) is modeled as a vector where each vector component represents a state of the particular neuron.

Distributed representations are defined as a form of vector representations, where each object is represented by a subset of vector components, and each vector component can belong to representations of many objects. This concerns (fully distributed) representations of objects of various complexity, from elementary features or atomic objects to complex scenes/objects that are represented by (hierarchical) compositional structures.

In distributed representations, the state of individual components of the representation cannot be interpreted without knowing the states of other components. In other words, in distributed representations the semantics of individual components of the representation are usually undefined, in distinction to localist representations.

In order to be useful in practice for engineering applications and cognitive modeling, distributed representations of similar objects should be similar (according to some similarity measure of the corresponding vector representations; see Section 2.2.2), thus addressing the semantic basis issue of symbolic and localist representations.

Distributed representations should have the following attractive properties:

In our opinion, the last two properties listed above are closely connected to the challenges posed to distributed representations in the late 1980s and early 1990s that we briefly discuss in the next section.

Let us consider several challenges faced by early distributed representations known as “superposition catastrophe”, e.g., [153, 188]. And, at a higher level, by demand for “systematicity” [22], and much later, for fast compositionality [53]. These challenges led to the necessity to make distributed representations “structure-sensitive” by introducing the “binding” operation (Section 2.2.3).

“Superposition catastrophe”. It was believed that connectionist representations cannot represent hierarchical compositional structures because of the superposition catastrophe, which manifestates itself in losing the information concerning object arrangements in structures, see [146, 153, 188] for discussion and references.

In the simplest case, let us activate binary localist representation elements corresponding to “a” and “b”, then to “b” and “c”. If we want to represent both “a” and “b”, and “b” and “c” simultaneously, we activate all three representations: “a”, “b”, “c”. Now, however, the information that “a” was with “b”, and “b” was with “c” is lost. For example, the same “a”, “b”, “c” activation could be obtained by “a” and “c”, and single “b”. The same situation occurs if “a”, “b”, “c” are represented by distributed patterns.

Fodor and Pylyshyn criticisms of connectionism. In [22], criticism of connectionism was concerned with the parallel distributed processing approach covered in [48]. Fodor and Pylyshyn claimed that connectionism lacks Productivity, Systematicity, Compositionality, and Inferential Coherence that are inherent to systems operating with symbolic representations. Their definitions of these intuitively appealing issues are rather vague and interrelated, and their criticism is constrained to the early particular connectionist model that the authors chose for their critique. Therefore, we restate these challenges as formulated in [139]:

Challenges to connectionism posed by Jackendoff. Four challenges to connectionism have been posed by Jackendoff [53], see also [34] for their HDC/VSA treatment. In principle, they are relevant to cognition but in particular they are related to language. The problem, in general, is how to neurally instantiate the rapid construction and transformation of the compositional structures.

The challenges presented above made it clear that an adequate representation of compositional structures requires preserving information about their grouping and order. For example, in symbolic representations, brackets and symbolic order can be used to achieve this. For the same purpose, some mechanism of binding (à la “grouping brackets”) was needed in distributed representations.

We view the binding operation as a means to form such a representation of an object that contains information about the context in which it was encountered. So, any implementation of the binding operation is expected to involve some modification of the representation of the original object. The context can essentially be anything, such as other homogeneous objects (e.g., data objects of the same type) or heterogeneous objects (e.g., data objects’ positions, and roles). The binding operation, together with other operations, should provide a mechanism for constructing representations of compositional objects that reflect their similarity in a way that is useful for the problem being solved. For example, different combinations of the same objects should be represented differently. Also, it is often required to support a recovery (of the representation) of the original composite object.

One of the approaches to binding in distributed representations is based on the temporal synchronization of constituent activations [50, 111, 172, 188]. Although this mechanism may be useful on a single level of composition, its capabilities to represent compositional structures with multiple levels of hierarchy are questionable as it requires many time steps and complex “orchestration” to represent compositional structures. Another major problem is that such a temporal representation cannot be immediately stored in a long-term memory.

An alternative approach to binding, which eliminates these issues, is the so-called conjunctive coding approach used in HDC/VSA. Its predecessors were “extra units” considered by [46] to represent various combinations of active units of distributed patterns as well as outer products [117, 174], which, however, increased the dimensionality of representations (see details in Section 2.3.2). In HDC/VSA, the binding operation does not change the dimensionality of distributed representations. Moreover, it does not require any training.

To form “rich” compositional representations in HDC/VSA, both binding and superposition operations are used. Therefore, implementations of these operations need to maintain their properties when used together. In particular, the binding operation should not be associative with respect to the superposition operation in order to overcome the superposition catastrophe (Section 2.1.2).

Importantly, the schemes for forming distributed representations of compositional structures that exploit the binding and superposition operations produce distributed representations that are similar for similar objects (i.e., they take into account the similarity of object elements, their grouping, and order at various levels of hierarchy).

In summary, the main motivation for developing HDC/VSA was to combine the advantages of early distributed representations and those of symbolic representations, while avoiding their drawbacks, in pursuit for more efficient information processing and, ultimately, for better AI systems. One of the goals was to address the above challenges faced by conventional connectionist representations. The properties of HDC/VSA models introduced below allow addressing these challenges to a varying degree.

When designing an HDC/VSA-based system it is common to define a set of the most basic objects/items/entities/concepts/symbols/scalars for the given problem and assign them HVs, which are referred to as atomic HVs. The process of assigning atomic HVs is often referred to as mapping, projection, embedding, formation, or transformation. To be consistent, we will use the term transformation.

For a given problem, we need to choose atomic representations of objects such that the similarity between the representations of objects corresponds to the properties we care about.⁵ HVs of other (compositional) objects are formed by the atomic HVs (see Section 2.2.4). As follows from their name, atomic HVs are high-dimensional vectors. Values of HV components could be binary, real, or complex numbers and this list is not exhaustive, as we will see in Section 2.3.

In the early days of HDC/VSA, most of the works were focused on symbolic problems. In the case of working with symbols one could easily imagine many problems where a reasonable assumption would be that symbols are not related at all. So their atomic HVs are generated at random and are considered dissimilar, i.e., their expected similarity value is considered to be “zero”. On the other hand, there are many problems where assigning atomic HVs fully randomly does not lead to any useful behavior of the designed system, see Section 3.2.

As mentioned above, in HDC/VSA random independent HVs generated from some distribution are used for representing objects that are considered independent and dissimilar. For example, randomly generated binary HVs with components from the set \(\lbrace 0,1\rbrace\) , with the probability of a 1-component being \(p_1 = 1\) /2 for dense binary representations [66] or for sparse binary representations [146] with \(p_1 = 1\) /100 or with a fixed number of \(M\lt \lt D\) randomly activated components. Such random HVs are analogous to “symbols” in symbolic representations. However, the introduction of new symbols in symbolic representations or nodes in localist representations requires changing the dimensionality of representations, whereas in HDC/VSA they are simply introduced as new HVs of fixed dimensionality. So HDC/VSA can accommodate N symbols with D-dimensional HVs such that \(N\gt \gt D\) . This happens because in high-dimensional spaces randomly chosen HVs are quasi-orthogonal to each other. The number of exactly orthogonal dimensions in a vector space is equal to its dimensionality D, but the number of quasi-orthogonal directions is exponential to the dimensionality. Exact orthogonality means that the angle between vectors is exactly 90 degrees, whereas quasi-orthogonality means that the angle between vectors is within some small interval around 90 degrees. As the dimensionality increases, the number of such quasi-orthogonal vectors increases exponentially even for a very tiny interval around 90 degrees. So, as the dimensionality increases, the number of quasi-orthogonal vectors increases rapidly while the mathematical properties of the quasi-orthogonal vectors become closer to those of exactly orthogonal vectors. This mathematical phenomenon is known as concentration of measure [103]. The peculiar property of this phenomenon is that quasi-orthogonality converges to exact orthogonality with increased dimensionality of HVs. This is sometimes referred to as the “blessing of dimensionality” [37]. Figure 1 provides a visual illustration of the case of bipolar HVs.

If the similarity between objects is important, then HVs are generated in such a way that their similarity characterizes the desired similarity between objects. The similarity between HVs is measured by standard vector similarity (or distance) measures.

Similarity measure for dense representations. For dense HVs with real or integer components the most commonly used measures are the Euclidean distance:the dot (inner, scalar) product:and the cosine similarity:

Note that these measures are closely related. In fact, when vectors \(\mathbf {a}\) and \(\mathbf {b}\) are normalized to have unit norm, there is exact correspondence: \(\text{sim}_{\text{dot}} (\mathbf {a}, \mathbf {b}) = \text{sim}_{\text{cos}} (\mathbf {a}, \mathbf {b}) = 1- \text{dist}_{\text{Euc}}(\mathbf {a}, \mathbf {b})^2/2\) (as follows from \(||\mathbf {a}-\mathbf {b}||_2^2 = ||\mathbf {a}||_2^2 + ||\mathbf {b}||_2^2 - 2\mathbf {a}^{\top } \mathbf {b}\) ). For dense HVs with binary components, the normalized Hamming distance is the most common choice:where \(\oplus\) denotes a component-wise XOR operation. If the binary values of \(\mathbf {a}\) and \(\mathbf {b}\) are mapped to bipolar values by replacing 0-components to \(-1\) s, the Hamming distance is connected to the dot product as \(\text{sim}_{\text{dot}} (\mathbf {2a-1}, \mathbf {2b-1}) = D(1-2\text{dist}_{\text{Ham}} (\mathbf {a}, \mathbf {b}))\) .

Similarity measures for sparse representations. \(\text{sim}_{\text{dot}}\) and \(\text{sim}_{\text{cos}}\) are frequently used as similarity measures in the case of sparse HVs, i.e., when the number of non-zero components in HV is small.

Sometimes, Jaccard similarity is also used for this purpose:where \(\wedge\) and \(\vee\) denote component-wise AND and OR operations, respectively.

Superposition and binding operations that do not change the HV dimensionality are of particular interest because they allow one to apply the same operations to their resultant HVs. But note that some applications might require changing the dimensionality of representations. Obviously, in biological systems one would be very surprised if dimensionalities were exactly equal. Assuming that the dimensionality is unchanged is a mathematical convenience for, e.g., using component-wise vector operations—rather than a requirement.

Superposition. The superposition is the most basic operation used to form an HV that represents several HVs. In analogy to simultaneous activation of neural patterns represented by HVs, this is usually modeled as a disjunction of binary HVs or addition of real-valued HVs. See more examples of particular implementations in Section 2.3. In HDC/VSA, this operation is known under several names: bundling, superposition, and addition. Due to its intuitive meaning, below, we will use the term superposition. The simplest unconstrained superposition (denoted as \({\bf s}\) ) is simply:

In order to be used at later stages, the resultant HV is often required to preserve certain properties of atomic HVs. Therefore, some kind of normalization is often used to preserve, for instance, the norm, e.g., Euclidean:or the type of components as in atomic HVs, e.g., integers in a limited range:where \(f()\) is a function limiting the range of values of \({\bf s}\) . For example, in the case of dense binary/bipolar HVs, the binarization of components via majority rule/sign function is used. In the case of sparse binary HVs, their superposition increases the density of the resultant HV. There are operations to “thin” the resultant HV and preserve the sparsity (see Section 2.3.8). Below, we will also use brackets \(\langle \cdot \rangle\) when some sort of normalization is applied.

After superposition, the resultant HV is similar to its input HVs. Moreover, the superposition of HVs remains similar to each individual HV but the similarity decreases as more HVs are superimposed together. This could be seen as an unstructured and additive kind of similarity preservation. A fundamental property of the superposition is that it is associative, meaning that the sum is independent of the order of its components. However, [166] proposed a non-associative implementation of the superposition operation, which allows building a filter in the temporal (sequential) as well as in the elements’ domains. Note that the properties above are desiderata for the superposition and, thus, any operation that has these properties shall be seen as an implementation of the superposition operation.

Binding. As mentioned above, the superposition operation alone leads to the superposition catastrophe (due to its associative property), i.e., during the recursive application of the superposition, the information about combinations of the initial objects (e.g., grouping) is lost since, e.g., \((\mathbf {a} + \mathbf {b}) + (\mathbf {c} + \mathbf {d}) = (\mathbf {a} + \mathbf {c}) + (\mathbf {b} + \mathbf {d}) = \dots = \mathbf {a} + \mathbf {b}+ \mathbf {c} + \mathbf {d}\) .

This issue needs to be addressed in order to represent compositional structures (see Section 2.1.3). A binding operation (denoted as \(\circ\) when no particular implementation is specified) is used in HDC/VSA as a solution. It is convenient to consider two types of binding: via multiplication-like operation and via permutation operation. These two types are selected because they are utilized by most applications. Let us first briefly discuss them before we consider concrete implementations of the binding operation by different HDC/VSA models in Section 2.3.

Multiplicative binding. Multiplication-like operations are used in different HDC/VSA models for implementing the binding operation when two or more HVs should be bound together. Concrete implementations of this type of binding are rather diverse and depend on a particular HDC/VSA model. Examples of multiplication-like binding operations include: conjunction for sparse binary HVs, component-wise multiplication for real-valued HVs, outer product for tensor product representations (TPR), and even matrix-vector multiplication when a role is represented by a matrix while a filler is represented by an HV. Please refer to Section 2.3 for concrete implementations for a particular HDC/VSA model. Figure 1 in [169] also presents the taxonomy of the most commonly used multiplicative bindings.

The HV obtained after binding together several input HVs depends on all of them. Similar input HVs produce similar bound HVs. However, the way the similarity is preserved after the binding is (generally) different from that of the superposition operation [138, 146]. We discuss this aspect in detail a few paragraphs further down.

In many situations, there is a need for an operation to reverse the result of the binding operation. This operation is called unbinding or release (denoted as \(\oslash\) ). It allows the recovery of a bound HV [38, 191], e.g.,:

Here, the binding of only two HVs is used for demonstration purposes. In general, many HVs can be bound to each other. As mentioned above, even matrix-vector multiplication can be used to implement binding. Note that when the unbinding operation is applied to a superposition of HVs, the result will not be exactly equal to the original bound HV. It will also contain crosstalk noise; therefore, the unbinding operation is commonly followed by a clean-up procedure (see Section 2.2.5) to obtain the original bound HV. In some models, the binding operation is self-inverse, which means that the unbinding operation is the same as the binding operation. We specify how the unbinding operations are realized in different models in Section 2.3.

Binding by permutation. Another type of HV binding is by permutation, which might correspond to, e.g., some position or some other role of an object. We denote the application of a permutation to an HV as \(\rho (\mathbf {a})\) ⁶; if the same permutation is applied i times as \(\rho ^i(\mathbf {a})\) and the corresponding inverse is then \(\rho ^{-i}(\mathbf {a})\) . Note that the permuted HV is dissimilar to the original one. However, the use of partial permutations [12, 92] allows preserving some similarity (see Section 3.4.1). For permutation binding, the unbinding is achieved by inverse permutation.

The permutation can be implemented via a “permutation vector” that associates each component’s index with its index after permutation or by a matrix multiplication with the corresponding permutation matrix having a single 1-component in each row and column. Implementation of the permutation via the multiplication by the permutation matrix highlights some connection to the multiplicative binding implemented by matrix-vector multiplication as in [31]. Though the structure of permutation matrices is very different from the dense random matrices used in [31] (see also Section 2.3.4). However, a typical implementation of the multiplicative binding will take two HVs as an input, while the permutation operation takes as input an HV as well as permutation’s identity. This difference in the type of input arguments explains why sometimes permutations are treated as the third type of basic HDC/VSA operations (in addition to superposition and binding).

Often in practice a special case of permutation—cyclic shift—is used as it is very simple to implement. The use of cyclic (as well as non-cyclic) shift for positional binding in HDC/VSA was originally proposed in [91] for representing 2D structures (images) and 1D structures (sequences). Permutations were also used for representing the order in [146]. In [33], the permutation operation was introduced as essential for representing compositional structures when the binding operation is self-inverse. In a similar spirit, a permutation was used in [55, 135] to avoid the commutativity of a multiplicative binding by circular convolution. In general, a permutation can be used to do so for all commutative operations in all HDC/VSA models (see, e.g., Section 3.6.7 in [134]).

Later [66, 168] introduced a primitive for representing a sequence in a compositional HV using multiple applications of the same fixed permutation to represent a position in the sequence. This primitive was popularized via applications to texts [56, 168].

Similarity preservation by operations. For the superposition operation, the “contribution” of any input HV on the resultant HV is the same independently of other input HVs. In binding, the “contribution” of any input HV on the result depends on all other input HVs.

Also, the superposition and binding operations preserve similarity in a different manner. There is unstructured similarity, which is the similarity of the resultant HV to the HVs used as input to the operation. The superposition operation preserves the similarity of the resultant HV to each of the superimposed HVs, that is, it preserves the unstructured similarity. For example, \(\mathbf {d} = \mathbf {a} + \mathbf {b}\) is similar to \(\mathbf {a}\) and \(\mathbf {b}\) . In fact, given some assumptions on the nature of the HVs being superimposed, it is possible to analytically analyze the information capacity of HVs (see Section 2.4 for details).

Most realizations of the binding operation do not preserve unstructured similarity. They do, however, preserve structured similarity, that is, the similarity of the bound HVs to each other. Let us consider two i.i.d. random HVs: \(\mathbf {a}\) and \(\mathbf {b}\) ; \(\mathbf {d} = \mathbf {a} \circ \mathbf {b}\) is similar to \(\mathbf {d}^{\prime } = \mathbf {a}^{\prime } \circ \mathbf {b}^{\prime }\) if \(\mathbf {a}\) is similar to \(\mathbf {a}^{\prime }\) and \(\mathbf {b}\) is similar to \(\mathbf {b}^{\prime }\) . Thus, most realizations of the binding operation preserve structured similarity in a multiplicative fashion. When \(\mathbf {a}\) and \(\mathbf {b}\) are independent, as well as \(\mathbf {a}^{\prime }\) and \(\mathbf {b}^{\prime }\) , then the similarity of \(\mathbf {d}\) to \(\mathbf {d}^{\prime }\) is equal to the product of the similarities of \(\mathbf {a}\) to \(\mathbf {a}^{\prime }\) and \(\mathbf {b}\) to \(\mathbf {b}^{\prime }\) . For instance, if \(\mathbf {a}=\mathbf {a}^{\prime }\) , the similarity of \(\mathbf {d}\) to \(\mathbf {d}^{\prime }\) will be equal to the similarity of \(\mathbf {b}\) to \(\mathbf {b}^{\prime }\) . If \(\mathbf {a}\) is not similar to \(\mathbf {a}^{\prime }\) , \(\mathbf {d}\) will have no similarity with \(\mathbf {d}^{\prime }\) irrespective of the similarity between \(\mathbf {b}\) and \(\mathbf {b}^{\prime }\) due to the multiplicative fashion of similarity preservation. The structured similarity is also preserved when using permutations since \(\rho (\mathbf {a})\) is similar to \(\rho (\mathbf {a}^{\prime })\) if \(\mathbf {a}\) is similar to \(\mathbf {a}^{\prime }\) . This type of similarity is different from the unstructured similarity of the superimposed HVs: \(\mathbf {d} = \mathbf {a} + \mathbf {b}\) will still be similar to \(\mathbf {e} = \mathbf {a} + \mathbf {c}\) even if \(\mathbf {b}\) is dissimilar to \(\mathbf {c}\) . Finally, it is worth noting that, in practice, it might be necessary to preserve both structured and unstructured similarities (e.g., for tasks involving a similarity search) when binding two objects. In that case, the HVs should be formed in such a way that both similarity types are taken into account. A simple example of such a representation is \(\mathbf {a} + \mathbf {b} + \mathbf {a} \circ \mathbf {b}\) that still preserves some similarity to, e.g., \(\mathbf {a} + \mathbf {c} + \mathbf {a} \circ \mathbf {c}\) .

As introduced in Section 2.1.1, compositional structures are formed from objects where the objects can be either atomic or compositional. Atomic objects are the most basic (irreducible) elements of a compositional structure. More complex compositional objects are composed from atomic ones as well as from simpler compositional objects. Such a construction may be considered as a part-whole hierarchy, where lower-level parts are recursively composed to form higher-level wholes (see, e.g., an example of a graph in Section 3.5.2).

In HDC/VSA, compositional structures are transformed into their HVs using HVs of their elements and the superposition and binding operations introduced above. As mentioned in 2.1.3, it is a common requirement that similar compositional structures be represented by similar HVs. Another requirement that was studied intensively in the early days of HDC/VSA (see Section 3.1.1 of Part II of the survey [84] and, e.g., [123, 134, 174]) is the ability to transform representations of compositional structures without requiring the recovery of the structures (as would be the case for the symbolic approach). Recovering the original representation from its compositional HV, which we describe in the next section, might be an additional requirement. In Section 3 below, we will review a number of approaches to the formation of atomic and compositional HVs.

Given a compositional HV, it is often desirable to find the particular input HVs from which it was constructed. We will refer to the procedure implementing this as recovery. This procedure is also known as decoding, reconstruction, restoration, decomposition, parsing, and retrieval.

For recovery, it is necessary to know the set of the input HVs from (some of) which the HV was formed. This set is usually called the dictionary, codebook, clean-up memory, or item memory.⁷ In its simplest form, the item memory is just a matrix storing HVs explicitly, but it can be implemented as, e.g., a content-addressable associative memory [180]. Note that the stored HVs do not have to be atomic, they could also be compositional representations. For example, recent work [178] studied how to store compositional HVs in content-addressable associative memories à la Hopfield network. Moreover, recent works suggested [18, 79, 170] that it is not always necessary to store the item memory explicitly, as it can be easily rematerialized. Also, the recovery requires knowledge about the structure of the compositional HV, that is, information about the operations used to form a given HV (see an example below).

Most of the HDC/VSA models produce compositional HVs not similar to the input HVs due to the properties of their binding operations. So, to recover the HVs involved in the compositional HV, the use of the unbinding operation (Section 2.2.3 above) will typically be required. If the superposition operation was used when forming the compositional HV, after unbinding the obtained HVs will be noisy versions of the input HVs to be recovered (noiseless version can be obtained in the case when only binding operations were used to form a compositional HV). Therefore, a “clean-up” procedure is used as part of the recovery. In its most general form, the clean-up procedure can be viewed as projecting a query HV onto the subspace spanned by the item memory and returning, e.g., the weighted superposition (see examples in [24, 74]). In practice, however, the most common way of using the clean-up procedure is by finding the HV in the item memory that is most similar to the noisy query HV. The item memory performs the nearest neighbor search (i.e., returns an HV corresponding to the winner-take-all activation) that can be implemented by, e.g., comparing the query HV to each of the item memory’s HVs or by some kind of search in an associative memory implementing the item memory. For HVs representing limited size compositional structures, there are guarantees for the exact recovery [183].

Let us consider a simple example. The compositional HV was obtained as follows: \(\mathbf {s} = \mathbf {a} \circ \mathbf {b} + \mathbf {c} \circ \mathbf {d}\) . For recovery from \(\mathbf {s}\) , we know that it was formed as the superposition of pair-wise bindings. In the simplest setup, we also know that the input HVs include \(\mathbf {a}\) , \(\mathbf {b}\) , \(\mathbf {c}\) , \(\mathbf {d}\) . The task is to find which HVs were in those pairs. Then, to find out which HV was bound with, e.g., \(\mathbf {a}\) we unbind it with \(\mathbf {s}\) as \(\mathbf {a} \oslash \mathbf {s}\) resulting in \(\mathbf {b} + \mathbf {a} \oslash (\mathbf {c} \circ \mathbf {d}) = \mathbf {b} + \mathrm{noise} \approx \mathbf {b}\) . Then, we use the clean-up procedure that returns \(\mathbf {b}\) as the closest match (using the corresponding similarity measure). That way we know that \(\mathbf {a}\) was bound with \(\mathbf {b}\) . In this setup, we immediately know that \(\mathbf {c}\) was bound with \(\mathbf {d}\) . We can check this in the same manner, by first calculating, e.g., \(\mathbf {d} \oslash \mathbf {s}\) , and then cleaning it up. Note that \(\mathrm{noise}\) is defined as whatever is quasi-orthogonal to the signals of interest, and this definition is operationalized by populating the item memory with the signals of interest. In other words, if the noise term in the above example ( \(\mathbf {a} \oslash (\mathbf {c} \circ \mathbf {d})\) ) was an HV in the item memory, then the system would not treat it as noise.

In a more complicated setup, it is not known that the input HVs were \(\mathbf {a}\) , \(\mathbf {b}\) , \(\mathbf {c}\) , \(\mathbf {d}\) , but the HVs in the item memory are known. So, we take those HVs, one by one, and repeat the operations above. A possible final step in the recovery procedure is to recalculate the compositional HV from the reconstructed HVs. The recalculated compositional HV should match the original compositional HV. Note that the recovery procedure becomes much more complex in the case where m HVs are bound instead of just pairs. It is easy to recover one of the HVs used in the binding, if the other \(m-1\) HVs are known. They can simply be unbound from the compositional HV.

When the other HVs used in the binding are not known, the simplest way to find the HVs used for binding is to compute the similarity with all possible bindings of HVs from the item memory. The complexity of this search grows exponentially with m. However, there is a recent work [24, 74] that proposed a mechanism called resonator network to address this problem.

Before exposing a reader to the details of each HDC/VSA model, it is important to make a comment on the nature of their diversity and enable an intuition behind selecting the best model for the practical usage.

The current diversity of the HDC/VSA models is a result of the evolutionary development of the main vector symbolic paradigm by independent research groups and individual researchers. Initially, the diversity comes from different initial assumptions, variations in the neurobiological inspiration and the particular mathematical background of the originators. Therefore, from a historical perspective, the question of selecting a candidate for the best model is ill-posed. Recent work [169] started to perform a systematic experimental comparison between models. We emphasize the importance of further investigations in this direction in order to facilitate conscious choice of one or another model for a given problem and to raise HDC/VSA to the level of matured engineering discipline.

However, the usage of different models could already be prioritized at this moment using the target computing hardware perspective. The recent developments of unconventional computing hardware aim at improving the energy efficiency over the conventional von Neumann architecture in AI applications. Another driving factor is reliability. As conventional hardware for the von Neumann architecture gets smaller, it eventually becomes unreliable, so there is a need for computing frameworks that are robust to unreliable hardware and HDC/VSA is a promising candidate. The availability of such a robust computing framework also opens the door to other types of hardware implementation that are inherently unreliable and would not be suitable for a von Neumann architecture. Various unconventional hardware platforms (see Section 4.3) deliver great promises in moving the borders of energy efficiency and operational speed beyond the current standards.

Independently on the type of the computing hardware, any HDC/VSA model can be seen as an abstraction of the algorithmic layer and can thus be used for designing computational primitives, which can then be mapped to various hardware platforms using different models [78]. In the next subsections, we present the details of the currently known HDC/VSA models.

The TPR model is one of the earliest models within HDC/VSA. The TPR model was originally proposed by Smolensky in [174]. However, it is worth noting that similar ideas were also presented in [117, 118] around the same time but received much less attention from the research community. Atomic HVs are vectors selected uniformly at random from the Euclidean unit sphere \(S^{(D-1)}\) . The binding operation is an outer product of HVs. So, the dimensionality of bound HVs grows exponentially with their number (e.g., two bound HVs have the dimension \(D^2\) , three bound vectors have the dimension \(D^3\) , and so on). The vectors need not be of the same dimension. This points to another important note that the TPR model may or may not be categorized as an HDC/VSA model depending on whether the fixed dimensionality of representations is considered a compulsory attribute of an HDC/VSA model or not. Historically, HDC/VSA models have required the operator result HVs to be the same dimensionality as the argument HVs. This was for engineering feasibility and mathematical convenience. It is useful to think of HDC/VSA as TPR with the result projected into a lower dimensional space in a way that retains some useful properties of TPR. One way is to think of TPR as a predecessor of HDC/VSA.

The superposition is implemented by (tensor) addition. Since the dimensionality grows, a recursive application of the binding operation is challenging. The resultant tensor also depends on the order in which the HVs are presented. Binding similar HVs will result in similar tensors. The unbinding is realized by taking the tensor product representation of a compositional structure and extracting from it the HV of interest. For linearly independent HVs, the exact unbinding is done as the tensor multiplication by the unbinding HV(s). Unbinding HVs are obtained as the rows of inverse of the matrix with atomic HVs in columns. Approximate unbinding is done using an atomic HV instead of the unbinding HV. If the atomic HVs are orthonormal, this results in the exact unbinding.

Though the similarity measure was not specified, we assume it to be \(\text{sim}_{\text{cos}} \ \mbox{or}\ \text{dist}_{\text{Euc}}\) .

The HRR model was developed by Plate in the early 1990s [132, 134, 135].

The HRR model was inspired by Smolensky’s TPR model [174], Hinton’s “reduced descriptions” [47], and many other works, e.g., [10, 19, 108, 120, 171] to mention a few. Note that due to the usage of HRR in Semantic Pointer Architecture Unified Network [20], sometimes the HRR model is also referred to as the Semantic Pointer Architecture. The most detailed source of information on HRR is Plate’s book [139].

In HRR, the atomic HVs for representing dissimilar objects are real-valued and their components are independently generated from the normal distribution with mean 0 and variance 1/D. For large D, the Euclidean norm is close to 1.⁹ The binding operation is defined on two HVs ( \(\mathbf {a}\) and \(\mathbf {b}\) ) and implemented via the circular convolution, which projects the outer product back onto D-dimensional space:where \(z_j\) is the jth component of the resultant HV \(\mathbf {z}=\mathbf {a} \circ \mathbf {b}\) .

The circular convolution multiplies norms, and it approximately preserves the unit norms of input HVs. The bound HV is not similar to the input HVs. However, the bound HVs of similar input HVs are similar. The unbinding is done by the circular correlation of the bound HV with one of the input HVs. The result is noisy, so a clean-up procedure (see Section 2.2.5) is required. There is also a recent proposal in [38] for an alternative realization of the binding operation called Vector-derived Transformation Binding. It is claimed to better suit implementations in spiking neurons and was demonstrated to recover HVs of sequences (transformed with the approach from [165]) better than the circular convolution. Another proposal related to HRR is Square Matrix Representations (SMR) [73], which use random square matrices with unit norm and matrix multiplication as the binding operation (see Table 2 for summary). Finally, in [49] a general formulation of “quadratic binding” that subsumes, e.g., outer product-based binding and circular convolution-based binding was proposed. It can, for example, be used to adjust the dimensionality of the bound HV to be within D and \(D^2\) .

The superposition operation is component-wise addition. Often, the normalization is used after the addition to preserve the unit Euclidean norm of compositional HVs. In HRR, both superposition and binding operations are commutative. The similarity measure is \(\text{sim}_{\text{dot}}\) or \(\text{sim}_{\text{cos}}\) .

In the Matrix Binding of Additive Terms (MBAT) model [31] by Gallant and Okaywe, it was proposed to implement the binding operation by matrix-vector multiplication. Such an option was also briefly mentioned in [139]. Generally, it can change the dimensionality of the resultant HV, if needed.

Atomic HVs are dense and their components are randomly selected independently from \(\lbrace -1,+1\rbrace\) . HVs with real-valued components are also possible. The matrices are random, e.g., with elements also from \(\lbrace -1,+1\rbrace\) . Matrix-vector multiplication results can be binarized by thresholding.

The properties of this binding are similar to those of most other models (HRR, Multiply-Add-Permute, and Binary Spatter Codes). In particular, similar input HVs will result in similar bound HVs. The bound HVs are not similar to the input HVs (which is especially evident here, since even the HV’s dimensionality could change). The similarity measure is either \(\text{dist}_{\text{Euc}}\) or \(\text{sim}_{\text{dot}}\) . The superposition is a component-wise addition, which can be normalized. The unbinding could be done using the (pseudo) inverse of the role matrix (or the inverse for square matrices, guaranteed for orthogonal matrices, as in [185]).

In order to check if a particular HV is present in the sum of HVs that were bound by matrix multiplication, that HV should be multiplied by the matrix and the similarity of the resultant HV should be calculated and compared to a similarity threshold.

The Fourier Holographic Reduced Representations (FHRR) model [134, 139] was introduced by Plate as a model inspired by HRR where the HVs’ components are complex numbers. The atomic HVs are complex-valued random vectors where each vector component can be considered as a phase angle (phasor) randomly selected independently from the uniform distribution over \((0, 2\pi ]\) . Usually, the unit magnitude of each component is used (unitary HVs). The similarity measure is the mean of sum of cosines of angle differences. The binding operation is a component-wise complex multiplication, which is often referred to as Hadamard product. The unbinding is implemented via the binding with an HV conjugate (component-wise angle subtraction modulo \(2\pi\) ). The clean-up procedure is used similarly to HRR.

The superposition is a component-wise complex addition, which can be followed by the magnitude normalization such that all components have the unit magnitude.

The Binary Spatter Codes (BSC) model¹⁰ was developed in a series of papers by Kanerva in the mid 1990s [60, 61, 62, 63]. As noted in [137], the BSC model could be seen as a special case of the FHRR model where angles are restricted to 0 and \(\pi\) . In BSC, atomic HVs are dense binary HVs with components from \(\lbrace 0,1\rbrace\) . The superposition is a component-wise addition, thresholded (binarized) to obtain an approximately equal density of ones and zeros in the resultant HV. This operation is usually referred to as the majority rule or majority sum. It selects zero or one for each component depending on whether the number of zeros or ones in the summands is higher and ties are broken, e.g., at random. In order to have a deterministic implementation of the majority rule, it is common to assign a fixed random HV, which is included in the superposition when the number of summands is even. Another alternative is to use a deterministic variant of tie breaking as proposed in [41].

The binding is a component-wise exclusive OR (XOR), which is equivalent to taking the diagonal of the XOR outer product of the arguments. That is, it is another example of projecting the outer product onto D dimensions. The bound HV is not similar to the input HVs but bound HVs of two similar input HVs are similar. The unbinding operation is also XOR, meaning that BSC has a binding operation with a self-inverse binding property, which may be an important property for choosing an appropriate model for a specific application. The connection to the FHRR model also indicates that only phase angles with binary discrete phase values can support the self-inverse binding property. But will not support fractional exponentiation (see Section 3.2.1) because they do not have sufficient resolution of the phase angles. The clean-up procedure is used similar to the models above. The similarity measure is often \(\text{dist}_{\text{Ham}}\) .

The Multiply-Add-Permute (MAP) model was proposed by Gayler [33]. It has several variants (see [169]); the bipolar one is isomorphic to the BSC model. In the MAP model, atomic HVs are dense bipolar HVs with components from \(\lbrace -1,1\rbrace\) . This is used more often than the originally proposed version with the real-valued components from \([-1,1]\) .

The binding is component-wise multiplication, which is equivalent to taking the diagonal of the outer product of the arguments. The bound HV is not similar to the input HVs. However, bound HVs of two similar input HVs are similar. The unbinding is also component-wise multiplication and requires knowledge of the bound HV and one of the input HVs for the case of two bound input HVs.

The superposition is a component-wise addition. The result of the superposition can either be left as is, normalized using the Euclidean norm, or binarized to \(\lbrace -1,1\rbrace\) with the sign function where ties for 0-components should be broken randomly (but deterministically for a particular input HV set). The choice of normalization is use-case dependent. The similarity measure is either \(\text{sim}_{\text{dot}}\) or \(\text{sim}_{\text{cos}}\) .

The Sparse Binary Distributed Representations (SBDR) model [154], also known as Binary Sparse Distributed Codes [146, 153], proposed by Kussul, Rachkovskij, et al. emerged as a part of Associative-Projective Neural Networks [99] (see Section 3.2.2 in Part II of this survey [84]). There are two variants of the binding operation in the SBDR model: Conjunction and Context-Dependent Thinning (CDT). The idea of both variants is that they produce a resultant HV where the 1-components are a subset of the 1-components of the input HVs. This subset depends on all of the input HVs, so that (with the proper control of the sparsity) a particular combination of the input HVs results in a unique resultant HV, thus binding the input HVs and preserving the information on the input HVs used to obtain the resultant HV.

Conjunction-Disjunction. The Conjunction-Disjunction is one of the earliest HDC/VSA models. It uses binary vectors for atomic HVs [150] (the model was also proposed independently in [65]). Component-wise conjunction of HVs is used as the binding operation. Component-wise disjunction of HVs is used as the superposition operation. Both basic operations are defined for two or more HVs. Both operations produce HVs similar to the HVs from which they were obtained (unstructured similarity). That is, the bound HV is similar to the input HVs, unlike most of the other known binding operations, except for CDT (see below).

The HVs resulting from both basic operations are similar if their input HVs are similar (structured similarity). The operations are commutative so the resultant HV does not depend on the order in which the input HVs are presented. However, as noted in Section 2.2.3, this could easily be changed with the use of some random permutations to represent the order of arguments.

The probability of 1-components in the resultant HV can be controlled by the probability of 1-components in the input HVs. Sparse HVs allow using efficient algorithms such as inverted indexing or auto-associative memories for similarity search. Since conjunction reduces the density of the resultant HV compared to the density of the input HVs, it could be considered as another form of “reduced descriptions” from [47]. At the same time, the recursive application of such binding leads to HVs with all components set to zero. However, the superposition by disjunction increases the number of 1-components. These operations can therefore be used to compensate each other (up to a certain degree; see the next section). A modified version of the Conjunction-Disjunction model preserves the density of 1-components in the resultant HV as shown in the following subsubsection.

Context-Dependent Thinning. The CDT procedure proposed by Rachkovskij and Kussul [153] can be considered either as a binding operation or as a combination of superposition and binding. It was already used in the 1990s [98, 99, 100, 150] under the name “normalization” since it approximately maintains the desired density of 1-components in the resultant HV. This density, however, also determines the “degree” or “depth” of the binding as discussed in [153].

The CDT procedure is implemented as follows. First, m input binary HVs are superimposed by component-wise disjunction as:

This leads to an increased number of 1-components in the resultant HV and the input HVs are not bound yet. Second, in order to bind the input HVs, the resultant HV \(\mathbf {z}\) from the first stage is permuted and the permuted HV is conjuncted with the HV from the first stage. This binds the input HVs and reduces the density of 1-components. Third, T such conjunctive bindings can be obtained by using different random permutations or by recursive usage of a single permutation. They are superimposed by disjunction to produce the resultant HV. These steps are described by the following equation:where \(\rho _{k}\) denotes the kth random permutation. The density of the resultant HV depends on the number of permutation-disjunctions T. If many of them are used, we eventually get the HV from the first stage, i.e., with no binding of the input HVs. The described version of the CDT procedure is called “additive”. Several alternative versions of the CDT procedure were proposed in [153]. The CDT procedure preserves the information about HVs that were bound together by their remaining 1-components. The CDT procedure is commutative. It is defined for several input HVs. As well as Conjunction-Disjunction, the CDT procedure preserves both unstructured and structured similarity.

Due to the fact that both variants of binding in SBDR preserve unstructured similarity, the resultant HV is similar to the input HVs. This leads to \(\langle \mathbf {a}, \mathbf {b} \rangle\) being similar to \(\langle \mathbf {a}, \mathbf {c} \rangle\) , which is not the case for, e.g., \(\mathbf {a} \circ \mathbf {b}\) and \(\mathbf {a} \circ \mathbf {c}\) in most other implementations of multiplicative binding. Here, the preservation of the unstructured similarity allows recovering the input HVs from the resultant HV by using the similarity search in the item memory containing all possible input HVs. This does not require any unbinding operation. However, it should be noted that an analog of unbinding could be constructed for SBDR using repeated binding. This was not investigated in details (except for some preliminary results in [98, 144]).

Finally, it is important to take note of the character of the structured similarity preservation by the CDT procedure. It depends on the thinning depth T. For a large T, the result is close to the superposition of the input HVs, and so the character of the structured similarity is close to additive. The smaller T is, the closer the character of the structured similarity is to the multiplicative one.

The main motivation behind the Sparse Block Codes (SBC) model, proposed by Laiho et al. [28, 102], is to use sparse binary HVs as in SBDR but with a binding operation where the bound HV is not similar to the input HVs (in contrast to SBDR).

Similar to SBDR, atomic HVs in the SBC model are sparse and binary with components from \(\lbrace 0,1\rbrace\) . HVs, however, are imposed with an additional structure. An HV is partitioned into blocks of equal size (so that the HV’s dimensionality is a multiple of the block size). In each block, there is only one non-zero component, i.e., the activity in each block is maximally sparse as in Potts associative memory, see, e.g., [39].

In SBC, the binding operation is defined for two HVs and implemented via circular convolution, which is applied block-wise. The unbinding is done by the block-wise circular correlation of the bound HV with one of the input HVs. Note that with maximally sparse blocks, the binding operation is equivalent to the modulo sum of the corresponding indices [102], which demonstrates that each block is effectively a phase angle that is discretized to the block size. When the block is not maximally sparse, it functions more like a superposition of phase angles.

The superposition operation is a component-wise addition. If the compositional HV is going to be used for binding, it might be binarized by leaving only one of the components with the largest magnitude within each block active. If there are several components with the same largest magnitude then a deterministic choice (e.g., the component with the largest position number could be chosen) can be made. The similarity measure is \(\text{sim}_{\text{dot}}\) or \(\text{sim}_{\text{cos}}\) .

The Modular Composite Representations (MCR) model [175] proposed by Snaider and Franklin shares some similarities with FHRR, BSC, and SBS. Components of atomic HVs are integers drawn uniformly from some limited range (denoted as r), which is a parameter of the MCR model. Note that this model is also connected to the FHRR model since the components of HVs can be seen as phase angles that are discretized to r values.

The binding operation is defined as component-wise modular addition (the module value depends on the range limit), which generalizes XOR used for binding in BSC. Binding properties are similar to most of other models. The unbinding operation is the component-wise modular subtraction.

The similarity measure is a variation of the Manhattan distance [175]:

The superposition operation resembles that of FHRR. Integers are interpreted as discretized phase angles on a unit circle. First, phase angles are superimposed for each component (i.e., vector addition is performed). Second, the result of the superposition is normalized by setting the magnitude to one and the phase to the nearest phase corresponding to an integer from the defined range.

Finally, there is also a recent proposal of the Cyclic Group Representations (CGR) model [196] that highly resembles the idea of the MCR model. As it also defines a model that operates with integer-valued HVs but uses slightly different definitions for the similarity measure and the superposition operation.

As its name suggests, the Geometric Analogue of Holographic Reduced Representations (GAHRR) model [4] (proposed by Aerts, Czachor, and De Moor) was developed as a model alternative to HRR. An earlier proposal was a geometric analogue to BSC [3]. The main idea is to reformulate HRR in terms of geometric algebra. The binding operation is implemented via the geometric product (a generalization of the outer product). Because of not projecting the geometric product onto a lower dimensional space, GAHRR is conceptually closest to the TPR model. The superposition operation is component-wise addition.

So far, GAHRR is mainly an interesting theoretical effort, as its advantages over conceptually simpler models are not particularly clear, but it might become more relevant in the future. Readers interested in GAHRR are referred to the original publications. An introduction to the model is given in [2, 4, 127, 131], examples of representing data structures with GAHRR are in [129] and some experiments comparing GAHRR to other HDC/VSA models are presented in [128, 130].

As mentioned in Section 2.2.1, the simplest data type is a symbol. Usually, different symbols are considered dissimilar, therefore, using i.i.d. random HVs for different symbols is a standard way of transforming symbols into HVs. Such HVs correspond to symbolic representations (Section 2.1.1) since they behave like symbols in the sense that the similarity between the HV and its copy is maximal (i.e., the same symbol) while the similarity between two i.i.d. random HVs is minimal (i.e., different symbols).

In localist representations, sets are often represented as binary “characteristic vectors”, where each vector component corresponds to a particular symbol from the universe of all possible symbols. Symbols present in a particular set are represented by one in the corresponding vector component. In the case of multisets, the values of the components of the characteristic vector are the counters of the corresponding symbols.

In HDC/VSA, a set of symbols is usually represented by an HV formed by the superposition of the symbols’ HVs. The compositional HV preserves the similarity to the symbols’ HVs. Notice that Bloom filter [8]—a well-known data structure for approximate membership—is an HV where the superposition operation is obtained by the component-wise disjunction of binary HVs of symbols (as in SBDR) and, thus, can be seen as a special case of HDC/VSA [85].

Also note that, in principle, multiplicative bindings can be used to represent sets [80, 115], however, the similarity properties will be completely different from the representation by the superposition.

Role-filler bindings, which are also called slot-filler or key-value pairs, are a very general way of representing structured data records. For example, in localist representations the role (key) is the component’s ID, whereas the filler (value) is the component’s value. In HDC/VSA, this is represented by the result of binding. Both multiplicative binding and binding by permutation can be used. With multiplicative binding, both the role and the filler are transformed to HVs, which are bound together. When using permutation, it is common that the filler is represented by its HV, while the role is associated with either a unique permutation or a number of times that some fixed permutation should be applied to the filler’s HV.

In the case of multiplicative binding, the associations do not have to be limited to two HVs and, therefore, data structures involving more associations can be represented. For example, the representation of “schemas” [121] might involve binding three HVs corresponding to {context, action, and result}.¹² It is worth mentioning that in symbolic role-filler bindings roles are usually atomic symbols. In HDC/VSA, role HVs are not required to be representations of atomic objects. They may represent compositional structures and may be constructed to have a desired similarity between roles.

A set of role-filler bindings is represented by the superposition of their HVs. Note that the number of (role-filler) HVs superimposed is limited to preserve the information contained in them, e.g., if one wants to recover the HVs being superimposed (see Sections 2.2.5 and 2.4).

Representation of scalars in the compositional approach. In order to represent close values of a scalar by similar HVs, correlated HVs should be generated, so that the similarity decreases with the increasing difference of scalar values. Usually, a scalar is first normalized into some pre-specified range (e.g., \([0,1]\) ). Next, it is quantized into finite grades (levels). Since HVs have a finite dimension and are random, they can reliably (with non-negligible differences in similarity) represent only a finite number of scalar grades. So only a limited number of grades are represented by correlated HVs, usually up to several dozens.

Early schemes for representing scalars by HVs were proposed independently in [150, 173]. These and other schemes [142, 158] can be considered as implementing some kind of “flow” from one component set of an HV to another, with or without substitution. For example, “encoding by concatenation” [158] uses two random HVs: one is assigned to the minimum grade in the range, while the second one is used to represent the maximum grade. The HVs for intermediate grades are formed using some form of interpolation, such as the concatenation of the parts of the HVs where the lengths of these two parts are proportional to the distances from the current grade to the minimum and maximum one. Note that in this scheme the representations of all values in the range have some non-zero degree of similarity to each other (apart from the two extreme values). It is possible to apply the same scheme to multiple concatenated ranges, so that the values separated by more than the range size have zero similarity.

Similar types of interpolations were proposed in [13, 86, 191]. The scheme in [173] and the “subtractive-additive” one [158] start from a random HV and recursively flip the values of some components to obtain the HVs of the following grades. This scheme was popularized, e.g., in [86, 160], for representing values of features when applying HDC/VSA in classification tasks. Another early scheme proposed for complex and real-valued HVs is called fractional power encoding (see Section 5.6 in [134]). It is based on the fact that complex-valued HVs (random angles on a unit circle) can be (component-wise) exponentiated to any value:where \(\mathbf {z}(x)\) is an HV representing scalar x, \(\mathbf {z}\) is a random HV called the base HV [25, 26], and \(\beta\) is the bandwidth parameter controlling the width of the resultant similarity kernel. This approach requires neither normalizing scalars in a pre-specified range nor quantizing them. The fractional power encoding can be immediately applied to FHRR and to any other representation where the phase angle resolution is sufficient. It is also used for HRR by making complex-valued HV using fast Fourier transform, exponentiating, and then making inverse fast Fourier transform to return to real-valued HVs (see [25, 26] for the details).

Representation of numeric vectors in the compositional approach. In the compositional approach to numeric vector representation by HVs ( [150, 159, 177], we first form HVs for scalars (i.e., for components of a numeric vector). Then, HVs corresponding to values of different components are combined (often via superposition but the multiplicative binding is used too) to form the compositional HV of the numeric vector.

When constructing the compositional HV, it is important to make sure that scalar values of different components are represented by dissimilar HVs (even when two components have the same value). Otherwise, the association between a component and its value is lost. If scalars in different components are represented by dissimilar HVs, they can be superimposed or bound as is. Sometimes it is more practical (e.g., to reduce memory requirements) to keep a single set of correlated HVs representing all considered values of scalars.¹³ The set is shared across all components. In this case, prior to applying the superposition operation, one has to associate the scalar’s HV with its component identity in the numeric vector. This can be done by representing role-filler bindings (see Section 3.1.3) with some form of binding. This means that when using the role-filler binding scheme, the HV of a scalar corresponds to the filler while its component identity in the numeric vector corresponds to the role. As presented in Section 3.1.3, the association between role’s and filler’s HV can be realized either via the multiplicative binding with a role’s HV or by permuting a filler’s HV using the permutation assigned to the role.

Note that different schemes and models provide different types of similarity, for example, Manhattan distance ( \(\text{dist}_{\text{Man}}\) ) in [159, 183]. Note also that, unlike usual vectors, where the components are orthogonal, role HVs of nearby components could be made to correlate (e.g., nearby filtration banks are more correlated than banks located far away) [150].

The receptive field approach is often called coarse coding since a numeric vector is coarsely represented by the receptive fields activated by the vector. There are several well-known schemes utilizing explicit receptive fields, such as Cerebellar Model Articulation Controller [6], Prager Codes [141], and Random Subspace Codes (RSC) [94, 101], where receptive fields are various kinds of randomly placed and sized hyperrectangles, often in only some of the input space dimensions. It is worth noting that these schemes form (sparse) binary HVs. Similar approaches, however, can produce real-valued HVs, e.g., by using Radial Basis Functions as receptive fields [5, 119]. Details and comparisons can be found in [156, 157]. In particular, a similarity function between the input numeric vectors represented by RSCs was obtained.

RP is another approach which allows forming an HV of a numeric vector \(\mathbf {x}\) by multiplying it by an RP matrix \(\mathbf {R}\) :and possibly performing some normalization, binarization, and/or sparsification (such as preserving k largest vector components).¹⁴ It is a well-known approach, which has been extensively studied in mathematics and computer science [52, 54, 126, 151, 187]. Originally, the RP approach was used in the regime where the resultant vectors \(\mathbf {R} \mathbf {x}\) of smaller dimensionality were produced (i.e., used for dimensionality reduction). This is useful for, e.g., fast estimation of \(\text{dist}_{\text{Euc}}\) of original high-dimensional numeric vectors (as well as \(\text{sim}_{\text{dot}}\) and \(\text{dist}_{\text{angle}}\) ). Well-known applications are in similarity search [52, 72] and compressed sensing [16]. First, a random \(\mathbf {R}\) with components from the normal distribution was investigated [52, 54, 187], but then the same properties were proved for RP matrices with components from \(\lbrace -1,+1\rbrace\) (bipolar matrices) and \(\lbrace -1,0,+1\rbrace\) (ternary and sparse ternary matrices) [1, 58, 106].

In the context of HDC/VSA, the RP approach using a sparse ternary matrix was first applied in [69] (see also Section 2.2.1 of Part II of the survey [84] for details). In [113, 114] (see Section 2.2.1 of Part II of the survey [84]), it was proposed to binarize or ternarize (by thresholding) the result of \(\mathbf {R} \mathbf {x}\) , which produced sparse HVs. The analysis of a \(\text{sim}_{\text{cos}}\) estimation by sparse HVs was first reported in 2006/2007 and then published in [147, 148, 155], whereby the RP with sparse ternary and binary matrices was used. In [15], the RP with sparse binary matrices was used for expanding the dimensionality of the original vector. Note that increasing the dimensionality of the original vector has been widely used previously in machine learning classification tasks as it allows linearizing non-linear class boundaries. In [7], thresholding of the HV produced by \(\mathbf {R} \mathbf {x}\) was investigated and applied in the context of fast (i.e., sublinear versus the number of objects in the base) similarity search. It is common to use a single RP matrix to form an HV, but there are some approaches that rely on the use of several RP matrices [163]: \(\mathbf {z}= \sum _{i=1}^{n} \lambda _i \mathbf {R}_i \mathbf {x}\) , where \(\lambda _i\) determines the contribution of the ith RP \(\mathbf {R}_i \mathbf {x}\) to the resultant HV and, thus, can be interpreted as a weighted superposition of n separate representations.

An appealing feature of the RP approach is that it can be applied to input vectors of an arbitrary dimensionality and number of component gradations, unlike the compositional or the receptive fields approaches. Some theoretical analysis of using RP for forming HVs was performed in [183].

One of the standard ways of representing a sequence is to use a superposition of role-filler binding HVs where the filler is a symbol’s HV and the role is its position’s HV [41, 98, 136]. In [98], the order was represented by the weight of the subset of components selected from the initial HVs of sequence symbols (e.g., each preceding symbol was represented by more 1-components of the binary HV than the succeeding symbol). In [144], it was proposed to use thinning from the HVs of the previous sequence symbols (context). It is, however, most common to use i.i.d. random HVs to represent positions. An approach to avoid storing HVs for different positions is to use HDC/VSA models where the binding operation is not self-inverse. In this case, the absolute position k in a sequence is represented by binding the position HV to itself k times [133, 136]. Such a mechanism is called “trajectory association”. The trajectory association can be convenient when the sequence representation is formed sequentially, allowing the sequence HV to be formed in an incremental manner. However, the sequence HV does not preserve the similarity to symbols’ HVs in nearby positions, since the position HVs are not similar and so the role-filler binding HVs are also dissimilar. The MBAT model (Section 2.3.4) has similar properties, where the position binding is performed by multiplying by a random position matrix.

In order to make symbols in nearby positions similar to each other, one can use correlated position HVs [134, 176] such that, e.g., shifted sequences will still result in a high similarity between their HVs. This approach was also proposed independently in [13, 191] (see Section 3.2). Similar ideas for 2D images are discussed in Section 3.4.2.

Also, the HV of the next symbol can be bound to the HV of the previous one, or to several (weighted) HVs of previous ones, i.e., implementing the “binding with the context” approach [144].

An alternative way of representing position in a sequence is to use permutation [66, 91, 168]. Before combining the HVs of sequence symbols, the order i of each symbol is represented by applying some specific permutation to its HV i times (e.g., \(\rho ^2(\mathbf {c})\) ). However, such a permutation-based representation does not preserve the similarity of the same symbols in nearby positions, since the permutation does not preserve the similarity between the permuted HV and the original one. To preserve the similarity of HVs when using permutations, in [12, 92], it was proposed to use partial (correlated) permutations.

Once symbols of a sequence are associated with their positions, the last step is to combine the sequence symbols’ HVs into a single compositional HV representing the whole sequence. There are two common ways to combine these HVs.

The first way is to use the superposition operation, similar to the case of sets in Section 3.1.2. For example, for the sequence (a,b,c,d,e) the resultant HV (denoted as \(\mathbf {s}\) ) is:

Here, we exemplified the representation of positions via permutations but the composition will be the same for other approaches. The advantage of the approach with the superposition operation is that it is possible to estimate the similarity of two sequences by measuring the similarity of their HVs.

The second way of forming the compositional HV of a sequence involves binding of the permuted HVs, e.g., the sequence above is represented as (denoted as \(\mathbf {p}\) ):

The advantage of this sequence representation is that it allows forming quasi-orthogonal HVs even for sequences that differ in only one position. Similar to the trajectory association, extending a sequence can be done by permuting the current sequence’s HV and adding or multiplying it with the next HV in the sequence, hence, incurring a fixed computational cost per symbol.

Note that compositional HVs of sequences can also be of a hybrid nature. For example, when representing positions via permutations the similarity of the same symbols in nearby positions is not preserved. One way to preserve similarity when some of the symbols in a sequence are permuted was presented in [81], where compositional HVs of sequences included two parts: a representation of a multiset of symbols constituting a sequence (a bag-of-symbols) and an ordered representation of symbols. The first part is transformed into an HV as a multiset of symbols (Section 3.1.2) by superimposing atomic HVs of all symbols present in a sequence. The second part is transformed into an HV as a sequence using permutations of symbols’ HVs to encode their order (Section 3.3.2). Both representations are superimposed together to obtain the compositional HV.

In [149, 152], it was proposed to form such representations of sequences that preserve similarity of sequence elements in nearby positions as well as being equivariant with respect to sequence shifts. The property of equivariance means that the HV of the shifted sequence can be obtained not only by transforming this sequence into an HV, but just by some transformation of the HV of the original sequence. To achieve these properties of representations, [149] formed symbols’ HVs using the superposition of several appropriately permuted versions of the corresponding atomic HV. The same idea was also used in [152] but with recursive applications of multiplicative binding instead of permutations.

n-grams are n consecutive symbols of a sequence. In the n-gram representation of a sequence, all n-grams of the sequence are extracted, sometimes for different n. Usually, a vector containing n-gram statistics is formed such that its components correspond to different n-grams. The value of the component is the frequency (counter) of the occurrence of the corresponding n-gram.

There are various transformations of n-grams into HVs. The possibility of representing multisets with HVs (Section 3.1.2) allows forming HVs representing n-gram statistics as the superposition of HVs of individual n-grams. So the representations of n-grams in HVs are usually different in the following aspects:

To form dissimilar HVs for different n-grams, each n-gram can be assigned an i.i.d. random HV. However, an n-gram HV is usually generated from the HVs of its symbols. So in order to save space for the HV representations of n-grams, it is possible to represent them using compositional HVs [56, 145] instead of generating an atomic HV for each n-gram. Thus, the approaches above for representing sequences can be used. In [56, 76, 145], the permuted HVs of the n-gram symbols were bound by multiplicative binding. Alternatively, in [55, 57], the multiplicative binding of pairs of HVs was performed recursively, whereby the left HV corresponds to the already represented part of the n-gram, and the right HV corresponds to the next element of the n-gram. The left and right HVs are permuted by different random permutations.

In contrast to the approaches above, in [51, 164] the permuted HVs of the n-gram symbols were not bound multiplicatively, but were superimposed forming similar HVs for similar n-grams. In [41], the superposition of HVs corresponding to symbols in their positions was also used. However, the position of a symbol in a bi-gram (possibly from non-adjacent symbols) was specified by the multiplicative binding with the HVs of the left and right positions instead of permutations.

A stack can be seen as a special case of a sequence where elements are either inserted or removed in a last-in-first-out manner. At any given moment, only the top-most element of the stack can be accessed and elements written to the stack before are inaccessible until all later elements are removed. HDC/VSA-based representations of stacks were proposed in [134, 179, 193]. So, the HV of a stack is a special kind of sequence representation and the permutation of stack’s HV plays the role of moving the elements in the stack.

As discussed in Section 2.2.3, permutations are used commonly as a way of implementing binding with the position information. In the context of 2D images, permutations can be used by assigning one permutation for the x-axis and another one for the y-axis.

The simplest way of implementing a permutation is to use a cyclic shift as in [91, 96]. In order to implement two orthogonal shifts corresponding to the x- and y-axes, the HVs of features were restructured as a 3D tensor with two dimensions corresponding to the retina size along the x- and y-axes. Then the HV of a feature at some \((x,y)\) position was obtained by shifting the corresponding 3D tensor x times along the x-axis and y times along the y-axis.

In a similar way, when using permutations as shift generalizations, to bind an HV of a pixel’s value (or of an image feature) with its position, x-axis permutation is applied x times and y-axis permutation is applied y times [80, 115]. The image is represented as a compositional HV containing the superposition of all permuted HVs of pixels’ values. HVs representing pixels’ (features’) values are formed using some approach for representing scalars. As permuted HVs are not similar in any of the works above, the same pixel’s value even in nearby positions will be represented by dissimilar HVs.

The issue of the absence of similarity in permutation-based representations was addressed in [92, 97] by using partial permutations. The number of permuted components of some pixel’s value HV (or, more generally, of some feature’s HV) increases with the coordinate change (within some radius), up to the full permutation. So, inside the radius the similarity decreases. Outside the radius, the same process is repeated again. As a result, a pixel’s value HV at (x, y) is similar to the HVs in nearby positions within the similarity radius. The features at positions were Random Local Descriptors (RLD) detected at some points of interest. For instance, for binary images, the RLD feature is detected if some (varied for different features) pixels inside a local receptive field take zero and one values. Note that RLD may be considered as a version of receptive fields approach (Section 3.2.2) and can form HVs representing images, as in the case of the LInear Receptive Area (LIRA) features [93].

In a 2D image, it is natural to represent a pixel’s position and its value as a role-filler binding, where the pixel’s position HV is the role and an HV corresponding to the pixel’s value is the filler. Similar to the case of using permutations, the image HV is formed as a compositional HV containing the superposition of all role-filler binding HVs.

When it comes to pixels position HVs, the simplest approach is to treat each position as if it would be a unique symbol so that it can be assigned with a unique random HV. This approach allows forming image HVs through the role-filler bindings. It was investigated in [83, 110] for real-valued and dense binary HVs and in [82] for sparse binary HVs. The approach with unique HVs for roles, however, neither imposes any relations between pixels’ position HVs nor does it allow preserving the similarity of pixels’ position HVs in a local neighborhood, which, e.g., is useful for robustness against small changes in the 2D image. To address the latter issue, in [95] a method was described where the HVs of nearby x and y coordinates were represented as correlated HVs, using approaches for representing scalars (Section 3.2). Then the pixel value in a position (x, y) was represented by binding three HVs: one representing a pixel’s value at (x, y) and two representing the x and y coordinates themselves (component-wise conjunction was used for the binding in [95]). Thus, similar values in close proximity were represented by similar HVs.

MOving MBAT (MOMBAT) [30] was proposed to explicitly preserve the similarity of HVs for nearby pixels. As in the MBAT model (Section 2.3.4), matrices were used as roles. In particular, the matrix for x and y coordinates was formed as a weighted superposition of two random matrices. Thus, for both x and y, nearby coordinates were represented by similar (correlated) matrices. Then, each (x, y) pair was represented by a matrix obtained by binding (multiplying) those corresponding matrices. A value of a pixel at (x, y) was represented by binding its HV with the matrix of (x, y). Finally, the obtained HVs were superimposed. All the HVs participating in MOMBAT had real-valued components. A similar approach with local similarity preservation for BSC and dense binary HVs was considered in [11].

The Fractional Power Encoding approach (Section 3.2.1) to the representation of 2D images was employed in [23, 90, 190]. It imposes a structure between pixels’ positions and can preserve similarity in a local neighborhood. In particular, the image HV is formed as follows. Two random base HVs are assigned for the x-axis and y-axis, respectively. In order to represent x and y coordinates, the corresponding base HVs are exponentiated to coordinate values; and \(\mathbf {z}(x, y)\) —the HV for the (x, y) coordinate—is formed as their binding (assuming \(\beta =1\) cf. (15)):

Finally, the binding with the HV of the pixel value is done. The variant of the fractional power encoding-based representation in hexagonal coordinates was presented in [89].

Since neural networks are currently one of the main tools for processing 2D images, it is becoming common to use them for producing HVs of 2D images. This is especially relevant since directly transforming pixel values into HVs does not usually provide a good representation for solving machine learning tasks with competitive performance. For example, there were some studies [43, 110] directly transforming 2D images in MNIST to HVs. Their accuracy was further improved by either enhancing the HV encoder (e.g., with a receptive field and max pooling layer [107], or learning binary class HVs from a cross-entropy loss [17]. However, the reported accuracy was, in fact, lower than the one obtained with, e.g., a kNN classifier applied directly to pixels’ values, or a simple neural network with two layers. Recall, however, that the HV representations must be designed to capture the information that is important for solving the problem and presenting it in a form that can be exploited by the HDC/VSA mechanisms. Therefore, it is important to either apply some feature extraction techniques (as was, e.g., the case in [93, 97]) or use neural networks as a front-end. One of the earliest attempts to demonstrate this was presented in [194, 195]. The proposed approach takes activations of one of the hidden layers of a convolutional neural network caused by an input image. These activations are then binarized and the result of the binarization is used as an initial grid state of an elementary cellular automaton. The automaton is evolved for several steps and the results of previous steps are concatenated together, the result of which is treated as an HV corresponding to the image. The automaton evolution performs non-linear feature extraction, in the spirit of LIRA and RLD.

Later works used activations of convolutional neural networks without extra cellular automata computations. For example, in [116, 122] pre-trained off-the-shelf convolutional neural networks were used, while in [44, 71] a network’s attention mechanism and loss function were specifically designed to produce HVs with desirable properties. For example, it directed the output of the neural network to assign quasi-orthogonal HVs to images from different classes [71]. This quasi-orthogonality of the class HVs also enables continual learning of many novel classes, from very few training samples, with small interference, which leads to the state-of-the art classification accuracy on natural and handwritten images [44]. It is important to note that obtaining HVs from neural networks is a rather new direction and there are no studies that would scrupulously compare HVs obtained from different neural network architectures.

A graph, denoted as G, consists of nodes and edges. Edges can either be undirected or directed. Figure 3 presents examples of both directed and undirected graphs. First, we consider the following simple transformation of graphs into HVs [35]. A random HV is assigned to each node of the graph, following Figure 3 node HVs are denoted by letters (i.e., \(\mathbf {a}\) for node “a” and so on). An edge is represented as the binding of HVs of the connected nodes, e.g., the edge between nodes “a” and “b” is \(\mathbf {a} \circ \mathbf {b}\) . The whole graph is represented simply as the superposition of HVs (denoted as \(\mathbf {g}\) ) representing all edges in the graph, e.g., the undirected graph in Figure 3 is:

To represent directed graphs, the directions of the edges should be included into their HVs. This could be done, e.g., by applying a permutation, so that the directed edge from node “a” to “b” in Figure 3 is represented as \(\mathbf {a} \circ \rho (\mathbf {b})\) . Thus, the directed graph in Figure 3 is represented by the following HV:

Note that such representation implicitly labels graph’s nodes. The elements of the graph \(\mathbf {g}\) can be recovered using an item memory (Section 2.2.5). For graphs that have the same node HVs, the dot product is a measure of the number of overlapping edges. The described graph representations do not represent isolated vertices, but this could be fixed, by, e.g., separate representations of the vertex set and edge set [35].

When modeling, e.g., analogical reasoning (Section 3.1.2 of Part II of the survey [84]) or knowledge graphs, it is common to use acyclic graphs where both edges and nodes have associated labels, and edges are directed. Let us consider the representation of a graph shown in Figure 4, which can also be written in the bracketed notation as: cause(bite(Spot,Jane),flee(Jane,Spot)).

Using the role-filler binding approach (see Section 3.1.3), a relation, e.g., bite(Spot,Jane), can be represented by HVs (in HRR or BSC) as follows [139]:

Here, \(\mathbf {bite}\) , \(\mathbf {bite}_{\text{agent}}\) , and \(\mathbf {bite}_{\text{object}}\) are atomic HVs, while \(\mathbf {spot} = \langle \mathbf {spotID} + \mathbf {dog} \rangle\) and \(\mathbf {jane} = \langle \mathbf {janeID} + \mathbf {human} \rangle\) are the compositional ones. In the same way, \(\mathbf {FLEE}\) is formed, and finally the HV of the whole graph:

This particular representation was chosen to influence the similarity of the graph HVs because of the multiplicative binding operation properties in HRR and BSC. In SBDR, the binding preserves structured similarity in a different manner so the formation of graph’s resultant HV could be made more compact [146]:

Also, Predicate-Arguments relation representation using random permutations to represent the order of relation arguments was proposed in [68, 146, 153]:

Note that in the last six equations \(\langle \cdot \rangle\) refers to the CDT procedure. These types of graph representations by HVs will be further discussed in Section 3.1.2 of Part II of the survey [84] when describing analogical reasoning with HDC/VSA.

Representations of knowledge graphs were further studied in [109]. The work proposed to use a Cauchy distribution to generate atomic HVs and demonstrated the state-of-the-art results on the task of inferring missing links using HVs of knowledge graphs as an input to a neural network. HVs of knowledge graphs can also be constructed using HVs of nodes and relations that are learned from data as proposed in [125], where HRR was used due to its differentiability.

Trees are an instance of graphs where a node at the lower level (child) belongs to a single higher-level node (parent). Therefore, trees can be represented in the same manner as the graphs above. Please refer to the examples of the transformations to HVs given, e.g., in [153] for unordered binary trees and in [24, 78] for ordered binary trees. A representation of trees for the TPR model, using cryptographic hashing algorithms, was investigated in [40].