Grammar-aware sentence classification on quantum computers

Abstract

Natural language processing (NLP) is at the forefront of great advances in contemporary AI, and it is arguably one of the most challenging areas of the field. At the same time, in the area of quantum computing (QC), with the steady growth of quantum hardware and notable improvements towards implementations of quantum algorithms, we are approaching an era when quantum computers perform tasks that cannot be done on classical computers with a reasonable amount of resources. This provides an new range of opportunities for AI, and for NLP specifically. In this work, we work with the Categorical Distributional Compositional (DisCoCat) model of natural language meaning, whose underlying mathematical underpinnings make it amenable to quantum instantiations. Earlier work on fault-tolerant quantum algorithms has already demonstrated potential quantum advantage for NLP, notably employing DisCoCat. In this work, we focus on the capabilities of noisy intermediate-scale quantum (NISQ) hardware and perform the first implementation of an NLP task on a NISQ processor, using the DisCoCat framework. Sentences are instantiated as parameterised quantum circuits; word-meanings are embedded in quantum states using parameterised quantum-circuits and the sentence’s grammatical structure faithfully manifests as a pattern of entangling operations which compose the word-circuits into a sentence-circuit. The circuits’ parameters are trained using a classical optimiser in a supervised NLP task of binary classification. Our novel QNLP model shows concrete promise for scalability as the quality of the quantum hardware improves in the near future and solidifies a novel branch of experimental research at the intersection of QC and AI.

Appendices

Appendix

In this supplementary material, we begin by briefly reviewing pregroup grammar. We then provide the necessary background to the graphical language of process theories describe our procedure for generating random sentence diagrams using a context-free grammar. For completeness we include the three labelled corpora of sentences we used in this work. Furthermore, we show details of our mapping from sentence diagrams to quantum circuits. Finally, we give details on the optimisation methods we used for our supervised quantum machine learning task and the specific compilation pass we used from CQC’s compiler, TKET.

Appendix A: Pregroup grammar

Pregroup grammars where introduced by Lambek as an algebraic model for grammar (Lambek 2008).

A pregroup grammar G is freely generated by the basic types in a finite set b ∈ B. Basic types are decorated by an integer $k\in \mathbb {Z}$, which signifies their adjoint order. Negative integers − k, with $k\in \mathbb {N}$, are called left adjoints of order k and positive integers $k\in \mathbb {N}$ are called right adjoints. We shall refer to a basic type to some adjoint order (include the zeroth order) simply as ‘type’. The zeroth order k = 0 signifies no adjoint action on the basic type and so we often omit it in notation, b⁰ = b.

The pregroup algebra is such that the two kinds of adjoint (left and right) act as left and right inverses under multiplication of basic types

$$ b^{k} b^{k+1} \to \epsilon \to b^{k+1} b^{k} ,$$

where 𝜖 ∈ B is the trivial or unit type. The left hand side of this reduction is called a contraction and the right hand side an expansion. Pregroup grammar also accommodates induced steps a → b for a,b ∈ B. The symbol ‘→’ is to be read as ‘type-reduction’ and the pregroup grammar sets the rules for which reductions are valid.

Now, to go from word to sentence, we consider a finite set of words called the vocabulary V. We call the dictionary (or lexicon) the finite set of entries $D \subseteq V \times (B\times \mathbb {Z})^{*}$. The star symbol A^∗ denotes the set of finite strings that can be generated by the elements of the set A. Each dictionary entry assigns a product (or string) of types to a word $t_{w} = {\prod }_{i} b_{i}^{k_{i}}$, $k_{i} \in \mathbb {Z}$.

Finally, a pregroup grammar G generates a language $L_{G} \subseteq V^{*}$ as follows. A sentence is a sequence (or list) of words σ ∈ V^∗. The type of a sentence is the product of types of its words $t_{\sigma } = {\prod }_{i} t_{w_{i}}$, where w_i ∈ V and i ≤|σ|. A sentence is grammatical, i.e. it belongs to the language generated by the grammar σ ∈ L_G, if and only if there exists a sequence of reductions so that the type of the sentence reduces to the special sentence-types ∈ B as $t_{\sigma } \to {\dots } \to s$. Note that it is in fact possible to type-reduce grammatical sentences only using contractions.

Appendix B: String diagrams

String diagrams describing process theories are generated by states, effects, and processes. In Fig. 6, we comprehensively show these generators along with constraining equations on them. String diagrams for process theories formally describe process networks where only connectivity matters, i.e. which outputs are connected to which inputs. In other words, the length of the wires carries no meaning and the wires are freely deformable as long as the topology of the network is respected.

Fig. 6

Diagrams are read from top to bottom. States have only outputs, effects have only inputs, processes (boxes) have both input and output wires. All wires carry types. Placing boxes side by side is allowed by the monoidal structure and signifies parallel processes. Sequential process composition is represented by composing outputs of a box with inputs of another box. A process transforms a state into a new state. There are special kinds of states called caps and effects called cups, which satisfy the snake equation which relates them to the identity wire (trivial process). Process networks freely generated by these generators need not be planar, and so there exists a special process that swaps wires and acts trivially on caps and cups

It is beyond the purposes of this work to provide a comprehensive exposition on diagrammatic languages. We provide the necessary elements which are used for the implementation of our QNLP experiments.

Appendix C: Random sentence generation with CFG

A context-free grammar generates a language from a set of production (or rewrite) rules applied on symbols. Symbols belong to a finite set Σ and There is a special type S ∈Σ called initial. Production rules belong to a finite set R and are of the form $T \to {\prod }_{i} T_{i}$, where T,T_i ∈Σ. The application of a production rule results in substituting a symbol with a product (or string) of symbols. Randomly generating a sentence amounts to starting from S and randomly applying production rules uniformly sampled from the set R. The production ends when all types produced are terminal types, which are non other than words in the finite vocabulary V.

From a process theory point of view, we represent symbols as types carried by wires. Production rules are represented as boxes with input and output wires labelled by the appropriate types. The process network (or string diagram) describing the production of a sentence ends with a production rule whose output is the S-type. Then we randomly pick boxes and compose them backwards, always respecting type-matching when inputs of production rules are fed into outputs of other production rules. The generation terminates when production rules are applied which have no inputs (i.e. they are states), and they correspond to the words in the finite vocabulary.

In Fig. 7 (on the left hand side of the arrows), we show the string-diagram generators we use to randomly produce sentences from a vocabulary of words composed of nouns, transitive verbs, intransitive verbs, and relative pronouns. The corresponding types of these parts of speech are N,TV,IV,RPRON. The vocabulary is the union of the words of each type, V = V_N ∪ V_TV ∪ V_IV ∪ V_RPRON.

Fig. 7

CFG generation rules used to produce the corpora K₃₀,K₆,K₁₆ used in this work, represented as string-diagram generators, where w_N ∈ V_N, w_TV ∈ V_TV, w_IV ∈ V_IV, w_RPRON ∈ V_RPRON. They are mapped to pregroup reductions by mapping CFG symbols to pregroup types, and so CFG-states are mapped to DisCoCat word-states and production boxes are mapped to products of cups and identities. Note that the pregroup unit 𝜖 is the empty wire and so it is never drawn. Pregroup type contractions correspond to cups and expansions to caps. Since grammatical reduction are achievable only with contractions, only cups are required for the construction of sentence diagrams

Having randomly generated a sentence from the CFG, its string diagram can be translated into a pregroup sentence diagram. To do so, we use the translation rules shown in Fig. 7. Note that a cup labeled by the basic type b is used to represent a contraction b^kb^k+ 1 → 𝜖. Pregroup grammars are weakly equivalent to context-free grammars, in the sense that they generate the same language (Buszkowski and Moroz 2007; Pentus 1993).

Appendix D: Corpora

Here, we present the sentences and their labels used in the experiments presented in the main text.

The types assigned to the words of this sentence are as follows. Nouns get typed as $t_{w\in V_{N}}=n^{0}$, transitive verbs are given type $t_{w\in V_{TV}}=n^{1} s^{0} n^{-1}$, intransitive verbs are typed t_w∈IV = n¹s⁰, and the relative pronoun is typed t_who = n¹n⁰s^− 1n⁰.

Corpus K₃₀ of 30 labeled sentences from the vocabulary

V_N = {’Dude’, ’Walter’}, V_TV = {’loves’, ’annoys’}, V_IV = {’abides’,’bowls’}, V_RPRON = {’who’}: [(’Dude who loves Walter bowls’, 1), (’Dude bowls’, 1), (’Dude annoys Walter’, 0), (’Walter who abides bowls’, 0), (’Walter loves Walter’, 1), (’Walter annoys Dude’, 1), (’Walter bowls’, 1), (’Walter abides’, 0), (’Dude loves Walter’, 1), (’Dude who bowls abides’, 1), (’Walter who bowls annoys Dude’, 1), (’Dude who bowls bowls’, 1), (’Dude who abides abides’, 1), (’Dude annoys Dude who bowls’, 0), (’Walter annoys Walter’, 0), (’Dude who abides bowls’, 1), (’Walter who abides loves Walter’, 0), (’Walter who bowls bowls’, 1), (’Walter loves Walter who abides’, 0), (’Walter annoys Walter who bowls’, 0), (’Dude abides’, 1), (’Dude loves Walter who bowls’, 1), (’Walter who loves Dude bowls’, 1), (’Dude loves Dude who abides’, 1), (’Walter who abides loves Dude’, 0), (’Dude annoys Dude’, 0), (’Walter who annoys Dude bowls’, 1), (’Walter who annoys Dude abides’, 0), (’Walter loves Dude’, 1), (’Dude who bowls loves Walter’, 1)]

Corpus K₆ of 6 labeled sentences from the vocabulary V_N = {’Romeo’, ’Juliet’}, V_TV = {’loves’}, V_IV = {’dies’}, V_RPRON = {’who’}: [(’Romeo dies’, 1.0), (’Romeo loves Juliet’, 0.0), (’Juliet who dies dies’, 1.0), (’Romeo loves Romeo’, 0.0), (’Juliet loves Romeo’, 0.0), (’Juliet dies’, 1.0)]

Corpus K₁₆ of 16 labeled sentences from the vocabulary V_N = {’Romeo’, ’Juliet’}, V_TV = {’loves’, ’kills’}, V_IV = {’dies’}, V_RPRON = {’who’}: [(’Juliet kills Romeo who dies’, 0), (’Juliet dies’, 1), (’Romeo who loves Juliet dies’, 1), (’Romeo dies’, 1), (’Juliet who dies dies’, 1), (’Romeo loves Juliet’, 1), (’Juliet who dies loves Juliet’, 0), (’Romeo kills Juliet who dies’, 0), (’Romeo who kills Romeo dies’, 1), (’Romeo who dies dies’, 1), (’Romeo who loves Romeo dies’, 0), (’Romeo kills Juliet’, 0), (’Romeo who dies kills Romeo’, 1), (’Juliet who dies kills Romeo’, 0), (’Romeo loves Romeo’, 0), (’Romeo who dies kills Juliet’, 0)]

Appendix E: Sentence to circuit mapping

Quantum theory has formally been shown to be a process theory. Therefore it enjoys a diagrammatic language in terms of string diagrams. Specifically, in the context of the quantum circuits we construct in our experiments, we use pure quantum theory. In the case of pure quantum theory, processes are unitary operations, or quantum gates in the context of circuits. The monoidal structure allowing for parallel processes is instantiated by the tensor product and sequential composition is instantiated by sequential composition of quantum gates.

In Fig. 8, we show the generic construction of the mapping from sentence diagrams to parameterised quantum circuits for the hyperparameters and parameterised word-circuits we use in this work.

Fig. 8

Mapping from sentence diagrams to parameterised quantum circuits. Here, we show how the generators of sentence diagrams are mapped to generators of circuits, for the hyperparameters we consider in this work

A wire carrying basic pregroup type b is given q_b qubits. A word-state with only one output wire becomes a one-qubit-state prepared from |0〉. For the preparation of such unary states we choose the sequence of gates defining an Euler decomposition of one-qubit unitaries R_z(𝜃₁) ∘ R_x(𝜃₂) ∘ R_z(𝜃₃). Word-states with more than one output wires become multiqubit states on k > 1 qubits prepared by an IQP-style circuit from ${\prod }_{i=1}^{k}\vert 0\rangle$. Such a word-circuit is composed of d-many layers. Each layer is composed of a layer of Hadamard gates followed by a layer in which every neighbouring pair of qubit wires is connected by a CR_z(𝜃) gate, $\left (\otimes _{i=1}^{k}H\right ) \circ \left (\otimes _{i=1}^{k-1} {CR_{z}}(\theta _{i})_{i,i+1}\right )$. Since all CR_z gates commute with each other it is justified to consider this as a single layer, at least abstractly. The Kronecker tensor with n-many output wires of type b is mapped to a GHZ state on nq_b qubits. Specifically, GHZ is a circuit that prepares the state ${\sum }_{x=0}^{2^{q_{b}}} \bigotimes _{i=1}^{n} \vert \text {bin}(x)\rangle$, where bin is the binary expression of an integer. The cup of pregroup type b is mapped to q_b-many nested Bell effects, each of which is implemented as a CNOT followed by a Hadamard gate on the control qubit and postselection on 〈00|.

Appendix F: Optimisation method

The gradient-free otpimisation method we use, Simultaneous Perturbation Stochastic Approximation (SPSA), works as follows. Start from a random point in parameter space. At every iteration pick randomly a direction and estimate the derivative by finite difference with step-size depending on c towards that direction. This requires two cost function evaluations. This provides a significant speed up the evaluation of L(𝜃). Then take a step of size depending on a towards (opposite) that direction if the derivative has negative (positive) sign. In our experiments we use minimizeSPSA from the Python package noisyopt https://github.com/andim/noisyopt, and we set a = 0.1 and c = 0.1, except for the experiment on ibmq for d = 3 for which we set a = 0.05 and c = 0.05.

Note that for classical simulations, we use just-in-time compilation of the cost function by invoking jit from jax (Bradbury et al. 2018). In addition, the choice of the squares-of-differences cost we defined in Eq.2 is not unique. One can as well use the binary cross entropy

$$L^{\text{BCE}}(\theta) = -\frac{1}{\vert{\Delta} \vert}{\sum}_{\sigma\in{\Delta}} l_{\sigma} \log l_{\sigma}^{\text{pr}}(\theta_{\sigma}) + (1-l_{\sigma}) \log(1-l_{\sigma}^{\text{pr}}(\theta_{\sigma}))$$

and the cost function can be minimised as well, as shown in Fig. 9.

Fig. 9

Minimisation of binary cross entropy cost function L^BCE with SPSA for the question answering task for corpus K₃₀

In our classical simulation of the experiment, we also used basinhopping (Olson et al. 2012) in combination with Nelder-Mead (Gao and Han 2010) from the Python package SciPy https://pypi.org/project/scipy. Nelder-Mead is a gradient-free local optimisation method. basinhopping hops (or jumps) between basins (or local minima) and then returns the minimum over local minima of the cost function, where each minimum is found by Nelder-Mead. The hop direction is random. The hop is accepted according to a Metropolis criterion depending on the the cost function to be minimised and a temperature. We used the default temperature value (1) and the default number of basin hops (100).

1.1 F.1: Error decay

In Fig. 10, we show the decay of mean training and test errors for the question answering task for corpus K₃₀ simulated classically, which is shown as inset in Fig. 4. Plotting in log-log scale we reveal, at least initially, an algebraic decay of the errors with the depth of the word-circuits.

Fig. 10

Algebraic decay of mean training and testing error for the data displayed in Fig. 4 (bottom) obtained by basinhopping. Increasing the depth of the word-circuits results in algebraic decay of the mean training and testing errors. The slopes are $\log e_{\text {tr}}\sim \log -1.2 d$ and $\log e_{\text {te}}\sim -0.3 \log d$. We attribute the existence of the plateau for e_tr at large depths is due the small scale of our experiment and the small values for our hyperparameters determining the size of the quantum-enhanced feature space

1.2 F.2: On the influence of noise to the cost function landscape

Regarding optimisation on a quantum computer, we comment on the effect of noise on the optimal parameters. Consider a successful optimisation of L(𝜃) performed on a NISQ device, returning $\theta ^{*}_{\text {NISQ}}$. However, if we instantiate the circuits $C_{\sigma }(\theta ^{*}_{\text {NISQ}})$ and evaluate them on a classical computer to obtain the predicted labels $l^{\text {CC}}_{\text {pr}}(\theta ^{*}_{\text {NISQ}})$, we observe that these can in general differ from the labels $l^{\text {NISQ}}_{\text {pr}}(\theta ^{*}_{\text {NISQ}})$ predicted by evaluating the circuits on the quantum computer. In the context of a fault-tolerant quantum computer, this should not be the case. However, since there is a non-trivial coherent-noise channel that our circuits undergo, it is expected that the optimiser’s result are affected in this way.

Appendix G: Quantum compilation

In order to perform quantum compilation, we use pytket (Sivarajah et al. 2020). It is a Python module for interfacing with CQC’s TKET, a toolset for quantum programming. From this toolbox, we need to make use of compilation passes.

At a high level, quantum compilation can be described as follows. Given a circuit and a device, quantum operations are decomposed in terms of the devices native gateset. Furthermore, the quantum circuit is reshaped in order to make it compatible with the device’s topology (Cowtan et al. 2019). Specifically, the compilation pass that we use is default_compilation_pass(2). The integer option is set to 2 for maximum optimisation under compilation https://github.com/CQCL/pytket.

Circuits written in pytket can be run on other devices by simply changing the backend being called, regardless whether the hardware might be fundamentally different in terms of what physical systems are used as qubits. This makes TKET it platform agnostic. We stress that on IBMQ machines specifically, the native gates are arbitrary single-qubit unitaries (‘U3’ gate) and entangling controlled-not gates (‘CNOT’ or ‘CX’). Importantly, CNOT gates show error rates which are one or even two orders of magnitude larger than error rates of U3 gates. Therefore, we measure the depth of or circuits in terms of the CNOT-depth. Using pytket this can be obtained by invoking the command depth_by_type(OpType.CX).

For both backends used in this work, ibmq_montreal and ibmq_toronto, the reported quantum volume is 32 and the maximum allowed number of shots is 2¹³.

Appendix H: Swap test and Hadamard test

In our binary classification NLP task, the predicted label is the norm squared of zero-to-zero transition amplitude where the unitary U represents the word-circuits and the circuits that implement the Bell effects as dictated by the grammatical structure. Estimating $\vert \langle 0{\dots } 0\vert U \vert 0{\dots } 0\rangle \vert ^{2}$, or the amplitude $\langle 0\dots 0\vert U \vert 0{\dots } 0\rangle$ itself in case one wants to define a cost function where it appears instead of its norm, can be done by postselecting on $\langle 0\dots 0\vert$. However, postselection costs exponential time in the number of postselected qubits; in our case, it needs to discard all bitstring sampled from the quantum computer that have Hamming weight other than zero. This is the procedure we follow in this proof of concept experiment, as we can afford doing so due to the small circuit sizes.

In such a setting, postselection can be avoided by using the swap test to estimate the normed square of the amplitude or the Hadamard test for the amplitude itself (Aharonov et al. 2008). See Fig. 11 for the corresponding circuits of those routines. In Fig. 12, we show how the swap test or the Hadamard test can be used to estimate the amplitude represented by the postselected sentence-circuit of Fig. 3. Furthermore, at least for tasks that are defined such that every sentence corresponds to a circuit, we argue that sentences do not grow arbitrarily long, and so the cost of evaluating the cost function is upper bounded in practical applications.

Fig. 11

(Left) Circuit for the Hadamard test. Measuring the control qubit in the computational basis allows one to estimate 〈Z〉 = Re(〈ψ|U|ψ〉) if b = 0, and 〈Z〉 = Im(〈ψ|U|ψ〉) if b = 1. The state ψ can be a multiqubit state, and in this work we are interested in the case $\psi = \vert 0{\dots } 0\rangle$. (Right) Circuit for the swap test. Sampling from the control qubit allows one to estimate 〈Z〉 = |〈ψ|ϕ〉|²

Fig. 12

Use of swap test (top) and Hadamard test (bottom) to estimate the norm-squared of the amplitude or the amplitude itself respectively, which is represented by the postselected circuit of Fig. 3