Language Design as Information Renormalization

Abstract

Here we consider some well-known facts in syntax from a physics perspective, allowing us to establish analogies between both fields with many consequences. Mainly, we observe that the operation MERGE, put forward by Chomsky (in: Evolution and Revolution in Linguistic Theory, Essays in honor of Carlos Otero., eds. Hector Campos and Paula Kempchinsky, 1995), can be interpreted as a physical information coarse-graining. Thus, MERGE in linguistics entails information renormalization in physics, according to different time scales. We make this point mathematically formal in terms of statistical language models. In this setting, MERGE amounts to a probability tensor implementing a coarse-graining, akin to a probabilistic context-free grammar. The probability vectors of meaningful sentences are given by stochastic tensor networks (TN) built from diagonal tensors and which are mostly loop-free, such as Tree Tensor Networks and Matrix Product States, thus being computationally very efficient to manipulate. We show that this implies the polynomially-decaying (long-range) correlations experimentally observed in language, and also provides arguments in favour of certain types of neural networks for language processing. Moreover, we show how to obtain such language models from quantum states that can be efficiently prepared on a quantum computer, and use this to find bounds on the perplexity of the probability distribution of words in a sentence. Implications of our results are discussed across several ambits.

Appendix A: More Analogies by Digging Deeper

Our paper is written having in mind a reader with background on physics and mathematics. However, the topic itself is strongly interdisciplinary. Because of this, in this appendix we would like to add some extra information useful for the reader with knowledge about theoretical linguistics. In particular, we would like to define a few concepts more precisely in linguistic jargon. Thanks to this, we will see that by digging deeper into the linguistic jargon, more analogies with physics will show up, in turn strengthening our thesis that MERGE in linguistics and RG in physics are deeply linked to each other.

To begin with, the term Universal Grammar (UG) is nothing but a label for the striking difference in cognitive capacity between “us and them”, i.e., humans versus the rest of animal species. UG is thus the research topic of generative grammar in its attempt to understand what it is and how it evolved in our species. Finding a satisfying answer to the latter question may be impossible with the tools we have right now, but any theory of UG seeking to address the former must meet a criterion of evolvability: any properties, mechanisms, etc. attributed to UG should have emerged in what appears to have been a unique and relatively sudden event on the evolutionary timescale [95, 96]. This line of thought presupposes that UG (the genetic encoding of the the human linguistic capacity) manifests bona fide traits of perfect design, in the sense that contains operations and mechanisms that follow from conceptual necessities, efficiency principles or interface demands. In this respect, linguistic expressions (sentences, phrases, words, etc.) are built up by adhering to these principles, therefore in an optimal fashion. While these notions are intuitively clear, their precise formulations remain vague and controversial.

One of the most important mathematical achievements of generative grammar is the so-called “Chomsky Hierarchy” [97], a classification of formal grammars according to their complexity. As Chomsky showed 60 years ago, human languages manifests both context-free and context-sensitive properties, needed to construct PHRASES and CHAINS respectively, shown in the Sentences A1(a,b):

$$\begin{aligned}&a.&\mathrm{John ~ killed ~ John.} \\& b.&\mathrm{John ~ was ~ killed ~ <John> } \nonumber \end{aligned}$$

(A1)

In Sen. A1(a) (a PHRASE) we have two tokens of the lexical item “John” that participate in phrasal dependencies to yield a compositional interpretation whereby the first John is the agent of a killing event, and the second John is the patient of such event. What we have in Sen. A1(b) (a CHAIN) is more complex. This time, we don’t have two tokens of “John”, but two occurrences of the same lexical item—as if they were one and the same object in two positions at the same time, where the notation $\mathrm{<John>}$ means that the word itself is not pronounced at that position, but it is also interpreted there from the logical point of view. This is what is called CHAIN in linguistics. In languages of the English type, the first (leftmost) occurence is spelled-out, whereas the second (rightmost) is necessary to keep a syntax-semantics homomorphism (that is, to capture the desideratum that a specific interpretation is tied to a specific position). Notice that the same type of object (a CHAIN) is necessary in Sen. A2, where “John” is pronounced to the left of seem, although it is interpreted as the patient of killed.

$$\begin{aligned} \mathrm{John ~ seems ~ to ~ have ~ been ~ klilled ~ <John> } \end{aligned}$$

(A2)

In order to account for these properties, generative grammar has resorted to phrase structure rules (PSR) and transformations. The most articulated version of PSR is known as X-bar Theory, which resorted to different devices that have been subject to a revision within minimalism. In particular, Chomsky [13] argued that the basic properties of PSR could be understood by means of a computational operation, dubbed MERGE [13], which captures two empirical properties of human language that are non-negotiable: discrete infinity and displacement. To be able to account for those properties, one must assume an operation that constructs hierarchically structured expressions with displacement. And that is what MERGE does. MERGE applies to two objects X and Y (be these words or bigger units), yielding a new one, K, which is the set containing X and Y, i.e., $\{ X, Y \}$. If X, Y are distinct (taken directly from the lexicon or independently assembled), K is constructed by what is called EXTERNAL MERGE (EM); if Y is part of X (if Y is contained in X), then we have what is called INTERNAL MERGE (IM). The latter scenario is that of Sentences A1(b) and A2 above, where MERGE turns “John” into a discontinuous object (a CHAIN). For completeness, if the operation is at the beginning of a derivation (e.g., with bare lexical items from a lexicon), it is called FIRST MERGE, and if it operates with partially-derived items (phrases), it is called ELSEWHERE MERGE.

Chomsky [13] takes MERGE to be strictly binary, as it is what is minimally necessary to create hierarchical structure. Generation by MERGE thus entails a restrictive class of recursively defined, binary-branching and discrete-hierarchical structures.

It is also worth mentioning that in X-bar Theory, the label identifies the properties of the entire phrase, at the cost of this being a theory-internal symbol that departs from inclusiveness demands. An alternative to this is a label-free representation (see Fig. 1), where endocentricity (the assumption that all phrases must be headed) is not preserved. This entails that syntactic objects can be exocentric, as seems to be necessary for objects formed by the combination of two phrases, $\{XP,YP \}$. Syntactic objects are “endocentric” if they contain an element that can be determined by Minimal Search—typically, a head. Given this logic, $\{X,YP \}$ is endocentric and $\{XP,YP \}$ exocentric. Consequently, such a system freely generates objects of different kinds, without stipulating their endocentric nature.

Moreover, MERGE is subject to efficiency and economy conditions. One such condition is inclusiveness, which precludes the introduction of extraneous objects, like the ones that X-bar Theory deployed: traces, bar-levels, projections, etc. Inclusiveness also bars introduction of features that are not present in lexical items.

To further clarify MERGE, we stress that the combination of two objects, X and Y, yields a new one, K, which is the set $\{ X, Y \}$. Once we have $\{X,Y \}$, we may want to merge K and some object W, which can be either internal to K or external to it (see above). In any event, the merger of W cannot change or tamper with $\{X,Y \}$, which behaves as a unit. More precisely, subsequent applications of MERGE must yield Eq. A3(a), not A3(b):

$$\begin{aligned}&a.&\mathrm{MERGE} (K,W) = \{ \{ X, Y \},W \} \\&b.&\mathrm{MERGE} (K,W) = \{ \{ X, W \},Y \} \nonumber \end{aligned}$$

(A3)

The driving force of this work is the fact that MERGE and renormalization seem to play a similar role on various respects. As noted above, MERGE takes two objects, X and Y, to yield a new one, K, thus removing X and Y from the computational workspace (WS). In the simplest scenario, MERGE maps $\mathrm{WS} = [X,Y]$ onto $\mathrm{WS'} = [\{X,Y \}]$, reducing the complexity of WS. Notice that MERGE never extends the WS, at least in terms of cardinality; thus $\mathrm{WS} = [\{X,Y \}]$ and $\mathrm{WS'} = [\{W,\{X,Y\}\}]$ are equally bigger, since they only contain one set. A new element can be added to WS (or $\mathrm{WS'}$) in only one way: by taking two items W, Z from the lexicon and introducing $\{W,Z \}$ into WS as a new element, yielding $\mathrm{WS''} = [\{W,Z\},\{X,Y\}]$. Of course, cardinality can be reduced if we apply EM (EXTERNAL MERGE) and neither of the elements are taken from the lexicon, as if we map $\mathrm{WS''} = [\{W,Z\},\{X,Y\}]$ onto $\mathrm{WS'''} = [\{\{W,Z\},\{X,Y\}\}]$. This idea is indeed very similar to that of a coarse-graining in physics, in the sense made precise throughout the paper.

Additionally, the possibility that computational load is reduced by MERGE is perhaps somewhat new, as this typically follows from a principle in linguistics that is called STRICT CYCLICITY. The notion of cycle (and thus cyclicity) goes back to the fifties, where work in phonology [99, 100] showed that the application of stress-assigning rules apply from innermost to outermost units of a word, putting aside linear order information. More generally, an object is build under cyclic principles if it is COMPOSITIONAL, which means that its interpretation is fixed by the elements it contains and the way in which they are combined. Consider this with Sentences A4, where the interpretation is crucially different (Brutus is an agent in (a), and a patient in (b)), although both examples contain the same three words:

$$\begin{aligned}&a.&\mathrm{Brutus ~ stabbed ~ Caesar.} \\&b.&\mathrm{Caesar ~ stabbed ~ Brutus.} \nonumber \end{aligned}$$

(A4)

The concept of STRICT CYCLICITY is a stronger version of cyclicity. The key intuition behind it is that for certain linguistic object constructed in a derivation (say, a VP), further computation should not modify it. Let us see this with the example in Eq. A5, where the verb “leave” is merged with the NP “the room” to yield the complex VP “leave the room”, which we can call K for ease of reference.

$$\begin{aligned} \mathrm{MERGE}(\mathrm{leave,\{the,room \}}) = \mathrm{\{leave,\{the,room\}\}}. \end{aligned}$$

(A5)

What is of interest here is that the interpretation of K (that is, of “leave the room”) is determined at that stage of the derivation (at that “cycle”), and cannot be changed at subsequent stages (“cycles”). Therefore, if we add “Mary” to obtain “Mary leaves the room” (call it $K'$), as in Eq. A6, the interpretation of K will be the same in Eq. A5 and in Eq. A6.

$$\begin{aligned} \mathrm{MERGE} (\mathrm{Mary},K) = \mathrm{\{Mary,\{leaves,\{the,room\}\}\} }. \end{aligned}$$

(A6)

In a nutshell, the interpretation of complex objects is constructed stepwise, in a step-by-step fashion, and whatever has been done at a stage s cannot be undone at statge $s+1$ (Eqs. A5 and A6 above). This, in turn, is quite analogue to the idea of irreversibility of RG flows in physics, which matches perfectly with our interpretation of MERGE as a coarse-graining of information.

Such stages at a derivation, where a “computation” is done and cannot be altered afterwards, correspond with the so-called linguistic PHASES, and the device responsible for ensuring that the interior of a PHASE is no longer accessible is the PHASE IMPENETRABILITY CONDITION (PIC for short). What has been called “phase” roughly corresponds with the notion of “cycle” described above. Using the physical interpretation that we introduce in this paper, one would say that a PHASE in linguistics is the analogous of an RG scale in physics.

To be more precise, a phase is defined in linguistics as a domain D where uninterpretable features (number and person features of verbs) are valued. When a phase is closed off, the complement domain $\Omega$ (which can itself be complex, in the sense of having some inner structure) of the phase head P cannot be modified from the outside; this means, for instance, that the case of an NP within $\Omega$ (e.g., “the book” in the VP “read the book”) cannot be changed once the phase headed by P is complete [31, 101]. Among other things, this entails that “the book”, which is the Direct Object of “read” Sentence A7 (it receives accusative case from “read”), cannot also be the Direct Object of the matrix verb “believe”:

$$\begin{aligned} \mathrm{I ~ believe ~ that ~ John ~ read ~ the ~ book.} \end{aligned}$$

(A7)

That “the book” is the Direct Object of “read” and not of “believe” is shown in Sentences A8, where we see that this NP can be passiviced in the embedded clause, but not in the matrix clause ($*$ signals ungrammaticality):

$$\begin{aligned}&a.&\mathrm{I ~ believe ~ that ~ the ~ book ~ was ~ read.} \\&b.&\mathrm{*The ~ book ~ was ~ believed ~ that ~ John ~ read.} \nonumber \end{aligned}$$

(A8)

This “shielding” effect that makes the VP impenetrable is captured by the phase impenetrability condition mentioned above. Physically, this is the irreversibility of the RG flow when moving from one RG scale to the next. There are various approaches to Phase Theory [98], but all of them share the key intuition that PHASES are domains where complexity is reduced by somehow allowing the system to “forget” about an amount of structure that has been created and which will be no longer accessible. This process of “forgetting” is, in fact, analogous to the process of “discarding irrelevant degrees of freedom” in an RG-step in physics.

Moreover, the “rescaling” step in RG has not been discussed in this paper, but also appears naturally when particularizing to specific models of language. For instance, in the Matrix Syntax model [65] this rescaling appears naturally in order to recover the correct linguistic labels after a MERGE operation (see Ref. [65] and the discussions therein for more information). We believe that this is a general feature: the “rescaling” in physics is nothing but the “mathematical relabelling” that one needs in order to recover the correct labels (NP, VP, etc) after a MERGE operation when dealing in practice with models of language.