The Typical Set and Entropy in Stochastic Systems with Arbitrary Phase Space Growth

Abstract

The existence of the typical set is key for data compression strategies and for the emergence of robust statistical observables in macroscopic physical systems. Standard approaches derive its existence from a restricted set of dynamical constraints. However, given its central role underlying the emergence of stable, almost deterministic statistical patterns, a question arises whether typical sets exist in much more general scenarios. We demonstrate here that the typical set can be defined and characterized from general forms of entropy for a much wider class of stochastic processes than was previously thought. This includes processes showing arbitrary path dependence, long range correlations or dynamic sampling spaces, suggesting that typicality is a generic property of stochastic processes, regardless of their complexity. We argue that the potential emergence of robust properties in complex stochastic systems provided by the existence of typical sets has special relevance to biological systems.

1. Introduction

Many living systems are characterized by a high degree of internal stochasticity and display processes that form organization of growing complexity [1,2,3,4]. Such complexification processes exist on various scales, from the evolutionary scale [5,6], to the scale of single organisms [7]. The increase of complexity of the different forms of living entities triggered the debate whether the existence of open-endedness is a defining trait of biological evolution [8,9,10,11,12,13,14], with the resulting challenge of finding a potential statistical-physics-like characterization of it. At the single organism scale, the developmental process consists in the emergence of an adult multicellular organism from a single cell [7]. This astonishingly fast process of complexification sets challenges in many directions. One of these challenges is the characterization of the evolution of the space of potential configurations the system may acquire in time: In early embryo morphogenesis, for example, not only does the number of cells increase exponentially, resulting in the corresponding increase of potential configurations, but also cells differentiate into specialized cell types and create collective structures implying, in statistical physics language, that new states enter the system. This process is almost completely irreversible and, although highly precise, it is known to have a strong stochastic component [15,16,17]. Away from biology, non stable configuration spaces and processes of complexification have been identified in systems with innovation [18,19,20]. On the other hand, one can consider processes where the potential number of configurations decreases with time. Away from biology, recent advances in decay dynamics in nuclear physics were achieved considering a mathematical framework based on the stochastic collapse of the phase space [21,22,23]. In Figure 1 we schematically show the kind of processes we are exploring, all taking place in dynamic phase spaces, that is, processes where the amount of available possibilities may change (either grow or decrease) in time. Despite the ubiquity of such phenomena, a comprehensive characterization of systems with dynamic phase spaces, in terms equivalent to the ensemble theory of statistical mechanics, is lacking.

Ensemble formalism in statistical mechanics can be grounded in the concept of typicality [24,25,26,27,28]. Informally speaking, given the set of all potential sequences of events resulting from a stochastic process, a subset, the typical set, carries most of the probability [24,25]. This should not be confused with the set of most probable sequences: in the case of the biased coin, for example, the most probable sequence is not in the typical set. In other words, for long enough sequences, the probability that the observed sequence or state belongs to the subset of sequences forming the typical set goes asymptotically to 1. Generalizing to continuous systems, this is known as the concentration of measure phenomenon [29]. Accordingly, a typical property for a stochastic system is robust and acts as a strong, almost deterministic attractor as the process unfolds [27], and one may expect to observe it in the vast majority of cases. Moreover, if such a typical property exists, one can use this single property to—at least partially—characterize the system, and hence avoid a detailed microscopic description of all of the system’s components. Arguably, considerations based on typicality drive the connection between microscopic dynamics and macroscopic observables and underlie the existence of the thermodynamic limit [28,30,31]. In the context of information theory, the existence of the typical set for a given information source has deep consequences in the context of data compression [24,25].

The size of the typical set gives us valuable information on how the stochastic process is filling the phase space. In equilibrium systems or for information sources drawing independently from identically distributed (i.i.d.) random variables, the Gibbs–Shannon entropic functional arises naturally in the characterization of the typical set [24,25], establishing a clear connection between thermodynamics and phase space occupation. In systems/processes with collapsing or exploding phase spaces, path dependence or strong internal correlations [4,18,19,20,21,32,33,34,35,36,37], the phase space may grow super- or sub-exponentially, and the emergence of the Shannon–Gibbs entropic functional derived from phase space volume occupancy considerations is no longer guaranteed. The same situation may arise in cases dealing with non-stationary information sources [38,39,40,41,42]. Generalized forms for entropies have been proposed to encompass these more general scenarios [43,44,45,46,47,48,49,50,51], some of them explicitly linking the entropic functional to the expected evolution of the phase space volumes [36,46,52,53,54,55,56]. Despite the notable advances reported also for systems with physical significance [37,57,58], the concept of typicality has not been yet explored for systems/processes with exploding or shrinking phase spaces, which display path dependent dynamics or are subject to emergent internal constraints and correlations.

The purpose of this paper is to fill this important gap in the theory of stochastic processes, providing results with potential implications in the theory of non-equilibrium systems, data compression and coding strategies. As we shall see, the typical set can be defined for processes arbitrarily away from the i.i.d. framework, by only assuming a very generic convergence criteria, satisfied by a broad class of stochastic processes, that we refer to here as compact stochastic processes.

2. Results

2.1. Compact Stochastic Processes

Let us consider a general class of stochastic processes $\eta$ [59,60]. This class encompasses almost any discrete stochastic process that can be conceived. A realization of t steps of the process is denoted as $\eta \left(t\right)$:

$$\eta \left(t\right)={\eta }_1{\eta }_2\dots {\eta }_{t-1}{\eta }_t,$$

where ${\eta }_1,{\eta }_2\dots ,{\eta }_{t-1},{\eta }_t$ are random variables themselves. Note that, in different realizations of t steps of the process, the sequence of random variables can be different, as the process may display path dependence, long term correlations, or changes of the state space dynamics itself (either shrinking or expanding). We denote a particular trajectory/path the process may follow as:

$$x\left(t\right)\equiv x_1{x}_2\dots x_{t-1}{x}_t\in \mathsf{\Omega }\left(t\right),$$

$\mathsf{\Omega }\left(t\right)$ being the set of all possible paths of the process $\eta$ up to time t. We focus on the family of stochastic processes where there exists (i) a positive, strictly concave and strictly increasing function $\mathsf{\Lambda }\in {\mathcal{C}}^2$ in the interval $[1,\infty )$, such that $\mathsf{\Lambda }\left(1\right)=0$, and (ii) a positive, strictly increasing, $g\in {\mathcal{C}}^2$, in the interval $(1,\infty )$, by which:

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}\frac{1}{g\left(t\right)}\mathsf{\Lambda }\left(\frac{1}{p\left(\eta \right(t\left)\right)}\right)=1,\end{array}$$

(1)

where the convergence is in probability [60]. We will call this family of stochastic processes compact stochastic processes (CSP). Given a CSP process $\eta$, a pair of functions $\mathsf{\Lambda },g$ by which Equation (1) is satisfied define a compact scale of the CSP process $\eta$. Note that these two functions may not be unique for a given process, meaning that the process can, in principle, have several compact scales.

It is straightforward to check that, if $\eta$ is a sequence of i.i.d. random variables $X_1,\dots ,X_t\sim X$, $\mathsf{\Lambda }=log$ and $g\left(t\right)$ is t times the Shannon entropy of a single realization, $H\left(X\right)$, the above condition holds, as it recovers the standard formulation of the Asymptotic Equipartition Property (AEP) [24,25]. Therefore, the drawing of i.i.d. random variables $\sim X$ is a CSP with compact scale $(log,H(X\left)t\right)$. However, the range of potential processes that are CSPs is, in principle, much broader. In consequence, the first question we ask concerns the constraints that the convergence condition (1) imposes on $\mathsf{\Lambda }$. Assuming that (1) holds, one finds that the candidates to characterize CSPs are the $\mathsf{\Lambda }$s satisfying the following condition; see Proposition A1 of the Appendix B.2 for details:

$$\underset{z\to \infty }{lim}\frac{\mathsf{\Lambda }\left(\lambda z\right)}{\mathsf{\Lambda }\left(z\right)}=1,\forall \lambda \in {\mathbb{R}}^{+}.$$

(2)

Typical candidates for $\mathsf{\Lambda }$ are of the form $\mathsf{\Lambda }\left(z\right)=c{(log\left(z\right))}^d$, where $c,d$ are two positive, real valued constants or, more generally:

$$\mathsf{\Lambda }\left(z\right)=c_1{(log(1+c_2{(log(1+c_3{(log(\dots ))}^{d_3}))}^{d_2}))}^{d_1},$$

where $c_1,\dots$ and $d_1,\dots$ are positive, real valued constants. In previous approaches, these constants have been identified as scaling exponents, enabling us to classify the different potential growth dynamics of the phase space [54]. We observe that the existence of the inverse function of $\mathsf{\Lambda }$, ${\mathsf{\Lambda }}^{-1}$, by which $({\mathsf{\Lambda }}^{-1}\circ \mathsf{\Lambda })\left(z\right)=z$, is guaranteed by the assumption made in the definition of CSPs that $\mathsf{\Lambda }$ is a strictly monotonically growing function.

The convergence condition defining CSPs has a direct consequence on how probabilities are distributed along the set of all potential paths. Indeed, from Equation (1) it follows that there exist two non-increasing sequences of positive numbers ${ϵ}_1,\dots {ϵ}_t,\dots$, ${\delta }_1,\dots {\delta }_t,\dots$, with ${lim}_{t\to \infty }{ϵ}_t={lim}_{t\to \infty }{\delta }_t=0$, from which one can define a sequence of subsets of paths $A\left[{ϵ}_1\right]\dots A\left[{ϵ}_t\right]$ (such that $A\left[{ϵ}_1\right]\subseteq \mathsf{\Omega }\left(1\right),\dots ,A\left[{ϵ}_t\right]\subseteq \mathsf{\Omega }\left(t\right)$) as follows: For all $x\left(t\right)\in A\left[{ϵ}_t\right]$:

$$\frac{1}{{\mathsf{\Lambda }}^{-1}\left((1+{ϵ}_t)g\left(t\right)\right)}\le p\left(x\left(t\right)\right)\le \frac{1}{{\mathsf{\Lambda }}^{-1}\left((1-{ϵ}_t)g\left(t\right)\right)},$$

(3)

and the probability of a given path to belong to $A\left[{ϵ}_t\right]$ is bounded as:

$$\mathbb{P}(x\left(t\right)\in A\left[{ϵ}_t\right])>1-{\delta }_t,$$

(4)

where:

$$\mathbb{P}(x\left(t\right)\in A\left[{ϵ}_t\right])=\sum _{x\left(t\right)\in A\left[{ϵ}_t\right]}p\left(x\left(t\right)\right).$$

We call the sequence of subsets $A\left[{ϵ}_1\right]\dots A\left[{ϵ}_t\right]$ of the respective sampling spaces $\mathsf{\Omega }\left(1\right)\dots \mathsf{\Omega }\left(t\right)$ a sequence of typical sets of η. Informally speaking, Equation (4) tells us that, for large enough t, the probability of observing a path that does not belong to the typical set becomes negligible. As a consequence, the typical set can be identified for CSPs: Given a CSP, the typical set $A\left[{ϵ}_t\right]$ absorbs all the probability, in the limit $t\to \infty$. We summarize the above considerations in a theorem:

Theorem 1.

If η is a CSP in $(\mathsf{\Lambda },g)$, then there exist (a) a non-increasing sequence of positive numbers ${ϵ}_1,{ϵ}_2,\dots ,{ϵ}_{t-1},{ϵ}_t,\dots$ with limit ${lim}_{t\to \infty }{ϵ}_t=0$, associated to η and hence (b) the respective sequence of typical sets $A\left[{ϵ}_1\right],\dots ,A\left[{ϵ}_t\right],\dots$ by which:

$$\underset{t\to \infty }{lim}\mathbb{P}(x\left(t\right)\in A\left[{ϵ}_t\right])=1.$$

Proof. 

Since $\eta$ is a CSP we know, by assumption, that, as $t\to \infty$ also $\mathsf{\Lambda }(1/p(\eta \left(t\right)\left)\right)/g\left(t\right)\to 1$ (in probability). We can rewrite this condition by stating that, for every $ϵ,\delta >0$, there exists a $t_0$ by which, for each $t>t_0$ [60]:

$$\mathbb{P}\left(\left|\frac{1}{g\left(t\right)}\mathsf{\Lambda }\left(\frac{1}{p\left(x\right(t\left)\right)}\right)-1\right|>ϵ\right)<\delta .$$

Let $\tau (\epsilon ,\delta )$ be the smallest such $t_0$. We can then use two arbitrary strictly monotonic decreasing functions ${ϵ}_n^{*}$ and ${\delta }_n^{*}$ that converge to zero and construct a monotonically increasing sequence of times $t_n=\tau ({ϵ}_n^{*},{\delta }_n^{*})$ such that for all $t\ge t_n$ it is true that

$$\mathbb{P}\left(\left|\frac{1}{g\left(t\right)}\mathsf{\Lambda }\left(\frac{1}{p\left(x\right(t\left)\right)}\right)-1\right|>{ϵ}_n^{*}\right)<{\delta }_n^{*}.$$

From that, it is straightforward to define a non-increasing sequence ${ϵ}_1,\dots ,{ϵ}_t,\dots$ converging to 0, by just taking:

$$(\forall t:t_n\le t<t_{n+1}),{ϵ}_t={ϵ}_n^{*}.$$

Finally, the condition:

$$\underset{t\to \infty }{lim}\mathbb{P}(x\left(t\right)\in A\left[{ϵ}_t\right])=1,$$

follows as a direct consequence of the construction of the sequence of typical sets $A\left[{ϵ}_1\right],$$\dots ,A\left[{ϵ}_t\right],\dots$, thereby concluding the proof. □

We omitted a direct reference to the process $\eta$ in the notation of the typical set (i.e., $A\left[{ϵ}_t\right]\equiv A\left[{ϵ}_t\right]\left(\eta \right)$) for the sake of readability, and we will explicitly refer to it only if it is strictly necessary. In the next section we provide more details on the specific bounds in size by studying a subclass of the CSPs, namely, the class of simple CSPs. For them, the characterization of the typical set can be achieved using generalized forms of entropy.

2.2. The Typical Set and Generalized Entropies

Equation (1) can be related to a general form of path entropy:

$$S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)=\sum _{x\left(t\right)\in \mathsf{\Omega }\left(t\right)}p\left(x\left(t\right)\right)\mathsf{\Lambda }\left(\frac{1}{p\left(x\right(t\left)\right)}\right),$$

(5)

It can be proven that $S_{\mathsf{\Lambda }}$ satisfies three of the four Shannon–Khinchin axioms expected by an entropic functional [25,61,62] in Khinchin’s formulation [62], to be referred to as SK1, SK2, SK3. In particular SK1 states that entropy must be a function of the probabilities, which is satisfied by $S_{\mathsf{\Lambda }}$, by construction. SK2 states that $S_{\mathsf{\Lambda }}$ is maximized by the uniform distribution q over $\mathsf{\Omega }\left(t\right)$, i.e.,

$$q\left(x\left(t\right)\right)=\frac{1}{\left|\mathsf{\Omega }\right(t\left)\right|}.$$

Finally, SK3 states that, if $p\left(x\right(t\left)\right)=0$, then $p\left(x\right(t\left)\right)$ does not contribute to the entropy, which implies:

$$\underset{p\left(x\right(t\left)\right)\to 0}{lim}p\left(x\left(t\right)\right)\mathsf{\Lambda }\left(\frac{1}{p\left(x\right(t\left)\right)}\right)=0,$$

satisfied as well for any $\mathsf{\Lambda }$ considered in the definition of the CSPs.

We further observe that $S_{\mathsf{\Lambda }}$ is a monotonically increasing function as well, in the case of uniform probabilities: Let us suppose two CSPs $\eta$ and ${\eta }^{\prime }$ that sample uniformly their respective sampling spaces, $\mathsf{\Omega }\left(t\right),{\mathsf{\Omega }}^{\prime }\left(t\right)$, such that $\left|\mathsf{\Omega }\left(t\right)\right|<|{\mathsf{\Omega }}^{\prime }\left(t\right)\left|\right)$. Let, in consequence, q and $q^{\prime }$ be the uniform distributions over $\mathsf{\Omega }\left(t\right)$ and ${\mathsf{\Omega }}^{\prime }\left(t\right)$, respectively. Then:

$$S_{\mathsf{\Lambda }}\left(q\right)=\mathsf{\Lambda }\left(\left|\mathsf{\Omega }\right(t\left)\right|\right)<\mathsf{\Lambda }\left(|{\mathsf{\Omega }}^{\prime }\left(t\right)|\right)=S_{\mathsf{\Lambda }}\left(q^{\prime }\right),$$

where $S_{\mathsf{\Lambda }}\left(q\right),S_{\mathsf{\Lambda }}\left(q^{\prime }\right)$ are the generalized entropies as defined in Equation (5) applied to distributions q and $q^{\prime }$. In the Proposition A3 of the Appendix C we provide details of the above derivations. We observe that SK4 is not generally satisfied: This axiom states that $S\left(AB\right)=S\left(A\right)+S\left(B\right|A)$, and one can only guarantee its validity in the case of Shannon entropy, where $\mathsf{\Lambda }=log$. In the general case, this condition may not be satisfied. A different arithmetic rule can substitute SK4 to accommodate other entropic forms [46]. Notice, however, that the use of Shannon (path) entropy, i.e., $\mathsf{\Lambda }=log$, in the compact scale of a CSP may be used in in a broad spectrum of cases, including systems with correlations or super-exponential sample space growth, as we will see in Section 2.3.

If the contributions to the above entropy of the paths belonging to the complementary set of $A\left[{ϵ}_t\right]$, that is, $\mathsf{\Omega }\left(t\right)\setminus A\left[{ϵ}_t\right]$, are negligible in the limit of $t\to \infty$, then we call the CSP simple. For simple CSPs with compact scale $(\mathsf{\Lambda },g)$ the following convergence condition holds:

$$\frac{S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)}{g\left(t\right)}\to 1.$$

(6)

Moreover, in simple CSPs, the typical set has the largest contribution to the entropy. First we define the contribution of the typical set to the entropy, $S_{\mathsf{\Lambda }}\left(A\left[{ϵ}_t\right]\right)$, as:

$$S_{\mathsf{\Lambda }}\left(A\left[{ϵ}_t\right]\right)=\sum _{x\left(t\right)\in A\left[{ϵ}_t\right]}p\left(x\left(t\right)\right)\mathsf{\Lambda }\left(\frac{1}{p\left(x\right(t\left)\right)}\right).$$

We then demonstrate the above claim with the following proposition:

Theorem 2.

If η is a simple CSP with compact scale $(\mathsf{\Lambda },g)$, there exists a non-increasing sequence of positive numbers ${ϵ}_1,{ϵ}_2,\dots ,{ϵ}_{t-1},{ϵ}_t,\dots$ with limit ${lim}_{t\to \infty }{ϵ}_t=0$ and its corresponding sequence of typical sets $A\left[{ϵ}_1\right],\dots ,A\left[{ϵ}_t\right],\dots$, such that:

$$\underset{t\to \infty }{lim}\frac{S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)}{g\left(t\right)}=\underset{t\to \infty }{lim}\frac{S_{\mathsf{\Lambda }}\left(A\left[{ϵ}_t\right]\right)}{g\left(t\right)}=1.$$

Proof. 

We will start with the second equality, namely:

$$\underset{t\to \infty }{lim}\frac{S_{\mathsf{\Lambda }}\left(A\left[{ϵ}_t\right]\right)}{g\left(t\right)}=1.$$

From the definition of typical sets we know that, for paths $x\left(t\right)\in A\left[{ϵ}_t\right]$, it is true that:

$$\frac{1}{{\mathsf{\Lambda }}^{-1}\left((1+{ϵ}_t)g\left(t\right)\right)}\le p\left(x\left(t\right)\right)\le \frac{1}{{\mathsf{\Lambda }}^{-1}\left((1-{ϵ}_t)g\left(t\right)\right)}.$$

In consequence, given that $\mathbb{P}(x\left(t\right)\in A\left[{ϵ}_t\right]))>1-{\delta }_t$, one can bound $S_{\mathsf{\Lambda }}\left(A\left[{ϵ}_t\right]\right)$ as:

$$(1-{\delta }_t)(1-{ϵ}_t)<\frac{S_{\mathsf{\Lambda }}\left(A\left[{ϵ}_t\right]\right)}{g\left(t\right)}<(1-{\delta }_t)(1+{ϵ}_t).$$

Since, by construction ${lim}_{t\to \infty }{ϵ}_t={lim}_{t\to \infty }{\delta }_t=0$, this second part of the theorem is proven. From that, the statement of the theorem:

$$\underset{t\to \infty }{lim}\frac{S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)}{g\left(t\right)}=\underset{t\to \infty }{lim}\frac{S_{\mathsf{\Lambda }}\left(A\left[{ϵ}_t\right]\right)}{g\left(t\right)}=1,$$

follows directly given the assumption of simplicity. □

In consequence, the typical set can be naturally defined for simple CSPs in terms of the generalized entropy $S_{\mathsf{\Lambda }}$. To see that, we first reword condition (1) for simple CSPs as:

$$\underset{t\to \infty }{lim}\frac{1}{S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)}\mathsf{\Lambda }\left(\frac{1}{p\left(\eta \right(t\left)\right)}\right)=1,$$

(in probability). We can rewrite the above condition in a more convenient form: Given a simple CSP $\eta$, there are two non-increasing sequences of positive numbers ${ϵ}_1,\dots {ϵ}_t,\dots$, ${\delta }_1,\dots {\delta }_t,\dots$, with ${lim}_{t\to \infty }{ϵ}_t={lim}_{t\to \infty }{\delta }_t=0$, by which:

$$\mathbb{P}\left(\left|\frac{1}{S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)}\mathsf{\Lambda }\left(\frac{1}{p\left(x\right(t\left)\right)}\right)-1\right|>{ϵ}_t\right)<{\delta }_t.$$

(7)

Then, for each $t>0$ there is a set of paths, the typical set $A\left[{ϵ}_t\right]\subseteq \mathsf{\Omega }\left(t\right)$, such that for all $x\left(t\right)\in A\left[{ϵ}_t\right]$:

$$\begin{array}{ccc}\hfill \frac{1}{{\mathsf{\Lambda }}^{-1}\left((1+{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right)}& \le & p\left(x\right(t\left)\right)\hfill \\ & \le & \frac{1}{{\mathsf{\Lambda }}^{-1}\left((1-{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right)},\hfill \end{array}$$

by which $\mathbb{P}(x\left(t\right)\in A\left[{ϵ}_t\right])>1-{\delta }_t$. Notice that, now, the typical set is characterized using the generalized entropy $S_{\mathsf{\Lambda }}$.

The next obvious question refers to the cardinality of the typical set $\left|A\right[{ϵ}_t\left]\right|$. We will see that it can be bounded from above and below in a way analogous to the standard one [24]. The first bound is obtained by observing that:

$$\begin{array}{ccc}\hfill 1-{\delta }_t& \le & \sum _{x\left(t\right)\in A\left[{ϵ}_t\right]}p\left(x\left(t\right)\right)\hfill \\ & \le & \frac{\left|A\right[{ϵ}_t\left]\right|}{{\mathsf{\Lambda }}^{-1}\left((1-{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right)},\hfill \end{array}$$

where ${\mathsf{\Lambda }}^{-1}$ is the inverse function of $\mathsf{\Lambda }$, i.e., $({\mathsf{\Lambda }}^{-1}\circ \mathsf{\Lambda })\left(z\right)=z$, which exists given the assumption that $\mathsf{\Lambda }$ is a monotonically growing function made in the definition of CSPs. From that, it follows that the cardinality of the typical set is bounded from below as:

$$|A\left[{ϵ}_t\right]|\ge (1-{\delta }_t){\mathsf{\Lambda }}^{-1}\left((1-{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right).$$

(8)

For the upper bound, we observe that:

$$\begin{array}{ccc}\hfill 1& \ge & \sum _{x\left(t\right)\in A\left[{ϵ}_t\right]}p\left(x\left(t\right)\right)\hfill \\ & \ge & \frac{\left|A\right[{ϵ}_t\left]\right|}{{\mathsf{\Lambda }}^{-1}\left((1+{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right)},\hfill \end{array}$$

leading to:

$$|A\left[{ϵ}_t\right]|\le {\mathsf{\Lambda }}^{-1}\left((1+{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right).$$

(9)

Given the bounds provided in Equations (8) and (9), one can (roughly) estimate the cardinality of the typical set as:

$$|A\left[{ϵ}_t\right]|\approx {\mathsf{\Lambda }}^{-1}\left(S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right).$$

(10)

We present a rigorous version of the above result as a proposition.

Proposition 1.

Let η be a simple CSP in $(g,\mathsf{\Lambda })$ with some typical localizer sequence ${ϵ}_1,\dots ,{ϵ}_t,\dots$, then:

$$\underset{t\to \infty }{lim}\frac{\mathsf{\Lambda }\left(\right|A\left[{ϵ}_t\right]\left|\right)}{S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)}=1.$$

Proof. 

From Equations (8) and (9), one can derive the following chain of inequalities:

$$\begin{array}{ccc}\hfill \mathsf{\Lambda }\left((1-{ϵ}_t){\mathsf{\Lambda }}^{-1}\left((1-{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right)\right)& <& \mathsf{\Lambda }\left(\right|A\left[{ϵ}_t\right]\left|\right)\hfill \\ & <& (1+{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right).\hfill \end{array}$$

The last term poses no difficulties. To explore the behavior of the first one, we just rename the term:

$$z\equiv {\mathsf{\Lambda }}^{-1}\left((1-{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right),$$

and rewrite the first term of the inequality:

$$\mathsf{\Lambda }\left((1-{ϵ}_t){\mathsf{\Lambda }}^{-1}\left((1-{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right)\right)=\mathsf{\Lambda }\left((1-{ϵ}_t)z\right).$$

We know, from Proposition A1, that the functions we are dealing with behave such that:

$$\underset{z\to \infty }{lim}\frac{\mathsf{\Lambda }\left(\lambda z\right)}{\mathsf{\Lambda }\left(z\right)}\to 1,(\forall z>0).$$

As a consequence:

$$\frac{\mathsf{\Lambda }\left((1-{ϵ}_t){\mathsf{\Lambda }}^{-1}\left((1-{ϵ}_t)S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right)\right)}{S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)}\to 1.$$

Therefore, since also the third term goes trivially to $\sim S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)$, we can conclude that:

$$\frac{\mathsf{\Lambda }\left(\right|A\left[{ϵ}_t\right]\left|\right)}{S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)}\to 1,$$

as we wanted to demonstrate. □

The above proven asymptotic equivalence gives us the opportunity of rewriting the entropy in a Boltzmann-like form:

$$S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\sim \mathsf{\Lambda }\left(\right|A\left[{ϵ}_t\right]\left|\right).$$

This identifies the cardinality of the typical set with Boltzmann’s W, the number of alternatives the system can effectively display: Finally, we notice that we can (roughly) approximate the typical probabilities as:

$$p\left(x\left(t\right)\right)\approx \frac{1}{{\mathsf{\Lambda }}^{-1}\left(S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)\right)}.$$

We thus provided a general proof that the typical set exists and that it can be properly defined for a wide class of stochastic processes, the CSPs, those satisfying convergence condition (1). Moreover, we show that its volume can be bounded and fairly approximated as a function of the generalized entropy emerging from the convergence condition, $S_{\mathsf{\Lambda }}$, as defined in Equation (5).

2.3. Example: A Path Dependent Process

We briefly explore the behavior of the typical set and its associated entropic forms through a model displaying both path dependence and unbounded growth of the phase space. The process $\eta$ works as follows: Let us suppose we have a restaurant with an infinite number of tables $m_1,\dots .,m_n,\dots$. At $t_0=0$ a customer enters the restaurant and sits at table $m_1$. At time t a new customer enters the restaurant where already $m\left(t\right)$ tables are occupied; the occupation number of each table is unbounded. The customer can chose either to sit at an already occupied table from the $m_1,\dots ,m_{m\left(t\right)}$ occupied tables, each with equal probability $\frac{1}{m\left(t\right)+1}$, or in the next unoccupied one, $m_{m\left(t\right)+1}$, again with probability $\frac{1}{m\left(t\right)+1}$. This process is a version of the so-called Chinese restaurant process [33,63], exhibiting a simple form of memory/path dependence. Hence, we refer to it as the Chinese restaurant process with memory (CRPM). In Figure 2 we sketch the rules of this process. Crucially, as $t\to \infty$, the random variable accounting for the number of tables $m\left(t\right)$ has the following convergent behavior; see Proposition A5 of the Appendix D.2 for details:

$$\frac{m\left(t\right)}{\sqrt{2t}}\to 1.$$

In Figure 3a we see that the prediction $m\left(t\right)\sim \sqrt{2t}$ is quite accurate when compared to numerical simulations of the process. This property enables us to demonstrate that the CRPM we are studying is actually a CSP with compact scale $(log,\frac{t}{2}logt)$; see Theorem A1 of the Appendix D.3. In particular, Equation (1) is satisfied, in this particular case as:

$$\underset{t\to \infty }{lim}\frac{1}{\frac{t}{2}logt}log\left(\frac{1}{p\left(\eta \right(t\left)\right)}\right)=1,$$

in probability. In addition, the process is simple; see Theorem A2. Since we are using $\mathsf{\Lambda }=log$, the entropy form that will arise is Shannon path entropy, by direct application of Equation (5), i.e., $S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)=H\left(\eta \left(t\right)\right)$, with $H\left(\eta \right(t\left)\right)$ defined as:

$$H\left(\eta \left(t\right)\right)=-\sum _{x\left(t\right)\in \mathsf{\Omega }\left(t\right)}p\left(x\left(t\right)\right)logp\left(x\left(t\right)\right).$$

(11)

It directly follows that:

$$\frac{H\left(\eta \right(t\left)\right)}{\frac{t}{2}logt}\to 1.$$

(12)

Given the compact scale used, one can estimate the evolution of the size of the typical set as:

$$|A\left[{ϵ}_t\right]|\sim \sqrt{\mathsf{\Gamma }\left(t\right)},$$

(13)

where $\mathsf{\Gamma }$ is the standard $\mathsf{\Gamma }$-function [64]. We see that the growth of the typical set as shown in Equation (13) is clearly faster than exponential. The phenomenon of concentration of measure [29] is clearly manifested here, as the size of the typical set vanishes in relation to the size of the whole potential set of outcomes, $\mathsf{\Omega }\left(t\right)$. A rough estimation leads to $\left|\mathsf{\Omega }\right(t\left)\right|\sim \mathsf{\Gamma }\left(t\right)$, leading to:

$$\frac{\left|A\right[{ϵ}_t\left]\right|}{\left|\mathsf{\Omega }\right(t\left)\right|}\sim \frac{1}{\sqrt{\mathsf{\Gamma }\left(t\right)}}\to 0,$$

despite $\mathbb{P}(x\left(t\right)\in A\left[{ϵ}_t\right])\to 1$.

Additionally, in Figure 3b we see that the prediction made in Equation (12) fits perfectly with the numerical realizations of the process. Note that we have shown the dependence on Shannon path entropy for the clarity in the exposition. Indeed, as pointed out above, a CSP $\eta$ may have several compact scales. For example, taking the compact scale that led to Shannon entropy, $(log,g(t\left)\right)$, with $g\left(t\right)=\frac{t}{2}logt$, one can construct another compact scale for the CRPM by composing $g^{-1}$ (which, by assumption, exists) with both functions. In consequence, one will have a new compact scale $(\mathsf{\Lambda },\tilde{g})$, defined as:

$$\mathsf{\Lambda }\left(t\right)=(g^{-1}\circ log)\left(t\right)\sim \frac{2log\left(t\right)}{\mathbf{W}(2log(t\left)\right)},\tilde{g}\left(t\right)=t,$$

(14)

where $\mathbf{W}$ is the positive, real branch of the Lambert function [64]. In Figure 3c we see that $S_{\mathsf{\Lambda }}\left(\eta \left(t\right)\right)$ fits perfectly $g\left(t\right)\sim t$, proving that $(\mathsf{\Lambda },t)$ is a compact scale for the CRPM; see also Appendix D.5. We observe that this particular compact scale makes the path entropy $S_{\mathsf{\Lambda }}$ extensive when applied to the CRPM.

3. Discussion

We demonstrated that, for a very general class of stochastic processes, which we refer to as compact stochastic processes, the typical set is well defined, as the probability measure tends to concentrate in clearly identifiable regions of the space of possible outcomes of the process. These processes can be path dependent, contain arbitrary internal correlations or display dynamic behavior of the phase space, showing sub- or super-exponential growth on the effective number of configurations the system can achieve. The only requirement is that there exist two functions $\mathsf{\Lambda },g$ for which Equation (1) holds. Along the existence of the typical set, a generalized form of entropy naturally arises, from which, in turn, the cardinality of the typical set can be computed.

The existence of the typical set in systems with arbitrary phase space growth opens the door to a proper characterization, in terms of statistical mechanics, of a number of processes, mainly biological, where the number of configurations and states changes over time. In particular, it paves the path towards the statistical-mechanics-like understanding of processes showing open-ended evolution on the basis of typicality. For example, this could encompass thermodynamic characterizations of—part of—the developmental paths in early stages of embryogenesis. The existence of the typical set, even in some extreme scenarios of stochasticity and phase space behavior, leads us to the speculative hypothesis that typicality may underlie the astonishing reproducibility and precision of some biological processes. In this scenario, stochasticity would drive the system to the set of correct configurations—those belonging to the typical set—with high accuracy. Selection, in turn, would operate on typical sets, thereby promoting certain stochastic processes over others. More specific scenarios are nevertheless required in order to make this intriguing hypothesis more sound. Further works should clarify the potential of the proposed probabilistic framework to accommodate generalized, consistent forms of thermodynamics and explore the complications that can arise due to the loss of ergodicity that is characteristic for some of the processes that are nonetheless compatible with the above description.

Importantly, our results provide a potential starting point for an ensemble formalism for systems with sub- or super-exponential phase space growth. This opens the possibility to extend the concept of thermodynamic limit to these systems without requiring further conditions such as microscopic detailed balance, which cannot be justified in a broad range of out-of-equilibrium processes. Questions like the definition of free energies or the possible need of extensivity, which have to be answered in order to progress towards a complete and consistent thermodynamic picture remain, however, open. For tentative answers to those questions, connections to early proposals could be drawn, both on the thermodynamic grounds; see, e.g., [36,65,66], and from the perspective of entropy characterization; see, e.g., [37,44,46,53,54]. In this paper we meet an equivalence relation underlying compact scales, which allows us to transform between compact scales that do not differ too strongly, i.e., not more than by a power, without essentially changing the structure of the typical set sequence, a fact that one can utilize to give the entropic functional particular properties. For instance making the entropy associated with the compact scale extensive, as we demonstrated above for the CRPM. However, this also introduces the possibility that processes may have more than one inequivalent compact scale. We are aware that, in order to demonstrate the full potential of the theory one should aim at examples more extreme than the Chinese restaurant process we present. However, this would require an additional inquiry into the potentially hierarchic structure of potentially inequivalent compact scales that both in technical terms and conceptual terms go beyond the scope of this paper. This intriguing issue may be related to different levels of coarse graining, and deserves further investigations.

We finally point out the implications of our results for the study of information sources, given the fundamental role the typical set plays in optimal coding and data compression. The existence of the typical set in these broad class of information sources, where, simply put, the information flow is not constant, may open the possibility of new compressing strategies. These strategies could be based, for example, on properties of the specific CSP representing the information source and the compact scale $\mathsf{\Lambda },g$ that characterize the process.