arXiv:1001.2152v1 [math.ST] 13 Jan 2010 Bernoulli 15(4), 2009, 1351–1367 DOI: 10.3150/09-BEJ191 Rate of convergence of predictive distributions for dependent data PATRIZIA BERTI1 , IRENE CRIMALDI2 , LUCA PRATELLI3 and PIETRO RIGO4 1 Dipartimento di Matematica Pura ed Applicata “G. Vitali”, Universita’ di Modena e ReggioEmilia, via Campi 213/B, 41100 Modena, Italy. E-mail: patrizia.berti@unimore.it 2 Dipartimento di Matematica, Universita’ di Bologna, Piazza di Porta San Donato 5, 40126 Bologna, Italy. E-mail: crimaldi@dm.unibo.it 3 Accademia Navale, viale Italia 72, 57100 Livorno, Italy. E-mail: pratel@mail.dm.unipi.it 4 Dipartimento di Economia Politica e Metodi Quantitativi, Universita’ di Pavia, via S. Felice 5, 27100 Pavia, Italy. E-mail: prigo@eco.unipv.it This paper deals with empirical processes of the type √ Cn (B) = n{µn (B) − P (Xn+1 ∈ B | X1 , . . . , Xn )}, P where (Xn ) is a sequence of random variables and µn = (1/n) n i=1 δXi the empirical measure. Conditions for supB |Cn (B)| to converge stably (in particular, in distribution) are given, where B ranges over a suitable class of measurable sets. These conditions apply when (Xn ) is exchangeable or, more generally, conditionally identically distributed (in the sense of Berti et al. [Ann. Probab. 32 (2004) 2029–2052]). By such conditions, in some relevant situations, one obtains that √ P supB |Cn (B)| → 0 or even that n supB |Cn (B)| converges a.s. Results of this type are useful in Bayesian statistics. Keywords: Bayesian predictive inference; central limit theorem; conditional identity in distribution; empirical distribution; exchangeability; predictive distribution; stable convergence 1. Introduction and motivations A number of real problems reduce to the evaluation of the predictive distribution an (·) = P (Xn+1 ∈ ·|X1 , . . . , Xn ) for a sequence X1 , X2 , . . . of random variables. Here, we focus on those situations where an cannot be calculated in closed form and one decides to estimate it based on the available data X1 , . . . , Xn . Related references are [1–3, 5, 6, 8, 10, 15, 18, 20]. This is an electronic reprint of the original article published by the ISI/BS in Bernoulli, 2009, Vol. 15, No. 4, 1351–1367. This reprint differs from the original in pagination and typographic detail. 1350-7265 c 2009 ISI/BS 1352 Berti, Crimaldi, Pratelli and Rigo For notational reasons, it is convenient to work in coordinate probability space. Accordingly, we fix a measurable space (S, B) and a probability P on (S ∞ , B ∞ ), and we let Xn be the nth canonical projection on (S ∞ , B ∞ , P ), n ≥ 1. We also let Gn = σ(X1 , . . . , Xn ) and X = (X1 , X2 , . . .). Since we are concerned with predictive distributions, it is reasonable to make some (qualitative) assumptions about them. In [6], X is said to be conditionally identically distributed (c.i.d.) when a.s. for all B ∈ B and k > n ≥ 0, E(IB (Xk )|Gn ) = E(IB (Xn+1 )|Gn ) where G0 is the trivial σ-field. Thus, at each time n ≥ 0, the future observations (Xk : k > n) are identically distributed given the past Gn . In a sense, this is a weak form of exchangeability. In fact, X is exchangeable if and only if it is stationary and c.i.d., and various examples of non-exchangeable c.i.d. sequences are available. In the sequel, X = (X1 , X2 , . . .) is a c.i.d. sequence of random variables. In that case, a sound estimate of an is the empirical distribution n µn = 1X δX . n i=1 i The choice of µn can be defended as follows. Let D ⊂ B and let k · k denote the sup-norm on D. Suppose also that D is countably determined, as defined in Section 2. (The latter is a mild condition, only needed to handle measurability issues.) Then a.s. kµn − an k = sup |µn (B) − an (B)| −→ 0, (1) B∈D provided (X is c.i.d. and) µn converges uniformly on D with probability 1; see [5]. For a.s. instance, kµn − an k −→ 0 whenever X is exchangeable and D is a Glivenko–Cantelli class. a.s. Also, kµn − an k −→ 0 if S = R, D = {(−∞, t] : t ∈ R}, and X1 has a discrete distribution or inf ε>0 lim inf n P (|Xn+1 − Xn | < ε) = 0; see [4]. To sum up, under mild assumptions, µn is a consistent estimate of an (with respect to uniform distance) for c.i.d. data. This is in line with de Finetti [10] in the particular case of exchangeable indicators. Taking (1) as a starting point, the next step is to investigate the convergence rate, that is, to investigate whether αn kµn − an k converges in distribution, possibly to a null limit, for suitable constants αn > 0. This is precisely the purpose of this paper. A first piece of information on the convergence rate of kµn − an k can be obtained as follows. For B ∈ B, define µ(B) = lim sup µn (B), n √ Wn (B) = n{µn (B) − µ(B)}. Rate of convergence of predictive distributions 1353 a.s. By the SLLN for c.i.d. sequences, µn (B) −→ µ(B); see [6]. Hence, for fixed n ≥ 0 and B ∈ B, one obtains E(µ(B)|Gn ) = lim E(µk (B)|Gn ) = lim k k k 1 X E(IB (Xi )|Gn ) k i=n+1 = E(IB (Xn+1 )|Gn ) = an (B) In turn, this implies that a.s. √ n{µn (B) − an (B)} = E(Wn (B)|Gn ) a.s., so 1 1 kµn − an k ≤ √ sup E(|Wn (B)||Gn ) ≤ √ E(kWn k|Gn ) n B∈D n a.s. If supn EkWn kk < ∞ for some k ≥ 1, it then follows that E{(αn kµn − an k)k } ≤  αn √ n k EkWn kk → 0 αn whenever √ → 0. n Even if obvious, this fact is potentially useful since sup EkWn kk < ∞ for all k ≥ 1, if X is exchangeable, n (2) for various choices of D; see Remark 3. In particular, (2) holds if D is finite. √ The intriguing case, however, is αn = n. For each B ∈ B and probability Q on (S ∞ , B ∞ ), write CnQ (B) = EQ (Wn (B)|Gn ) and √ Cn (B) = CnP (B) = n{µn (B) − an (B)}. In Theorem 3.3 of [6], the asymptotic behavior of Cn (B) is investigated for fixed B. Here, instead, we are interested in kCn k = sup |Cn (B)| = B∈D √ nkµn − an k. Our main result (Theorem 1) is the following. Fix a random probability measure N on R and a probability Q on (S ∞ , B ∞ ) such that kCnQ k → N stably under Q and kWn k is uniformly integrable under both P and Q. Then, kCn k → N stably whenever P ≪ Q. (3) 1354 Berti, Crimaldi, Pratelli and Rigo A remarkable particular case is N = δ0 . Suppose, in fact, that for some Q, one has Q kCnQ k → 0 and kWn k uniformly integrable under P and Q. Then, P kCn k → 0 whenever P ≪ Q. Stable convergence (in the sense of Rényi) is a stronger form of convergence in distribution. The definition is recalled in Section 2. In general, one cannot dispense with the uniform integrability condition. However, this condition is often true. For instance, kWn k is uniformly integrable (under P and Q) provided D meets (2) and X is exchangeable (under P and Q). To make (3) concrete, a large list of reference probabilities Q is needed. Various examples are available in the Bayesian nonparametrics framework; see, for example, [16] and references therein. The most popular is perhaps the Ferguson–Dirichlet law, denoted by Q0 . If P = Q0 , then X is exchangeable and an (B) = αP (X1 ∈ B) + nµn (B) α+n a.s. for some constant α > 0. P Since kµn −an k ≤ (α/n) when P = Q0 , something more than kCn k → 0 can be expected in the case P ≪ Q0 . Indeed, we prove that √ converges a.s. nkµn − an k = nkCn k whenever P ≪ Q0 with a density satisfying a certain condition; see Theorem 2 and Corollary 5. One more example should be mentioned. Let Xn = (Yn , Zn ), where Zn > 0 and Pn αP (Y1 ∈ B) + i=1 Zi IB (Yi ) Pn P (Yn+1 ∈ B|Gn ) = a.s. α + i=1 Zi for some constant α > 0. Under some conditions, X is c.i.d. (but not necessarily exchangeable), kWn k is uniformly integrable and kCn k converges stably; see Section 4. The above material takes a nicer form when the condition P ≪ Q can be given a simple characterization. This happens, for instance, if S = {x1 , . . . , xk , xk+1 } is finite, X exchangeable and P (X1 = x) > 0 for all x ∈ S. Then, P ≪ Q0 (for some choice of Q0 ) if and only if (µ{x1 }, . . . , µ{xk }) has an absolutely continuous distribution with respect to Lebesgue measure. In this particular case, however, a part of our results can also be obtained through the Bernstein– von Mises theorem; see Section 3. Finally, we make two remarks: (i) If X is exchangeable, our results apply to Bayesian predictive inference. Suppose, in fact, that S is Polish and B the Borel σ-field, so that de Finetti’s theorem applies. Then, Rate of convergence of predictive distributions 1355 P is a unique mixture of product probabilities on B ∞ and the mixing measure is called the prior distribution in a Bayesian framework. Now, given Q, P ≪ Q is just an assumption on the prior distribution. This is plain in the last example where S = {x1 , . . . , xk , xk+1 }. In Bayesian terms, such an example can be summarized as follows. For a multinomial P statistical model, √ kCn k → 0 if the prior is absolutely continuous with respect to Lebesgue measure, and nkCn k converges a.s. if the prior density satisfies a certain condition. (ii) To our knowledge, there is no general representation for the predictive distributions of an exchangeable sequence. Such a representation would be very useful. Even if only partially, results like (3) contribute to filling the gap. As an example, for fixed B ∈ B, one obtains an (B) = µn (B) + oP ( √1n ), provided X is exchangeable and P ≪ Q for some Q Q such that CnQ (B) → 0 and Wn (B) is uniformly integrable. 2. Main results A few definitions need to be recalled. Let T be a metric space, BT the Borel σ-field on T and (Ω, A, P ) a probability space. A random probability measure on T is a mapping N on Ω × BT such that: (i) N (ω, ·) is a probability on BT for each ω ∈ Ω; (ii) N (·, B) is A-measurable for each B ∈ BT . Let (Zn ) be a sequence of T -valued random variables and N a random probability measure on T . Both (Zn ) and N are defined on (Ω, A, P ). We say that Zn converges stably to N in the case where P (Zn ∈ ·|H) → E(N (·)|H) weakly for all H ∈ A such that P (H) > 0. Clearly, if Zn → N stably, then Zn converges in distribution to the probability law E(N (·)) (just let H = Ω). Stable convergence has been introduced by Rényi in [17] and subsequently investigated by various authors; see [9] for more information. Next, we say that D ⊂ B is countably determined in the case where, for some fixed countable subclass D0 ⊂ D, one obtains supB∈D0 |ν1 (B) − ν2 (B)| = supB∈D |ν1 (B) − ν2 (B)| for every pair ν1 , ν2 of probabilities on B. A sufficient condition is that for some countable D0 ⊂ D, and for every ε > 0, B ∈ D and probability ν on B, there is B0 ∈ D0 satisfying ν(B∆B0 ) < ε. Most classes D involved in applications are countably determined. For instance, D = {(−∞, t] : t ∈ Rk } and D = {closed balls} are countably determined if S = Rk and B is the Borel σ-field. As another example, D = B is countably determined if B is countably generated. We are now in a position to state our main result. Let N be a random probability measure on R, defined on the measurable space (S ∞ , B ∞ ), and let Q be a probability on (S ∞ , B ∞ ). Theorem 1. Let D be countably determined. Suppose kCnQ k → N stably under Q, and (kWn k : n ≥ 1) is uniformly integrable under P and Q. Then, kCn k = √ nkµn − an k → N stably whenever P ≪ Q. 1356 Berti, Crimaldi, Pratelli and Rigo Proof. Since D is countably determined, there are no measurability problems in taking supB∈D . In particular, kWn k and kCn k are random variables and kCn k is Gn -measurable. Let f be a version of dP dQ and Un = f − EQ (f |Gn ). Then, Cn (B) = E(Wn (B)|Gn ) = = CnQ (B) + Letting Mn = EQ (|Un |kWn k|Gn ) EQ (f |Gn ) EQ (f Wn (B)|Gn ) EQ (f |Gn ) EQ (Un Wn (B)|Gn ) , EQ (f |Gn ) P -a.s., for each B ∈ B. and taking supB∈D , it follows that kCnQ k − Mn ≤ kCn k ≤ kCnQ k + Mn , P -a.s. We first assume f to be bounded. Since kCnQ k → N stably under Q, given a bounded random variable Z on (S ∞ , B ∞ ), one obtains Z Z Q φ(kCn k)Z dQ −→ N (φ)Z dQ for each bounded continuous φ : R → R, where N (φ) = Z φ(x)N (·, dx). Letting Z = f IH /P (H) with H ∈ B ∞ and P (H) > 0, it follows that kCnQ k → N stably under P . Therefore, it suffices to prove that EMn → 0. Given ε > 0, since kWn k is uniformly integrable under Q, there exists some c > 0 such that EQ {kWn kI{kWn k>c} } < ε sup f for all n. Since Mn is Gn -measurable, EMn = EQ (f Mn ) = EQ (EQ (f |Gn )Mn ) = EQ (|Un |kWn k) ≤ cEQ |Un | + (sup f )EQ (kWn kI{kWn k>c} ) < cEQ |Un | + ε for all n. Therefore, the martingale convergence theorem implies that lim sup EMn ≤ c lim sup EQ |Un | + ε = ε. n n This concludes the proof when f is bounded. Next, let f be any density. Fix k > 0 such that P (f ≤ k) > 0 and define K = {f ≤ k} and PK (·) = P (·|K). Then, PK has the bounded density f IK /P (K) with respect to Q. By what has already been proven, kCnPk k → N stably under PK , where CnPk (B) = EPK (Wn (B)|Gn ) = E{IK Wn (B)|Gn } , E(IK |Gn ) PK -a.s. Rate of convergence of predictive distributions 1357 Letting Rn = IK − E(IK |Gn ), it follows that   E{Rn Wn (B)|Gn } Pk E{IK kCn − Cn k} = E IK sup E(IK |Gn ) B∈D   E{|Rn |kWn k|Gn } = E{|Rn |kWn k} ≤ E IK E(IK |Gn ) ≤ cE|Rn | + E{kWn kI{kWn k>c} } for all c > 0. Since E|Rn | → 0 and kWn k is uniformly integrable under P , arguing as above gives that EPK |kCn k − kCnPk k| ≤ E{IK kCn − CnPk k} −→ 0. P (K) Therefore, kCn k → N stably under PK . Finally, fix H ∈ B ∞ , P (H) > 0 and a bounded continuous function φ : R → R. Then P (H ∩ K) = P (H ∩ {f ≤ k}) > 0 for k sufficiently large and P (H)|E(φ(kCn k)|H) − E(N (φ)|H)| ≤ 2 sup |φ|P (f > k) + |E(φ(kCn k)|H ∩ K) − E(N (φ)|H ∩ K)|. Since E(φ(kCn k)|H ∩ K) → E(N (φ)|H ∩ K) as n → ∞ and P (f > k) → 0 as k → ∞, this concludes the proof.  Next, we deal with the particular case Q = Q0 , where Q0 is a Ferguson–Dirichlet law on (S ∞ , B ∞ ). If P ≪ Q0 with a density satisfying a certain condition, the convergence rate of kµn − an k can be remarkably improved. Theorem 2. Suppose D is countably determined and supn EQ0 kWn k2 < ∞. Then, √ nkCn k = nkµn − an k converges a.s., provided P ≪ Q0 and   dP 1 2 2 for some version f of . EQ0 (f ) − EQ0 {EQ0 (f |Gn ) } = O n dQ0 Proof. Let Dn (B) = mined) and √ nCn (B). Then, kDn k is Gn -measurable (as D is countably deter- E(kDn+1 k|Gn ) = E sup n+1 X B∈D i=1 ≥ sup E B∈D = sup n X B∈D i=1 n+1 X i=1 IB (Xi ) − (n + 1)E(µ(B)|Gn+1 ) Gn IB (Xi )|Gn ! ! − (n + 1)E(µ(B)|Gn ) IB (Xi ) − nE(µ(B)|Gn ) = kDn k a.s. 1358 Berti, Crimaldi, Pratelli and Rigo Since kDn k is a Gn -submartingale, it suffices to prove that supn EkDn k < ∞. Let Un = f − E0 (f |Gn ), where E0 stands for EQ0 . By assumption, there exist c1 , c2 > 0 such that E0 kWn k2 ≤ c1 , nE0 Un2 = n{E0 (f 2 ) − E0 (E0 (f |Gn )2 )} ≤ c2 for all n. As noted in Section 1, since Q0 is a Ferguson–Dirichlet law, there is an α > 0 such that √ √ nkCnQ0 k = n sup |E0 (Wn (B)|Gn )| ≤ α for all n. B∈D n |kWn k|Gn ) Define Mn = E0 (|U and recall that kCn k ≤ kCnQ0 k + Mn , P -a.s.; see the proof E0 (f |Gn ) of Theorem 1. Then, for all n, one obtains √ √ √ EkDn k = nEkCn k ≤ n(EkCnQ0 k + EMn ) ≤ α + nE0 (f Mn ) √ √ p = α + nE0 (|Un |kWn k) ≤ α + n E0 Un2 E0 kWn k2 p √  ≤ α + c1 nE0 Un2 ≤ α + c1 c2 . Finally, we clarify a point raised in Section 1. Remark 3. There is a long list of (countably determined) choices of D such that sup EkWn kk ≤ c(k) n for all k ≥ 1, if X is i.i.d., where c(k) is some universal constant; see, for example, Sections 2.14.1 and 2.14.2 of [21]. Fix one such D, k ≥ 1, and suppose that S is Polish and B is the Borel σ-field. If X is exchangeable, then de Finetti’s theorem yields E(kWn kk |T ) ≤ c(k) a.s. for all n, where T is the tail σ-field of X. Hence, EkWn kk = E{E(kWn kk |T )} ≤ c(k) for all n. This proves inequality (2). 3. Exchangeable data with finite state space When X is exchangeable and S finite, there is some overlap between Theorem 1 and a result of Bernstein and von Mises. 3.1. Connections with the Bernstein–von Mises theorem For each θ in an open set Θ ⊂ Rk , let Pθ be a product probability on (S ∞ , B ∞ ) (that is, X is i.i.d. under Pθ ). Suppose the map θ 7→ Pθ (B) is Borel measurable for fixed B ∈ B ∞ . Given a (prior) probability π on the Borel subsets of Θ, define Z P (B) = Pθ (B)π(dθ), B ∈ B∞. Rate of convergence of predictive distributions 1359 Roughly speaking, the Bernstein–von Mises (BvM) theorem can be stated as follows. Suppose π is absolutely continuous with respect to Lebesgue measure and the statistical model (Pθ : θ ∈ Θ) is suitably “smooth” (we refer to [13] for a detailed exposition of what “smooth” means). For each n, suppose that θ admits a (consistent) maximum likelihood estimator θbn . Further, suppose the prior π possesses the first moment and denote by θn∗ the posterior mean of θ. Then, √ Pθ 0 n(θbn − θn∗ ) −→ 0 for each θ0 ∈ Θ such that the density of π is strictly positive and continuous at θ0 . Actually, the BvM theorem yields much more than asserted; what is reported above is just the corollary connected to this paper. We refer to [13] and [14] for more information and historical notes; see also [18]. Assuming a smooth, finite-dimensional statistical model is fundamental; see, for example, [11]. Indeed, the BvM theorem does not apply when the only information is that X is exchangeable (or even c.i.d.) and P ≪ Q for some reference probability Q. One exception, however, is when S is finite. Let us suppose S = {x1 , . . . , xk , xk+1 }, X is exchangeable, P (X1 = x) > 0 for all x ∈ S and D = B = power set of S. Also, let λ denote Lebesgue measure on Rk and π the probability distribution of θ = (µ{x1 }, . . . , µ{xk }). As noted in Section 1, π ≪ λ if and only if P ≪ Q0 for some choice of Q0 . Since D is finite and X exchangeable under P and Q0 , kWn k is uniformly integrable under P P and Q0 . Thus, Theorem 1 yields kCn k → 0 whenever π ≪ λ. On the other hand, π is the prior distribution for this problem. The underlying statistical model is smooth and finite-dimensional (it is just a multinomial model). Further, for each n, the maximum likelihood estimator and the posterior mean of θ are, respectively, θbn = (µn {x1 }, . . . , µn {xk }), θn∗ = (an {x1 }, . . . , an {xk }). P Thus, the BvM theorem implies that kCn k → 0, provided π ≪ λ and the density of π is continuous on the complement of a π-null set. To sum up, in this particular case, the same conclusions as from Theorem 1 can be drawn from the BvM theorem. Unlike the latter, however, Theorem 1 does not require any conditions on the density of π. 3.2. Some consequences of Theorems 1 and 2 In this subsection, we focus on S = {0, 1}. Thus, D = B = power set of S and λ denotes Lebesgue measure on R. Let N (0, a) denote the one-dimensional Gaussian law with mean 1360 Berti, Crimaldi, Pratelli and Rigo 0 and variance a ≥ 0 (where N (0, 0) = δ0 ). Our first result allows π to have a discrete part. Corollary 4. With S = {0, 1}, let π be the probability distribution of µ{1} and ∆ = {θ ∈ [0, 1] : π{θ} > 0}, A = {ω ∈ S ∞ : µ(ω, {1}) ∈ ∆}. Define the random probability measure N on R as N = (1 − IA )δ0 + IA N (0, µ{1}(1 − µ{1})). If X is exchangeable and π does not have a singular continuous part, then Cn {1} → N stably and kCn k → N ◦ h−1 stably, where h(x) = |x|, x ∈ R, is the modulus function. Proof. By standard arguments, the corollary holds when π(∆) ∈ (0, 1), provided it holds when π(∆) = 0 and π(∆) = 1. Let π(∆) = 0. Then, π ≪ λ as π does not have a singular continuous part, and the corollary follows from Theorem 1. Thus, it can be assumed that π(∆) = 1. Since Cn {0} = −Cn {1}, kCn k = |Cn {1}| and the modulus function is continuous, it suffices to prove that Cn {1} → N stably. Next, exchangeability of X implies that Wn {1} → N (0, µ{1}(1 − µ{1})) stably; see, for example, Theorem 3.1 of [6]. Since π(∆) = 1, we have N = N (0, µ{1}(1 − µ{1})) a.s. Hence, it is enough to show that E|Cn {1} − Wn {1}| → 0. Fix ε > 0 and let Mn = Wn {1}. Since X is exchangeable, Mn is uniformly integrable. Therefore, there exists some c > 0 such that ε sup E(|Mn |I{|Mn |>c} ) < . 4 n Define φ(x) = x if |x| ≤ c, φ(x) = c if x > c, and φ(x) = −c if x < −c. Since Cn {1} = E(Mn |Gn ) a.s., it follows that E|Cn {1} − Wn {1}| ≤ E|E(Mn |Gn ) − E(φ(Mn )|Gn )| + E|E(φ(Mn )|Gn ) − φ(Mn )| + E|φ(Mn ) − Mn | ≤ E|E(φ(Mn )|Gn ) − φ(Mn )| + 4E(|Mn |I{|Mn |>c} ) < E|E(φ(Mn )|Gn ) − φ(Mn )| + ε for all n. √ Write ∆ = {a1 , a2 , . . .} and Mn,j = n(µn {1} − aj ). Since σ(Mn,j ) ⊂ Gn and P (µ{1} ∈ ∆) = π(∆) = 1, one also obtains E|E(φ(Mn )|Gn ) − φ(Mn )| X E|E(φ(Mn,j )I{µ{1}=aj } |Gn ) − φ(Mn,j )I{µ{1}=aj } | = j Rate of convergence of predictive distributions = X j ≤c 1361 E|φ(Mn,j ){P (µ{1} = aj |Gn ) − I{µ{1}=aj } }| m X j=1 E|P (µ{1} = aj |Gn ) − I{µ{1}=aj } | + 2c X j>m π{aj } for all m, n. By the martingale convergence theorem, E|P (µ{1} = aj |Gn ) − I{µ{1}=aj } | → 0 as n → ∞, for each j. Thus, X π{aj } for all m. lim sup E|Cn {1} − Wn {1}| ≤ ε + 2c n j>m Taking the limit as m → ∞ completes the proof.  If π is singular continuous, we conjecture that Cn {1} converges stably to a non-null limit. However, we do not have a proof. In the next result, a real function g on (0, 1) is said to be almost Lipschitz in the case where x 7→ g(x)xa (1 − x)b is Lipschitz on (0, 1) for some reals a, b < 1. Corollary 5. Suppose S = {0, 1}, X is exchangeable and π is the probability distribution √ of µ{1}. If π admits an almost Lipschitz density with respect to λ, then nkCn k converges a.s. to a real random variable. Proof. Let V = µ{1}. By assumption, there exist a, b < 1 and a version g of dπ dλ such that φ(θ) = g(θ)θa (1 − θ)b is Lipschitz on (0, 1). For each u1 , u2 > 0, we can take Q0 such that V has a beta-distribution with parameters u1 , u2 under Q0 . Let Q0 be such that V has a beta-distribution with parameters u1 = 1 − a and u2 = 1 − b under Q0 . Then, for any n ≥ 1 and x1 , . . . , xn ∈ {0, 1}, one obtains P (X1 = x1 , . . . , Xn = xn ) Z 1 = θr (1 − θ)n−r π(dθ) 0 = Z 1 0 =c Z θr−a (1 − θ)n−r−b φ(θ) dθ V r (1 − V )n−r φ(V ) dQ0 , where r = n X xi and c > 0 is a constant. i=1 dP Let h = cφ. Then, h is Lipschitz and f = h(V ) is a version of dQ . 0 Let Vn = E0 (V |Gn ), where E0 stands for EQ0 . Since h is Lipschitz, |f − E0 (f |Gn )| ≤ |h(V ) − h(Vn )| + E0 (|h(V ) − h(Vn )||Gn ) ≤ d|V − Vn | + dE0 (|V − Vn ||Gn ), 1362 Berti, Crimaldi, Pratelli and Rigo where d is the Lipschitz constant of h. Since E0 kCnQ0 k2 ≤ E0 kWn k2 and √ n|V − Vn | = |CnQ0 {1} − Wn {1}| ≤ kCnQ0 k + kWn k, it follows that 2 E0 (f 2 ) − E0 (E0 (f |Gn )2 ) = E0 {(f − E0 (f |Gn )) } ≤ 4d2 E0 {(V − Vn )2 } ≤ 16d2 4d2 E0 {(kCnQ0 k + kWn k)2 } ≤ E0 kWn k2 . n n Since supn E0 kWn k2 < ∞, we have E0 (f 2 ) − E0 (E0 (f |Gn )2 ) = O(1/n). An application of Theorem 2 completes the proof.  Corollaries 4 and 5 deal with S = {0, 1}, but similar results can be proven for any finite S; see also [12] and [19]. 4. Generalized Pólya urns In this section, based on Examples 1.3 and 3.5 of [6], the asymptotic behavior of kCn k is investigated for a certain c.i.d. sequence. Let (Y, BY ) be a measurable space, B+ the Borel σ-field on (0, ∞) and S = Y × (0, ∞), B = BY ⊗ B+ , Xn = (Yn , Zn ), where Yn (ω) = yn , Zn (ω) = zn for all ω = (y1 , z1 , y2 , z2 , . . .) ∈ S ∞ . Given a law P on B ∞ , it is assumed that Pn αP (Y1 ∈ B) + i=1 Zi IB (Yi ) Pn P (Yn+1 ∈ B|Gn ) = α + i=1 Zi P (Zn+1 ∈ C|X1 , . . . , Xn , Yn+1 ) = P (Z1 ∈ C) a.s., n ≥ 1, a.s., n ≥ 0, (4) (5) for some constant α > 0 and all B ∈ BY , C ∈ B+ . Note that (Zn ) is i.i.d. and Zn+1 is independent of (Y1 , Z1 , . . . , Yn , Zn , Yn+1 ) for all n ≥ 0. In real problems, the Zn should be viewed as weights, while the Yn describe the phenomenon of interest. As an example, consider an urn containing white and black balls. At each time n ≥ 1, a ball is drawn and then replaced together with Zn more balls of the same color. Let Yn be the indicator of the event {white ball at time n} and suppose that Zn is chosen according to a fixed distribution on the integers, independently of (Y1 , Z1 , . . . , Yn−1 , Zn−1 , Yn ). The predictive distributions of X are then given by (4)–(5). Also, note that the probability law of (Yn ) is Ferguson–Dirichlet in the case where Zn = 1 for all n. It is not hard to prove that X is c.i.d. We state this fact as a lemma. Lemma 6. The sequence X assessed according to (4)–(5) is c.i.d. Rate of convergence of predictive distributions 1363 Proof. Fix k > n ≥ 0 and A ∈ BY ⊗ B+ . By a monotone class argument, it can be assumed that A = B × C, where B ∈ BY and C ∈ B+ . Further, it can be assumed that k = n + 2. Let n = 0 and G0 be the trivial σ-field. Since X2 ∼ X1 (as is easily seen), E(IB (Y2 )IC (Z2 )|G0 ) = E(IB (Y1 )IC (Z1 )|G0 ) a.s. If n ≥ 1, define Gn∗ = σ(X1 , . . . , Xn , Zn+1 ). Noting that E(IB (Yn+1 )|Gn∗ ) = E(IB (Yn+1 )|Gn ) a.s., one obtains E(IB (Yn+2 )|Gn∗ ) = E{E(IB (Yn+2 )|Gn+1 )|Gn∗ } P αP (Y1 ∈ B) + ni=1 Zi IB (Yi ) + Zn+1 E(IB (Yn+1 )|Gn∗ ) = Pn+1 α + i=1 Zi Pn (α + i=1 Zi )E(IB (Yn+1 )|Gn ) + Zn+1 E(IB (Yn+1 )|Gn ) = . Pn+1 α + i=1 Zi = E(IB (Yn+1 )|Gn ) = E(IB (Yn+1 )|Gn∗ ) a.s. Finally, since Gn ⊂ Gn∗ , the previous equality implies that E(IB (Yn+2 )IC (Zn+2 )|Gn ) = P (Z1 ∈ C)E{E(IB (Yn+2 )|Gn∗ )|Gn } = P (Z1 ∈ C)E{E(IB (Yn+1 )|Gn∗ )|Gn } = E(IB (Yn+1 )IC (Zn+1 )|Gn ) a.s. Therefore, X is c.i.d.  Usually, one is interested in predicting Yn more than Zn . Thus, in the sequel, we focus on P (Yn+1 ∈ B|Gn ). For each B ∈ BY , we write Cn (B) = Cn (B × (0, ∞)), an (B) = an (B × (0, ∞)) = P (Yn+1 ∈ B|Gn ), and so on. In Example 3.5 of [6], assuming EZ12 < ∞, it is shown that 2 Cn (B) → N (0, σB ) 2 stably, where σB = var(Z1 ) µ(B)(1 − µ(B)). (EZ1 )2 Here, we prove that Cn converges stably when regarded as a map Cn : S ∞ → l∞ (D), where l∞ (D) is the space of real bounded functions on D equipped with uniform distance; see Section 1.5 of [21]. In particular, stable convergence of Cn as a random element of l∞ (D) implies stable convergence of kCn k = supB∈D |Cn (B)|. Intuitively, the stable limit of Cn (when it exists) is connected to the Brownian bridge. Let B1 , B2 , . . . be pairwise disjoint elements of BY and D = {Bk × (0, ∞) : k ≥ 1}, T0 = 0, Tk = k X i=1 µ(Bi ). 1364 Berti, Crimaldi, Pratelli and Rigo Also, let G be a standard Brownian bridge process on some probability space (Ω0 , A0 , P0 ). For fixed ω ∈ S ∞ , p var(Z1 ) {G(Tk (ω)) − G(Tk−1 (ω))} L(ω, Bk ) = EZ1 is a real random variable on (Ω0 , A0 , P0 ). Since the Bk are pairwise disjoint and G has continuous paths, L(ω, Bk ) → 0 as k → ∞. It thus makes sense to define M (ω, ·) as the probability distribution of L(ω) = (L(ω, B1 ), L(ω, B2 ), . . .), that is, M (ω, A) = P0 (L(ω) ∈ A) for each Borel set A ⊂ l∞ (D). Similarly, let N (ω, ·) be the probability distribution of supk≥1 |L(ω, Bk )|, that is,   N (ω, A) = P0 sup |L(ω, Bk )| ∈ A k≥1 for each Borel set A ⊂ R. Theorem 7. Suppose B1 , B2 , . . . ∈ BY are pairwise disjoint and D, M , N are defined as above. Let X be assessed according to (4)–(5) with a ≤ Z1 ≤ b a.s. for some constants 0 < a < b. Then, s  [  2 sup EkWn k ≤ c P Y1 ∈ Bk (6) n k for some constant c independent of the Bk , and Cn → M stably (in the metric space l∞ (D)). In particular, kCn k → N stably. Let Q1 denote the probability law of a sequence X satisfying (4)–(5) and a ≤ Z1 ≤ b a.s. In view of Theorem 7, Q1 can play the role of Q in Theorem 1. That is, for an arbitrary c.i.d. sequence X with distribution P , one has kCn k → N stably, provided P ≪ Q1 and kWn k is uniformly integrable under P . The condition of pairwise disjoint Bk is actually rather strong. However, it holds in at least two relevant situations: when a single set B is involved, and when S = {x1 , x2 , . . .} is countable and Bk = {xk } for all k. Proof of Theorem 7. This proof involves some simple but long calculations. Accordingly, we provide only a sketch of the proof and refer to [7] for details. Since X is c.i.d., for fixed B ∈ BY , one has an (B) = E(µ(B)|Gn ) a.s. Hence, (an (B) : n ≥ a.s. 1) is a Gn -martingale with an (B) −→ µ(B) and this implies that 2 E{(an+1 (B) − µ(B)) } = E X (aj (B) − aj+1 (B)) j>n 2  = X j>n 2 E{(aj (B) − aj+1 (B)) }. Replacing aj (B) by (4) and usingPthe fact that a ≤ Zi ≤ b a.s. for all i, a long but straightforward calculation yields j>n E{(aj (B) − aj+1 (B))2 } ≤ cn1 P (Y1 ∈ B), where Rate of convergence of predictive distributions 1365 c1 is a constant independent of B. It follows that o X n 2 2 Ekan+1 − µk2 = E sup(an+1 (Bk ) − µ(Bk )) ≤ E{(an+1 (Bk ) − µ(Bk )) } k = XX k j>n k 2 E{(aj (Bk ) − aj+1 (Bk )) } ≤  [  c1 = P Y1 ∈ Bk n c1 X P (Y1 ∈ Bk ) n k as the Bk are pairwise disjoint. k Precisely as above, after some algebra, one obtains s  [  c2 2 P Y1 ∈ Bk Ekµn − an+1 k ≤ n k for some constant c2 independent of B1 , B2 , . . . . Therefore, s  [  EkWn k = nEkµn − µk ≤ 2nEkµn − an+1 k + 2nEkan+1 − µk ≤ c P Y1 ∈ Bk , 2 2 2 2 k where c = 2(c1 + c2 ). This proves inequality (6). It remains to prove that Cn → M stably (in the metric space l∞ (D)). For each m ≥ 1, let Σm be the m × m matrix with elements σk,j = var(Z1 ) (µ(Bk ∩ Bj ) − µ(Bk )µ(Bj )), (EZ1 )2 k, j = 1, . . . , m. By Theorems 1.5.4 and 1.5.6 of [21], for Cn → M stably, it is enough that: (i) (finite-dimensional convergence): (Cn (B1 ), . . . , Cn (Bm )) → Nm (0, Σm ) stably for each m ≥ 1, where Nm (0, Σm ) is the m-dimensional Gaussian law with mean 0 and covariance matrix Σm ; (ii) (asymptotic tightness): for each ε, δ > 0, there exists some m ≥ 1 such that   lim sup P sup |Cn (Br ) − Cn (Bs )| > ε < δ. n r,s>m P Fix m ≥ 1, b1 , . . . , bm ∈ R and define Rn = m k=1 bk IBk (Yn ). Since (Rn : n ≥ 1) is c.i.d., arguing exactly as in Example 3.5 of [6], one obtains m X k=1 bk Cn (Bk ) = Pn i=1 {Ri   X − E(Rn+1 |Gn )} √ −→ N 0, bk bj σk,j n k,j stably. 1366 Berti, Crimaldi, Pratelli and Rigo Since b1 , . . . , bm are arbitrary, (i) holds. To check (ii), given ε, δ > 0, take m such that   2 2  [ ε δ , P Y1 ∈ Br < 4c r>m where c is the constant involved in (6). By what has already been proven,     P sup |Cn (Br ) − Cn (Bs )| > ε ≤ P 2 sup |Cn (Br )| > ε r,s>m r>m o     4 n ≤ P 2E sup |Wn (Br )||Gn > ε ≤ 2 E sup Wn (Br )2 ε r>m r>m s   [ 4c ≤ 2 P Y1 ∈ Br < δ. ε r>m Thus, (ii) holds and this completes the proof.  Acknowledgments This paper benefited from the helpful suggestions of two anonymous referees. References [1] Algoet, P.H. (1992). Universal schemes for prediction, gambling and portfolio selection. Ann. Probab. 20 901–941. MR1159579 [2] Algoet, P.H. (1995). Universal prediction schemes (correction). Ann. Probab. 23 474–478. MR1330780 [3] Berti, P. and Rigo, P. (2002). A uniform limit theorem for predictive distributions. Statist. Probab. Lett. 56 113–120. MR1881164 [4] Berti, P., Pratelli, L. and Rigo, P. (2002). Almost sure uniform convergence of empirical distribution functions. Int. Math. J. 2 1237–1250. MR1939015 [5] Berti, P., Mattei, A. and Rigo, P. (2002). Uniform convergence of empirical and predictive measures. Atti Semin. Mat. Fis. Univ. Modena 50 465–477. MR1958292 [6] Berti, P., Pratelli, L. and Rigo, P. (2004). Limit theorems for a class of identically distributed random variables. Ann. Probab. 32 2029–2052. MR2073184 [7] Berti, P., Crimaldi, I., Pratelli, L. and Rigo, P. (2008). Rate of convergence of predictive distributions for dependent data. Technical report. Available at http://amsacta.cib.unibo.it/2538/1/be-cri-pra-ri-preprint.pdf. [8] Blackwell, D. and Dubins, L.E. (1962). Merging of opinions with increasing information. Ann. Math. Statist. 33 882–886. MR0149577 [9] Crimaldi, I., Letta, G. and Pratelli, L. (2007). A strong form of stable convergence. In Seminaire de Probabilites XL. Lecture Notes in Math. 1899 203–225. Berlin: Springer. MR2409006 Rate of convergence of predictive distributions 1367 [10] de Finetti, B. (1937). La prevision: ses lois logiques, ses sources subjectives. Ann. Inst. H. Poincaré 7 1–68. MR1508036 [11] Freedman, D. (1999). On the Bernstein–von Mises theorem with infinite-dimensional parameters. Ann. Statist. 27 1119–1140. MR1740119 [12] Ghosh, J.K., Sinha, B.K. and Joshi, S.N. (1982). Expansions for posterior probability and integrated Bayes risk. In Statistical Decision Theory and Related Topics III 1 403–456. New York: Academic Press. MR0705299 [13] Ghosh, J.K. and Ramamoorthi, R.V. (2003). Bayesian Nonparametrics. New York: Springer. MR1992245 [14] Le Cam, L. and Yang, G.L. (1990). Asymptotics in Statistics: Some Basic Concepts. New York: Springer. MR1066869 [15] Morvai, G. and Weiss, B. (2005). Forward estimation for ergodic time series. Ann. Inst. H. Poincaré Probab. Statist. 41 859–870. MR2165254 [16] Pitman, J. (1996). Some developments of the Blackwell–MacQueen urn scheme. In: Statistics, Probability and Game Theory (T. S. Ferguson, L. S. Shapley and J. B. MacQueen, eds.). IMS Lecture Notes Monogr. Ser. 30 245–267. Hayward, CA: Inst. Math. Statist. MR1481784 [17] Renyi, A. (1963). On stable sequences of events. Sankhyā A 25 293–302. MR0170385 [18] Romanovsky, V. (1931). Sulle probabilita’ “a posteriori”. Giornale dell’Istituto Italiano degli Attuari 4 493–511. [19] Strasser, H. (1977). Improved bounds for equivalence of Bayes and maximum likelihood estimation. Theory Probab. Appl. 22 349–361. MR0440778 [20] Stute, W. (1986). On almost sure convergence of conditional empirical distribution functions. Ann. Probab. 14 891–901. MR0841591 [21] van der Vaart, A. and Wellner, J.A. (1996). Weak Convergence and Empirical Processes. New York: Springer. MR1385671 Received February 2008 and revised December 2008 View publication stats