Probabilistic Convergence, Laws of Large Numbers, and Central Limit Theorems

An estimator is justified only after its limiting claim has been matched to the right mode of convergence and to a theorem whose hypotheses actually hold. Consistency is usually a statement such as $\widehat\theta_n\xrightarrow{\mathbb P}\theta$, justified by a law of large numbers. Pathwise stabilization calls for almost-sure convergence, while control of a mean error calls for $L^p$ convergence. A Gaussian fluctuation approximation is different: it is a convergence-in-distribution statement for a centered and rescaled error, usually supplied by a central limit theorem. Gallager 2011, slides 9–31 compares these convergence modes and the law-of-large-numbers-to-central-limit progression in a common set of examples.

For independent identically distributed observations, an integrable first moment is enough for the strong law of large numbers, and a finite positive variance is enough for the classical central limit theorem. Those statements do not by themselves give a finite-sample error bound, convergence of moments, or a central limit theorem for correlated Monte Carlo output. This page makes those boundaries explicit and ends with one finite-regulator Euclidean QFT calculation where every probabilistic assumption can be checked.

Required background. Probability Spaces, Random Variables, and Conditional Expectation, supplies probability spaces, expectation, integrability, and independence.

Helpful background. Limits, Completeness, and Modes of Convergence provides the general convergence language used to compare almost-sure, probability, $L^p$, and distributional limits.

Modes of probabilistic convergence

Readers who need a shorter review can use the statistical ensembles and probability repair.

Unless stated otherwise, $X_1,X_2,\ldots,X$ are random vectors on one probability space $(\Omega,\mathcal F,\mathbb P)$, taking values in a finite-dimensional Euclidean space. The norm is $\lVert\cdot\rVert$, and $n\to\infty$ is understood in every limit. We write

$$\begin{aligned} X_n\xrightarrow{\mathrm{a.s.}}X &\quad\text{for almost-sure convergence},\\ X_n\xrightarrow{\mathbb P}X &\quad\text{for convergence in probability},\\ X_n\xrightarrow{L^p}X &\quad\text{for convergence in }L^p,\\ X_n\Rightarrow X &\quad\text{for convergence in distribution}. \end{aligned}$$

The first three notions compare random variables on a common probability space. Convergence in distribution compares their laws, so the variables can even be realized on different probability spaces. In the QFT application, the sampling limit is taken at fixed finite regulator and fixed finite volume. No continuum or thermodynamic limit is interchanged with the sample size limit.

Four notions of probabilistic convergence

Almost-sure convergence

The sequence converges to $X$ almost surely when

$$\mathbb P\!\left( \left\{\omega: \lVert X_n(\omega)-X(\omega)\rVert\longrightarrow0 \right\} \right)=1.$$

Outside one fixed null event, every sample path eventually follows the ordinary pointwise limit. This is the natural conclusion when a theorem is supposed to describe the long-run behavior of almost every repeated experiment.

Convergence in probability

The sequence converges to $X$ in probability when, for every $\varepsilon>0$,

$$\mathbb P\!\left( \lVert X_n-X\rVert>\varepsilon \right)\longrightarrow0.$$

This is the usual mathematical form of consistency. For a parameter $\theta$, the claim $\widehat\theta_n\xrightarrow{\mathbb P}\theta$ says that every fixed error tolerance is violated with a probability tending to zero. It does not give a rate unless the proof supplies one.

Convergence in mean

For $1\leq p<\infty$ and $X_n,X\in L^p(\Omega;\mathbb R^d)$, the sequence converges to $X$ in $L^p$ when

$$\mathbb E\!\left[ \lVert X_n-X\rVert^p \right]\longrightarrow0.$$

Thus $L^1$ convergence controls mean absolute error, and $L^2$ convergence controls mean-square error. Membership of both the sequence and its limit in $L^p$ is part of the definition; requiring only an integrable difference would fail to exclude a shared non-$L^p$ component.

Convergence in distribution

The sequence converges to $X$ in distribution when

$$\mathbb E[f(X_n)]\longrightarrow\mathbb E[f(X)]$$

for every bounded continuous function $f$. Equivalently, the probability laws of $X_n$ converge weakly to the law of $X$. For real-valued variables, this is equivalent to

$$\mathbb P(X_n\leq x)\longrightarrow\mathbb P(X\leq x)$$

at every continuity point $x$ of the distribution function of $X$. Distributional convergence is therefore the right language for a central limit theorem, but it is deliberately insensitive to unbounded test functions such as $f(x)=x$ or $f(x)=x^2$.

What implies what

On a probability space, for $1\leq p<q<\infty$,

$$L^q \quad\Longrightarrow\quad L^p \quad\Longrightarrow\quad \text{probability} \quad\Longrightarrow\quad \text{distribution},$$

and separately

$$\text{almost surely} \quad\Longrightarrow\quad \text{probability}.$$

If the proposed limit is a constant $c$, there is also the useful equivalence

$$X_n\Rightarrow c \quad\Longleftrightarrow\quad X_n\xrightarrow{\mathbb P}c.$$

Here the symbol $c$ on the left denotes the degenerate random variable equal to $c$.

Proof of the moment and probability arrows. Since $\mathbb P(\Omega)=1$, Hölder’s inequality gives

$$\lVert X_n-X\rVert_{L^p} \leq \lVert X_n-X\rVert_{L^q}.$$

For every $\varepsilon>0$, Markov’s inequality then gives

$$\mathbb P\!\left(\lVert X_n-X\rVert>\varepsilon\right) \leq \frac{\mathbb E[\lVert X_n-X\rVert^p]}{\varepsilon^p}.$$

Thus $L^q$ convergence implies $L^p$ convergence, and $L^p$ convergence implies convergence in probability.

Proof of the almost-sure arrow. If $X_n\to X$ almost surely, then $\mathbf 1_{\{\lVert X_n-X\rVert>\varepsilon\}}\to0$ almost surely. The indicators are bounded by $1$, so dominated convergence shows that their expectations, which are the relevant probabilities, tend to zero.

The implication from probability to distribution and the constant-limit converse are standard weak-convergence results. One route to the converse is to apply the Portmanteau theorem to the closed complement of every ball about $c$. Durrett 2019, § 3.2, pp. 116–124, PDF proves these weak-convergence tools.

The missing arrows are genuinely missing

The following examples are not pathologies to be ignored; they identify the extra hypotheses needed in later theorems.

Probability need not imply almost surely. On $([0,1),\mathcal B,\lambda)$, write each $n\geq1$ uniquely as $n=2^k+j$ with $0\leq j<2^k$, and define

$$X_n =\mathbf 1_{[j2^{-k},(j+1)2^{-k})}.$$

Then

$$\mathbb P(X_n=1)=2^{-k}\longrightarrow0,$$

so $X_n\xrightarrow{\mathbb P}0$ and even $X_n\xrightarrow{L^p}0$ for every finite $p$. Yet each $\omega\in[0,1)$ lies in one selected interval at every level $k$, so $X_n(\omega)=1$ infinitely often. There is no almost-sure convergence to zero.

Almost surely need not imply $L^p$. On $(0,1)$ with Lebesgue measure, set

$$Y_n=n^{1/p}\mathbf 1_{(0,1/n)}.$$

For every $\omega>0$, eventually $\omega>1/n$, so $Y_n(\omega)\to0$. Nevertheless,

$$\mathbb E[|Y_n|^p]=1$$

for every $n$. The shrinking exceptional set is offset by a growing spike.

Distribution need not imply probability. Let $X$ take the values $-1$ and $1$ with equal probability, and define $Z_n=-X$ for every $n$. Then $Z_n$ and $X$ have the same law, so $Z_n\Rightarrow X$, but

$$\mathbb P(|Z_n-X|>1)=1.$$

Weak convergence sees the marginal laws, not the coupling between the two variables.

Weak convergence need not preserve expectations. Taking $p=1$ in the spike example gives $Y_n\to0$ almost surely, hence in distribution, while

$$\mathbb E[Y_n]=1 \qquad\text{for every }n.$$

Bounded continuous test functions cannot detect the missing mass in the unbounded first moment.

Moving limits through functions and expectations

Limit theorems become useful for estimators only after one knows which operations preserve their conclusions.

Continuous mapping

Continuous mapping theorem. Suppose $X_n\Rightarrow X$, and let $g:\mathbb R^d\to\mathbb R^m$ be measurable. If the set $D_g$ of discontinuity points of $g$ satisfies

$$\mathbb P(X\in D_g)=0,$$

then

$$g(X_n)\Rightarrow g(X).$$

This is a cited theorem, not proved here; Durrett 2019, § 3.2, pp. 116–124, PDF gives the Portmanteau and continuous-mapping arguments. The condition on $D_g$ cannot simply be deleted. For example, $X_n=1/n$ converges to $0$, but with $g(x)=\mathbf 1_{\{0\}}(x)$ one has $g(X_n)=0$ and $g(0)=1$.

Slutsky’s theorem

Slutsky’s theorem. For real-valued variables, if $X_n\Rightarrow X$ and $Y_n\xrightarrow{\mathbb P}c$ for a constant $c$, then

$$\begin{aligned} X_n+Y_n&\Rightarrow X+c,\\ X_nY_n&\Rightarrow cX, \end{aligned}$$

and, when $c\neq0$,

$$\frac{X_n}{Y_n}\Rightarrow\frac{X}{c}.$$

Independence of $X_n$ and $Y_n$ is not required. The key fact is the joint convergence $(X_n,Y_n)\Rightarrow(X,c)$; continuous mapping then supplies the displayed conclusions. Vector and matrix versions follow by choosing the corresponding continuous map. This is what makes replacement of an unknown standard deviation by a consistent estimate legitimate.

Uniform integrability

Convergence in probability controls the probability of a fixed-size error, not its expectation. A family of random vectors $\{X_n\}$ is uniformly integrable when

$$\lim_{K\to\infty} \sup_n \mathbb E\!\left[ \lVert X_n\rVert \mathbf 1_{\{\lVert X_n\rVert>K\}} \right]=0.$$

The spike example is bounded in $L^1$ but is not uniformly integrable, so an $L^1$ bound alone is insufficient. A convenient sufficient condition is

$$\sup_n\mathbb E[\lVert X_n\rVert^{1+\delta}]<\infty \qquad\text{for some }\delta>0,$$

because the tail in the definition is then bounded by a constant times $K^{-\delta}$.

Vitali convergence theorem. If $X_n\xrightarrow{\mathbb P}X$ and $\{X_n\}$ is uniformly integrable, then $X\in L^1$ and

$$\mathbb E[\lVert X_n-X\rVert]\longrightarrow0.$$

Conversely, $L^1$ convergence implies uniform integrability of the sequence. More generally, if $X_n,X\in L^p$, $X_n\xrightarrow{\mathbb P}X$, and $\{\lVert X_n-X\rVert^p\}$ is uniformly integrable, then $\mathbb E[\lVert X_n-X\rVert^p]\to0$. If instead $X_n\to X$ almost surely and $\lVert X_n\rVert\leq Y$ for a common $Y\in L^p$, dominated convergence gives $X_n\xrightarrow{L^p}X$ directly.

For real random variables, weak convergence plus uniform integrability of $\{|X_n|\}$ is enough to recover convergence of first moments. Without such a tail condition, a CLT or any other weak limit cannot be used to pass an expectation through the limit. Durrett 2019, § 4.6, pp. 244–247, PDF develops uniform integrability and its connection with $L^1$ convergence.

Laws of large numbers

Let $X_1,X_2,\ldots$ be iid real random variables, and define

$$S_n=\sum_{r=1}^nX_r, \qquad \overline X_n=\frac{S_n}{n}.$$

The independence, identical-distribution, and moment assumptions below are part of the theorem. More general laws exist, but none follows merely from the notation $\overline X_n$.

Weak law

Weak law of large numbers. If

$$\mathbb E[|X_1|]<\infty, \qquad \mu=\mathbb E[X_1],$$

then

$$\overline X_n\xrightarrow{\mathbb P}\mu.$$

The integrable iid theorem is cited from Durrett 2019, § 2.2, pp. 56–65, PDF. Here is the complete elementary proof under the stronger assumption $\operatorname{Var}(X_1)=\sigma^2<\infty$. Independence gives

$$\mathbb E[\overline X_n]=\mu, \qquad \operatorname{Var}(\overline X_n)=\frac{\sigma^2}{n}.$$

Chebyshev’s inequality therefore yields, for every $\varepsilon>0$,

$$\mathbb P\!\left( |\overline X_n-\mu|\geq\varepsilon \right) \leq \frac{\sigma^2}{n\varepsilon^2} \longrightarrow0.$$

This proof supplies a conservative probability bound as well as consistency. Finite variance is sufficient for the argument, but it is not necessary for the weak law.

Strong law

Strong law of large numbers. Under the same iid first-moment hypothesis,

$$\mathbb E[|X_1|]<\infty,$$

one has

$$\overline X_n\xrightarrow{\mathrm{a.s.}}\mu.$$

This full-strength result is cited from Durrett 2019, §§ 2.3–2.5, pp. 67–87, PDF. A proof roadmap is as follows: truncate the variables to control rare large values, use summability and the Borel–Cantelli lemmas to show that truncation changes only finitely many terms almost surely, control the centered truncated sums, and then remove the truncation. Each step is substantive; the finite-variance Chebyshev calculation above is not a proof of this almost-sure statement.

The strong law answers a pathwise question and implies the weak law. It still does not specify how rapidly the error decreases or what its rescaled distribution looks like.

Why the assumptions matter

If $X_1$ has a standard Cauchy law, then every average $\overline X_n$ has the same Cauchy law. There is no finite mean $\mu$ for the averages to estimate, and they do not converge in probability to a constant.

Independence also does real work. If $X_r=Z$ for all $r$, with $Z$ a nondegenerate integrable random variable, then

$$\overline X_n=Z$$

for every $n$. Identical marginal distributions alone do not create averaging. Dependent sequences require an ergodic theorem or another law of large numbers with its own hypotheses.

Central limit theorems

A law of large numbers describes the unscaled error $\overline X_n-\mu$. A central limit theorem describes its leading random fluctuation after multiplication by $\sqrt n$.

The iid finite-variance theorem

Classical central limit theorem. Let $X_1,X_2,\ldots$ be iid with

$$\mathbb E[X_1]=\mu, \qquad 0<\operatorname{Var}(X_1)=\sigma^2<\infty.$$

Then

$$\frac{S_n-n\mu}{\sigma\sqrt n} = \frac{\sqrt n(\overline X_n-\mu)}{\sigma} \Rightarrow \mathcal N(0,1).$$

No third moment and no moment-generating function are required. The conclusion concerns the standardized sum, not the law of an individual $X_r$, and it is asymptotic rather than an assertion that the finite-$n$ distribution is exactly Gaussian.

Proof sketch by characteristic functions. Put $Y_r=(X_r-\mu)/\sigma$. Finite second moment gives the expansion

$$\varphi_Y(u) =\mathbb E[e^{iuY_1}] =1-\frac{u^2}{2}+o(u^2) \qquad(u\to0).$$

Independence then gives

$$\begin{aligned} \varphi_{n^{-1/2}\sum_{r=1}^nY_r}(t) &=\left[\varphi_Y\!\left(\frac{t}{\sqrt n}\right)\right]^n\\ &\longrightarrow e^{-t^2/2}. \end{aligned}$$

The limit is the characteristic function of a standard normal law, and the Lévy continuity theorem completes the proof. The expansion and continuity theorem are the key lemmas; their systematic development belongs to Characteristic Functions, Moments, Cumulants, and Generating Functionals. Durrett 2019, § 3.4.1, pp. 143–146, PDF gives the complete theorem and characteristic-function proof.

Independent but non-identical summands

The plural “central limit theorems” matters because the iid theorem is not the only useful form. Let

$$\{X_{n,k}:1\leq k\leq k_n\}, \qquad n=1,2,\ldots,$$

be a triangular array whose variables are independent within each row, centered, and have finite variances. Define

$$s_n^2 =\sum_{k=1}^{k_n}\operatorname{Var}(X_{n,k})>0.$$

Lindeberg–Feller central limit theorem. If, for every $\varepsilon>0$,

$$\frac{1}{s_n^2} \sum_{k=1}^{k_n} \mathbb E\!\left[ X_{n,k}^2 \mathbf 1_{\{|X_{n,k}|>\varepsilon s_n\}} \right] \longrightarrow0,$$

then

$$\frac{1}{s_n}\sum_{k=1}^{k_n}X_{n,k} \Rightarrow\mathcal N(0,1).$$

This is a cited theorem; see Durrett 2019, § 3.4.2, pp. 147–152, PDF. The Lindeberg condition says that no asymptotically significant fraction of the total variance comes from rare terms comparable to the full row scale. It replaces identical distribution, but not row-wise independence. A Lyapunov moment condition is a common stronger sufficient condition.

What controls finite-sample normal accuracy

The classical CLT alone gives no universal value of $n$ at which a Gaussian approximation becomes accurate. Under the additional third-moment assumption

$$\rho=\mathbb E[|X_1-\mu|^3]<\infty,$$

the Berry–Esseen theorem gives a quantitative bound. If $G$ is the standard normal distribution function, then a universal constant $C$ exists such that

$$\sup_{x\in\mathbb R} \left| \mathbb P\!\left( \frac{S_n-n\mu}{\sigma\sqrt n}\leq x \right)-G(x) \right| \leq \frac{C\rho}{\sigma^3\sqrt n}.$$

This controls absolute error between distribution functions. It is not a relative tail bound and does not guarantee that densities are close. Durrett 2019, § 3.4.4, pp. 157–158, PDF states and proves a Berry–Esseen estimate.

Vector observables and studentized errors

Let $Y_1,Y_2,\ldots$ be iid random vectors in $\mathbb R^d$ with

$$\mathbb E[\lVert Y_1\rVert^2]<\infty, \qquad m=\mathbb E[Y_1], \qquad \Sigma=\operatorname{Cov}(Y_1).$$

Then the multivariate central limit theorem states

$$\sqrt n(\overline Y_n-m) \Rightarrow \mathcal N_d(0,\Sigma).$$

The covariance matrix may be singular. A proof sketch follows from the scalar theorem: for every fixed $t\in\mathbb R^d$,

$$t^{\mathsf T}\sqrt n(\overline Y_n-m) \Rightarrow \mathcal N(0,t^{\mathsf T}\Sigma t),$$

and the Cramér–Wold device reconstructs the joint weak limit.

Durrett 2019, § 3.10, pp. 199–201, PDF gives the multivariate convergence criterion and central limit theorem. Moreover,

$$\operatorname{Cov}(\overline Y_n)=\frac{\Sigma}{n}$$

exactly for iid samples.

Return to iid real $X_r$ under the classical CLT assumptions. For $n\geq2$, define the sample variance

$$\widehat\sigma_n^2 =\frac{1}{n-1} \sum_{r=1}^n(X_r-\overline X_n)^2.$$

The identity

$$\widehat\sigma_n^2 =\frac{n}{n-1} \left( \frac1n\sum_{r=1}^nX_r^2-(\overline X_n)^2 \right)$$

and the strong law applied to $X_r$ and $X_r^2$ show that $\widehat\sigma_n^2\to\sigma^2$ almost surely. Let $\widehat\sigma_n$ be the nonnegative square root. Slutsky’s theorem therefore gives the studentized asymptotic limit

$$\frac{\sqrt n(\overline X_n-\mu)}{\widehat\sigma_n} \Rightarrow\mathcal N(0,1).$$

On the event $\{\widehat\sigma_n=0\}$, the ratio may be defined arbitrarily; because $\widehat\sigma_n\xrightarrow{\mathbb P}\sigma>0$, the probability of that event tends to zero. Thus the result justifies replacing $\sigma$ by a consistent sample estimate in the limit. It does not assert a finite-$n$ Student $t$ law unless the additional normal-sample assumptions for that exact result hold.

Sampling estimates in a regulated Euclidean theory

Consider a finite regulator with $M$ real field coordinates, a measurable action $S_E:\mathbb R^M\to\mathbb R$, and a positive, normalizable Euclidean weight

$$\pi(\mathrm d\phi) =\frac{1}{Z}e^{-S_E(\phi)}\,\mathrm d^M\phi, \qquad 0<Z=\int_{\mathbb R^M}e^{-S_E(\phi)}\,\mathrm d^M\phi<\infty.$$

For independent exact draws $\phi^{(1)},\ldots,\phi^{(N)}\sim\pi$ and a measurable scalar observable $\mathcal O$, set

$$\mathcal O_r=\mathcal O(\phi^{(r)}), \qquad \widehat\mu_N=\frac1N\sum_{r=1}^N\mathcal O_r.$$

The theorem to use is now selected by the integrability that the regulated measure actually provides:

• If $\mathbb E_\pi[|\mathcal O|]<\infty$, the strong law gives $\widehat\mu_N\to\mathbb E_\pi[\mathcal O]$ almost surely.
• If also $0<\operatorname{Var}_\pi(\mathcal O)=\sigma_{\mathcal O}^2<\infty$, the classical CLT gives $\sqrt N(\widehat\mu_N-\mathbb E_\pi[\mathcal O]) \Rightarrow\mathcal N(0,\sigma_{\mathcal O}^2)$.
• For a vector of observables with finite second moments, the multivariate theorem carries their full covariance matrix into the Gaussian limit.

These are sampling statements. They do not prove that the regulator may be removed, that the volume may be sent to infinity, or that a formal continuum functional integral exists.

An exact two-mode check

Continue the finite Gaussian model from the prerequisite page,

$$S_E(\phi,\chi) =\frac12(a\phi^2+2b\phi\chi+c\chi^2), \qquad a>0, \quad c>0, \quad \Delta=ac-b^2>0.$$

Under its normalized Euclidean measure, the first coordinate $\Phi$ is Gaussian with mean zero and variance

$$v=\frac{c}{\Delta}.$$

For $N$ independent exact configurations, let $\Phi_r$ denote that coordinate and estimate its two-point moment by

$$\widehat C_N =\frac1N\sum_{r=1}^N\Phi_r^2.$$

The Gaussian moments give

$$\mathbb E[\Phi_r^2]=v, \qquad \operatorname{Var}(\Phi_r^2) =\mathbb E[\Phi_r^4]-v^2 =2v^2.$$

Consequently, the strong law and CLT yield

$$\widehat C_N\xrightarrow{\mathrm{a.s.}}v, \qquad \frac{\sqrt N(\widehat C_N-v)}{\sqrt2\,v} \Rightarrow\mathcal N(0,1).$$

This example also exposes the difference between exact and asymptotic normality. Since $\Phi_r/\sqrt v$ are independent standard normals,

$$\frac{N\widehat C_N}{v} =\sum_{r=1}^N\frac{\Phi_r^2}{v} \sim\chi_N^2.$$

Thus $\widehat C_N$ is not Gaussian at finite $N$; its centered and scaled law only approaches a Gaussian. By contrast, $\sqrt{N/v}\,\overline\Phi_N$ is exactly standard normal for every $N$ because a linear combination of Gaussian variables is Gaussian. The comparison is a useful check that a CLT conclusion should not be stronger than the finite-sample law permits.

Why Markov-chain output needs a different theorem

Practical lattice calculations usually record successive states of a Markov chain rather than independent exact draws. For a real, second-order stationary observable sequence $(\mathcal O_r)_{r\in\mathbb Z}$, sample the terms $\mathcal O_1,\ldots,\mathcal O_N$ and define

$$\gamma(h) =\operatorname{Cov}(\mathcal O_0,\mathcal O_h).$$

Counting all pairs at each lag gives the exact identity

$$\operatorname{Var}(\overline{\mathcal O}_N) =\frac1N \left[ \gamma(0) +2\sum_{h=1}^{N-1} \left(1-\frac{h}{N}\right)\gamma(h) \right].$$

If every nonzero-lag covariance vanishes, this expression reduces to $\gamma(0)/N$; vanishing covariance alone does not establish independence. If the covariance series is absolutely summable, the exact identity and dominated convergence give

$$N\operatorname{Var}(\overline{\mathcal O}_N) \longrightarrow \sigma_{\mathrm{as}}^2 =\gamma(0)+2\sum_{h=1}^{\infty}\gamma(h).$$

If, in addition, theorem-specific moment and dependence hypotheses establish a CLT with this covariance-sum variance, then

$$\sqrt N \left( \overline{\mathcal O}_N-\mathbb E[\mathcal O_0] \right) \Rightarrow \mathcal N(0,\sigma_{\mathrm{as}}^2).$$

The phrase “if a suitable CLT has been proved” is essential. Stationarity and ergodicity alone do not imply a central limit theorem, and covariance summability by itself is not a universal substitute for mixing, regeneration, spectral, or other theorem-specific hypotheses. Wolff 2004/2006, §§ 1–2, pp. 2–6 starts from equilibrated chain output, makes its normal approximation and finite correlation-scale assumptions explicit, and shows how autocorrelations enter the variance.

The mathematical hypotheses for Markov-chain laws and CLTs belong to Markov Generators, Semigroups, Ergodicity, and Correlated-Sample Error. The developed physical treatment of estimator covariance, blocking, jackknife/bootstrap, ratios, and autocorrelation-aware error analysis belongs to Estimators, Covariance, and Resampling.

Finally, the probability model itself must exist. A positive normalized finite-dimensional Euclidean weight can support the claims above. A Lorentzian factor $e^{iS}$, a sign-changing weight, or a merely formal continuum path integral is not a probability measure to which these theorems can be applied directly.

Choosing the conclusion before choosing the theorem

Intended claim	Mathematical target	Typical iid hypothesis	What still needs separate work
Consistency of a sample mean	$\overline X_n\xrightarrow{\mathbb P}\mu$	$\mathbb E[\lvert X_1\rvert]<\infty$	rate and finite-$n$ coverage
Pathwise long-run average	$\overline X_n\xrightarrow{\mathrm{a.s.}}\mu$	iid and $\mathbb E[\lvert X_1\rvert]<\infty$	convergence rate
Mean-square accuracy	$\mathbb E[\lvert\widehat\theta_n-\theta\rvert^2]\to0$	estimator-specific bounds	tail control and bias
Gaussian root-$n$ fluctuations	$\sqrt n(\overline X_n-\mu)\Rightarrow\mathcal N(0,\sigma^2)$	iid and $0<\sigma^2<\infty$	finite-$n$ accuracy
Studentized Gaussian limit	replace $\sigma$ by $\widehat\sigma_n$	CLT plus consistent $\widehat\sigma_n$	exact finite-$n$ law
Correlated-sample Gaussian limit	dependent-sequence CLT	theorem-specific dependence and moment conditions	autocorrelation estimation and diagnostics

This table gives sufficient hypotheses for standard routes, not necessary conditions for every possible model. The decisive habit is to write the desired convergence statement first and then verify the theorem’s assumptions against the actual random variables.

Common pitfalls

Calling every limiting statement “convergence.” Almost-sure, probability, $L^p$, and distributional convergence answer different questions. State the mode explicitly and use only the implications that have been proved.

Reading moment convergence from a CLT. Weak convergence tests bounded continuous functions, whereas moments are unbounded. Add uniform integrability or another justified tail bound before passing expectations through a weak limit.

Treating a CLT as a finite-sample Gaussian identity. A CLT describes a centered, scaled sequence as $n\to\infty$. Berry–Esseen needs an additional third moment, and even that theorem controls distribution functions rather than making the data exactly normal.

Applying iid formulas to a Markov chain. Correlation changes the variance of a mean and can invalidate the iid proof entirely. Establish an appropriate ergodic theorem and dependent-sequence CLT before interpreting a Monte Carlo standard error.

Using a formal QFT weight as a probability law. Positivity, normalization, measurability, and the relevant moments are hypotheses, not notation. Check them at the regulated level and keep regulator limits separate from sampling limits.

Check your understanding

1. Retrieve the implication structure

Suppose $X_n\xrightarrow{L^2}X$. Which of $L^1$, probability, and distributional convergence follow automatically? Does almost-sure convergence follow?

Solution

Because the underlying measure has total mass one,

$$\lVert X_n-X\rVert_{L^1} \leq \lVert X_n-X\rVert_{L^2},$$

so $L^1$ convergence follows. Markov’s inequality then gives convergence in probability, which implies convergence in distribution. Almost-sure convergence does not follow in general; the moving-interval example above converges in every finite $L^p$ while taking the value $1$ infinitely often on every sample path.

2. Diagnose missing hypotheses

For each construction, identify why the usual sample-mean conclusion fails:

1. $X_r=Z$ for every $r$, where $Z$ is integrable and nondegenerate.
2. $X_r$ are iid standard Cauchy variables.

Solution

In the first construction the variables are identically distributed but not independent, and $\overline X_n=Z$ for all $n$. It cannot converge in probability to the constant $\mathbb E[Z]$ unless $Z$ is already degenerate.

In the second construction independence holds, but the first absolute moment does not exist. Stability of the Cauchy law gives $\overline X_n\stackrel{d}=X_1$ for every $n$, so there is no deterministic law-of-large-numbers limit and no finite-variance classical CLT.

3. Extract a finite-sample guarantee

Let $X_r$ be iid with mean $\mu$ and variance $0<\sigma^2<\infty$. For $\varepsilon>0$ and $0<\alpha<1$, find a sufficient sample size for

$$\mathbb P(|\overline X_n-\mu|\geq\varepsilon)\leq\alpha,$$

using only the information stated.

Solution

Chebyshev’s inequality gives

$$\mathbb P(|\overline X_n-\mu|\geq\varepsilon) \leq \frac{\sigma^2}{n\varepsilon^2}.$$

Therefore it is sufficient to choose

$$n\geq \left\lceil \frac{\sigma^2}{\alpha\varepsilon^2} \right\rceil.$$

This bound can be conservative, but unlike the bare CLT it is a valid finite-$n$ statement under only a finite-variance assumption.

4. Transfer the result to a correlated sequence

Suppose a stationary scalar sequence has autocovariance

$$\gamma(h)=\gamma(0)\rho^{|h|}, \qquad |\rho|<1,$$

and suppose separately that a suitable CLT for the sequence has been established. Compute its asymptotic variance for the sample mean.

Solution

The geometric series gives

$$\begin{aligned} \sigma_{\mathrm{as}}^2 &=\gamma(0)+2\sum_{h=1}^{\infty}\gamma(h)\\ &=\gamma(0) \left(1+2\sum_{h=1}^{\infty}\rho^h\right)\\ &=\gamma(0)\frac{1+\rho}{1-\rho}. \end{aligned}$$

Positive $\rho$ inflates the iid variance, while negative $\rho$ reduces it. The covariance calculation does not itself prove the assumed CLT; that requires separate dependence hypotheses.

Synthesis and continuations

Use convergence in probability to state ordinary consistency, almost-sure convergence for pathwise stabilization, $L^p$ convergence for mean-error control, and convergence in distribution for a rescaled fluctuation law. For iid data, integrability supplies both weak and strong laws of large numbers, while finite positive variance supplies the classical central limit theorem. Uniform integrability is the extra bridge needed when unbounded moments must pass through a limit, and dependence requires a different LLN or CLT rather than an iid formula with new notation.

The next page develops the transform language used in the CLT proof: Characteristic Functions, Moments, Cumulants, and Generating Functionals. For correlated stochastic dynamics, continue later to Markov Generators, Semigroups, Ergodicity, and Correlated-Sample Error. For production lattice estimators and uncertainty analysis, continue to Estimators, Covariance, and Resampling.

References

• Rick Durrett, Probability: Theory and Examples, fifth edition, PDF, Cambridge University Press, 2019. §§ 2.2–2.5, pp. 56–87, support the convergence modes and weak and strong laws; § 3.2, pp. 116–124, supports weak convergence, Portmanteau, and continuous mapping; §§ 3.4.1–3.4.2, pp. 143–152, support the iid and Lindeberg–Feller central limit theorems; § 3.4.4, pp. 157–158, supports the Berry–Esseen bound; § 3.10, pp. 199–201, supports multivariate weak convergence and the multivariate CLT; and § 4.6, pp. 244–247, supports uniform integrability and $L^1$ convergence.
• Robert Gallager, “Lecture 3: Laws of Large Numbers, Convergence”, MIT OpenCourseWare 6.262, Spring 2011, slides 9–21. This is the teaching source for the distinction among mean-square, probability, distributional, and almost-sure convergence and for the LLN-to-CLT progression.
• Ulli Wolff, “Monte Carlo Errors with Less Errors”, arXiv:hep-lat/0306017v4, 2006 revision, §§ 1–2, pp. 2–6. These sections derive error estimates for equilibrated Markov-chain output, autocovariance contributions, integrated autocorrelation time, and the explicit assumption behind a Gaussian error approximation.