Statistical ensembles and probability repair

Statistical reasoning in field theory becomes unreliable when three different questions are answered with the same formula. A probability law determines expectations once the law is given. A physical ensemble explains why a particular law models the system. An estimator and its uncertainty describe what finite data can reveal about a target quantity. This lesson separates those layers through one finite canonical ensemble, one dependence example, and one correlated-sample calculation.

Required background. Linear and tensor methods supplies finite sums, quadratic forms, and covariance matrices. If those operations are comfortable, no measure theory or thermodynamic limit is needed to begin.

Probability laws, ensembles, and samples answer different questions

Suppose a finite set of configurations is labeled by $i$, a normalized law is $p_i$, and an observable has value $A_i$. Then

$$\mathbb E_p[A]=\sum_i p_iA_i, \qquad \operatorname{Var}_p(A) =\sum_i p_i\left(A_i-\mathbb E_p[A]\right)^2$$

are probability identities. They are true for any declared normalized law; they do not explain where $p_i$ came from.

A canonical ensemble adds physical content:

$$p_i(\beta)=\frac{e^{-\beta E_i}}{Z(\beta)}, \qquad Z(\beta)=\sum_i e^{-\beta E_i}.$$

This law represents equilibrium with a heat bath at fixed inverse temperature under specified Hamiltonian, volume, conserved quantities, and boundary conditions. The algebra that follows from $p_i(\beta)$ is exact, but the claim that a laboratory system or simulation is described by that law is a model assumption requiring separate support.

Finally, observations $A^{(1)},\ldots,A^{(N)}$ introduce an inference problem. The sample mean

$$\overline A_N=\frac1N\sum_{n=1}^N A^{(n)}$$

is an estimator of an ensemble mean only after its sampling process is specified. Independent equilibrium draws, a correlated stationary Markov chain, and a chain that has not equilibrated can have the same list of recorded values but justify different uncertainty statements.

Keep this classification visible:

Layer	Question	Typical statement
Probability	What follows from the declared law?	$\operatorname{Var}(A)=\mathbb E[A^2]-\mathbb E[A]^2$
Physical ensemble	Why should this law describe the system?	The system is in canonical equilibrium at fixed $\beta$ and volume.
Inference	What does finite, possibly dependent data establish?	$\overline A_N$ estimates $\mathbb E[A]$ with a covariance-aware standard error.

The first statement can be proved algebraically, the second is part of the physical model, and the third needs sampling assumptions. None can substitute for the others.

A finite canonical ensemble gives an exact response identity

Consider three nondegenerate configurations with energies

$$E_0=0, \qquad E_1=\Delta, \qquad E_2=2\Delta, \qquad \Delta>0.$$

Assume canonical equilibrium at inverse temperature $\beta>0$, fixed volume, and $k_{\mathrm B}=1$. These are the physical assumptions. No samples or estimators have entered. Writing $x=e^{-\beta\Delta}$ gives

$$Z=1+x+x^2, \qquad (p_0,p_1,p_2)=\frac{(1,x,x^2)}{Z}.$$

The mean and second moment are

$$\langle E\rangle =\Delta\frac{x+2x^2}{Z}, \qquad \langle E^2\rangle =\Delta^2\frac{x+4x^2}{Z},$$

so

$$\operatorname{Var}(E) =\Delta^2\left[ \frac{x+4x^2}{Z} -\left(\frac{x+2x^2}{Z}\right)^2 \right].$$

For any finite canonical law,

$$\frac{\partial p_i}{\partial\beta} =-\left(E_i-\langle E\rangle\right)p_i.$$

Multiplying by $E_i$ and summing gives

$$-\frac{\partial\langle E\rangle}{\partial\beta} =\operatorname{Var}(E) =\frac{\partial^2\log Z}{\partial\beta^2}.$$

Because $\beta=1/T$ in the chosen units, the heat capacity at fixed external parameters is

$$C=\frac{\partial\langle E\rangle}{\partial T} =\beta^2\operatorname{Var}(E)\geq0.$$

This response–fluctuation identity is exact for the declared finite canonical ensemble. It is not evidence that an arbitrary dataset has equilibrated, and it does not hold unchanged after replacing the canonical law by a different ensemble. The general partition-function method is developed in Partition functions and thermodynamic response and Kardar 2007, chs. 1–3.

Two limits check the arithmetic. As $\beta\Delta\to\infty$, $x\to0$ and the ground state dominates:

$$\langle E\rangle\longrightarrow0, \qquad \operatorname{Var}(E)\longrightarrow0.$$

As $\beta\Delta\to0$, all three configurations become equally weighted:

$$\langle E\rangle\longrightarrow\Delta, \qquad \operatorname{Var}(E)\longrightarrow\frac{2\Delta^2}{3}.$$

These are temperature limits of a fixed finite system. Since $Z$ is a finite sum of positive analytic functions for real $\beta$, it has no finite-system thermodynamic nonanalyticity. A phase transition, spontaneous symmetry breaking, or equivalence of ensembles requires an appropriate infinite-system limit and its hypotheses.

Zero covariance does not imply independence

Let $X$ be uniform on $\{-1,0,1\}$ and set $Y=X^2$. Then

$$\mathbb E[X]=0, \qquad \mathbb E[Y]=\frac23, \qquad \mathbb E[XY]=\mathbb E[X^3]=0,$$

so

$$\operatorname{Cov}(X,Y) =\mathbb E[XY]-\mathbb E[X]\mathbb E[Y]=0.$$

Nevertheless, $X$ and $Y$ are dependent: $Y$ is determined by $X$, and

$$\mathbb P(Y=0\mid X=0)=1 \quad\text{while}\quad \mathbb P(Y=0)=\frac13.$$

Covariance tests one bilinear moment. Independence requires the entire joint law to factorize, equivalently $\mathbb P(X\in B,Y\in C)=\mathbb P(X\in B)\mathbb P(Y\in C)$ for all relevant events. In a Gaussian model, zero covariance has stronger consequences, but that is an additional distributional assumption, not the definition of independence. See Durrett 2019, § 2.1, pp. 43–49, PDF for the standard independence criteria.

This example is purely probabilistic. It makes no physical ensemble claim and uses no finite-data inference.

Correlated samples change the uncertainty of the mean

Let $A_1,\ldots,A_N$ be a stationary sequence with common mean $\mu$, variance $\sigma^2$, and autocorrelation

$$\rho_k =\frac{\operatorname{Cov}(A_n,A_{n+k})}{\sigma^2}.$$

Stationarity and the common mean imply $\mathbb E[\overline A_N]=\mu$, so correlation alone does not bias this estimator. It does change its variance:

$$\begin{aligned} \operatorname{Var}(\overline A_N) &=\frac1{N^2}\sum_{i,j=1}^N \operatorname{Cov}(A_i,A_j)\\ &=\frac{\sigma^2}{N} \left[ 1+2\sum_{k=1}^{N-1} \left(1-\frac{k}{N}\right)\rho_k \right]. \end{aligned}$$

Define the finite-$N$ integrated autocorrelation time and a variance-equivalent effective sample size by

$$\tau_{\mathrm{int},N} =\frac12+ \sum_{k=1}^{N-1} \left(1-\frac{k}{N}\right)\rho_k, \qquad N_{\mathrm{eff}}=\frac{N}{2\tau_{\mathrm{int},N}}.$$

Then

$$\operatorname{SE}(\overline A_N) =\frac{\sigma}{\sqrt{N_{\mathrm{eff}}}}.$$

For the covariance model $\rho_k=\rho^k$ with $0\leq\rho<1$,

$$N_{\mathrm{eff}} =\frac{N}{ 1+2\sum_{k=1}^{N-1}(1-k/N)\rho^k} \sim N\frac{1-\rho}{1+\rho}$$

at fixed $\rho$ and large $N$. With $N=100$ and $\rho=0.8$, the exact finite sum gives $N_{\mathrm{eff}}\approx11.6$. The correlation-aware standard error is therefore about $0.293\sigma$, almost three times the independent-sample value $0.1\sigma$.

Effective sample size is a summary of the variance for this estimator and observable; it is not a literal count of independent configurations. Negative correlations can even produce $N_{\mathrm{eff}}>N$. When $\sigma^2$ and the autocorrelation tail are estimated from the same finite series, window or block choices add uncertainty. Stationarity, equilibration, and a suitable central limit theorem must be justified separately before turning a standard error into a confidence interval. Autocorrelation sums, window selection, and error-estimate stability are treated in Wolff 2004/2006, §§ 2 and 3.1–3.3, pp. 4–13.

Do not confuse these quantities:

• $\sigma^2$ is the ensemble fluctuation of one draw;
• $\operatorname{Var}(\overline A_N)$ is the sampling variance of an estimator;
• a confidence interval needs a sampling-distribution argument as well as an estimated standard error; and
• discretization, finite-volume, model, and equilibration errors are distinct possible biases, not contributions automatically captured by the standard error.

Finite-data and thermodynamic limits are different

Several limits can appear in one field-theory calculation. State which one is being taken and what remains fixed.

Limit	Question answered	What it does not establish by itself
$N\to\infty$ samples	Does an estimator converge under the sampling law?	That the physical ensemble or simulation algorithm is correct
Volume $V\to\infty$	Does a thermodynamic state, phase, or nonanalyticity emerge?	A central limit theorem for the recorded samples
Lattice spacing $a\to0$	Does a continuum observable exist after tuning and renormalization?	Ensemble equivalence or equilibration
Source $h\to0$	Does a response or order parameter persist?	That the order of $h\to0$ and $V\to\infty$ is interchangeable

For the finite three-state example, a microcanonical law at exact energy $E=\Delta$ places all probability on the middle configuration and gives zero energy variance. The canonical law mixes all three energies and generally has nonzero variance. Relating those descriptions requires additional degrees of freedom, a declared thermodynamic limit, and appropriate regularity or concavity conditions. Use Thermodynamic limits, phases, and ensemble equivalence before making an equivalence or phase-transition claim.

Exercises

Normalize first, then condition

Three outcomes $a,b,c$ have weights $1,2,3$, respectively. Let the observable $X$ take values $0,1,2$ on those outcomes. Normalize the law, compute $\mathbb E[X]$ and $\operatorname{Var}(X)$, and find $\mathbb P(X=2\mid X\geq1)$. Classify the calculation by layer.

Solution

The total weight is $6$, so

$$(p_a,p_b,p_c)=\left(\frac16,\frac13,\frac12\right).$$

The first two moments are

$$\mathbb E[X] =0\cdot\frac16+1\cdot\frac13+2\cdot\frac12 =\frac43,$$

and

$$\mathbb E[X^2] =0+\frac13+4\cdot\frac12 =\frac73.$$

Therefore

$$\operatorname{Var}(X) =\frac73-\left(\frac43\right)^2 =\frac59.$$

The conditioning event has probability $\mathbb P(X\geq1)=1/3+1/2=5/6$, so

$$\mathbb P(X=2\mid X\geq1) =\frac{1/2}{5/6}=\frac35.$$

Every step is a probability identity once the weights and observable are declared. No physical reason for those weights and no sample-based estimator has been supplied.

Generate a mean, variance, and covariance

For finite configurations with energy $E_i$ and another observable $A_i$, set

$$Z(\beta,h)=\sum_i e^{-\beta E_i+hA_i}.$$

Show that derivatives of $\log Z$ generate $\langle A\rangle$, $\operatorname{Var}(A)$, and $\operatorname{Cov}(E,A)$. Identify the physical assumption and the probability identities.

Solution

The normalized law is

$$p_i(\beta,h) =\frac{e^{-\beta E_i+hA_i}}{Z(\beta,h)}.$$

Differentiating once gives

$$\frac{\partial\log Z}{\partial h} =\frac1Z\sum_i A_i e^{-\beta E_i+hA_i} =\langle A\rangle.$$

A second $h$ derivative differentiates both the numerator and the normalization:

$$\frac{\partial^2\log Z}{\partial h^2} =\langle A^2\rangle-\langle A\rangle^2 =\operatorname{Var}(A).$$

Similarly,

$$-\frac{\partial^2\log Z}{\partial\beta\,\partial h} =\langle EA\rangle-\langle E\rangle\langle A\rangle =\operatorname{Cov}(E,A).$$

Choosing the Boltzmann form as a model of canonical equilibrium is the physical ensemble assumption. Once that normalized law is accepted, the derivative relations are probability identities. No inference choice appears because no finite sample has been introduced.

Retain every covariance term

Let $N=3$, $\operatorname{Var}(A_i)=\sigma^2$, and $\operatorname{Cov}(A_i,A_j)=\sigma^2(1/2)^{\lvert i-j\rvert}$. Compute $\operatorname{Var}(\overline A_3)$, $N_{\mathrm{eff}}$, and the ratio of the correct standard error to the independent-sample standard error.

Solution

There are three diagonal terms, four ordered pairs at lag one, and two ordered pairs at lag two. Hence

$$\begin{aligned} \operatorname{Var}(\overline A_3) &=\frac{\sigma^2}{9} \left[3+4\left(\frac12\right) +2\left(\frac14\right)\right]\\ &=\frac{11}{18}\sigma^2. \end{aligned}$$

Matching this to $\sigma^2/N_{\mathrm{eff}}$ gives

$$N_{\mathrm{eff}}=\frac{18}{11}\approx1.64,$$

well below the three recorded observations. The independent formula would give variance $\sigma^2/3$, so the standard-error ratio is

$$\frac{\sqrt{11/18}\,\sigma}{\sigma/\sqrt3} =\sqrt{\frac{11}{6}} \approx1.35.$$

Thus the correct standard error is about $35\%$ larger than the independent-sample value for this finite covariance model. This conclusion assumes the stated stationary covariance law is known; estimating it from three data points would not be credible.

Return when the three layers stay separate

You are ready to return when you can do all three of the following without looking back:

1. normalize a finite law, compute moments or conditional probabilities, and test independence from the joint law rather than covariance alone;
2. state the physical assumptions selecting an ensemble and derive one response–fluctuation identity without treating it as evidence of equilibration; and
3. name an estimand, retain covariance terms in its uncertainty, and state which theorem or sampling assumptions would be needed for an interval or large-$N$ claim.

Re-run the statistical and probability diagnostic with a different finite spectrum and correlation parameter. If all three tasks are Demonstrated, return to the page that sent you here. If correlated uncertainty is now clear but precision, convergence, testing, or reproducibility is still blocking the work, continue to Numerical and reproducibility repair and then repeat the computational and evidence diagnostic.

References

• Durrett, Rick. Probability: Theory and Examples. 5th ed. Cambridge University Press, 2019. Official open PDF.
• Kardar, Mehran. Statistical Physics of Particles. Cambridge University Press, 2007. DOI.
• Wolff, Ulli. “Monte Carlo Errors with Less Errors.” Computer Physics Communications 156, no. 2 (2004): 143–153. DOI; arXiv:hep-lat/0306017.