Diagnose statistical and probability readiness

This untimed diagnostic asks whether you can keep three layers of reasoning separate: identities that follow from a probability law, physical assumptions that select an ensemble, and inference choices made when estimating a quantity from finite data. The work matters more than speed. Use notes, a calculator, or a short script, but show the law, assumptions, estimator, and uncertainty model before using a tool to check the arithmetic.

Complete the tasks in any accessible format that makes the equations and reasoning inspectable. Judge each task independently as Demonstrated, Uncertain, or Not yet demonstrated. These descriptions apply only to the work shown here; they are not grades, percentages, or judgments about the person doing it.

Probability laws, conditioning, and independence

For $-1\leq\varepsilon\leq1$, let the joint law of binary random variables $X,Y\in\{0,1\}$ be

$X$	$Y$	$p_\varepsilon(X,Y)$
0	0	$(1+\varepsilon)/4$
0	1	$(1-\varepsilon)/4$
1	0	$(1-\varepsilon)/4$
1	1	$(1+\varepsilon)/4$

1. Explain why the stated range of $\varepsilon$ makes this a probability law, and verify normalization.
2. Find both marginal laws and compute $\mathbb E[X]$, $\mathbb E[Y]$, $\mathbb E[XY]$, and $\operatorname{Cov}(X,Y)$.
3. For $\varepsilon=0.6$, compute $\mathbb P(Y=1\mid X=1)$.
4. Determine for which value of $\varepsilon$ the variables are independent. Use the definition of independence, not covariance alone.
5. Explain why this family does not prove that zero covariance implies independence for arbitrary random variables.

Answer guide

Every entry is nonnegative precisely when $-1\leq\varepsilon\leq1$, and

$$2\frac{1+\varepsilon}{4}+2\frac{1-\varepsilon}{4}=1.$$

Both marginals are uniform:

$$\mathbb P(X=0)=\mathbb P(X=1) =\mathbb P(Y=0)=\mathbb P(Y=1)=\frac12.$$

Consequently,

$$\mathbb E[X]=\mathbb E[Y]=\frac12, \qquad \mathbb E[XY]=\mathbb P(X=1,Y=1)=\frac{1+\varepsilon}{4},$$

and

$$\operatorname{Cov}(X,Y) =\mathbb E[XY]-\mathbb E[X]\mathbb E[Y] =\frac{\varepsilon}{4}.$$

At $\varepsilon=0.6$,

$$\mathbb P(Y=1\mid X=1) =\frac{\mathbb P(X=1,Y=1)}{\mathbb P(X=1)} =\frac{0.4}{0.5}=0.8.$$

Independence requires $p_\varepsilon(x,y)=p_X(x)p_Y(y)=1/4$ for every pair $(x,y)$, so it holds exactly when $\varepsilon=0$. In this specially constructed binary family, zero covariance and independence happen at the same parameter value. That is not a general theorem: zero covariance constrains one mixed second moment, whereas independence constrains the entire joint law.

Interpret the work on this task as follows:

• Demonstrated: normalization and marginals follow from the table; conditioning uses the correct event probability; and independence is tested against the product of the marginal laws.
• Uncertain: the numerical results are right, but the conditioning event, allowed parameter range, or independence criterion remains implicit.
• Not yet demonstrated: a conditional probability uses the wrong denominator, the table is not checked as a probability law, or covariance is treated as the definition of independence.

A physical ensemble and its fluctuations

Consider a nondegenerate two-level system with energies $E_0=0$ and $E_1=\Delta>0$. For this task, assume that the system is in canonical equilibrium at inverse temperature $\beta>0$.

1. Construct the partition function $Z$, the probabilities $p_0,p_1$, the mean energy, and the energy variance.
2. Derive a fluctuation identity by differentiating $\log Z$ or $\langle E\rangle$ with respect to $\beta$.
3. Check the limits $\beta\Delta\to\infty$ and $\beta\Delta\to0$ and interpret them physically.
4. Sort each step of your solution into one of these categories: physical ensemble assumption, probability identity, or inference choice. If a category is unused, say why.
5. Describe what changes in an exact finite-system microcanonical description at fixed energy. Do not assume equivalence of ensembles.

Answer guide

Canonical equilibrium supplies the Boltzmann law. For the stated spectrum,

$$Z=1+e^{-\beta\Delta}, \qquad p_0=\frac1Z, \qquad p_1=\frac{e^{-\beta\Delta}}{Z} =\frac{1}{1+e^{\beta\Delta}}.$$

Once that probability law has been selected, expectation and variance are probability identities:

$$\langle E\rangle=\Delta p_1, \qquad \operatorname{Var}(E) =\langle E^2\rangle-\langle E\rangle^2 =\Delta^2p_1(1-p_1).$$

Differentiation gives the consistent forms

$$-\frac{\partial\log Z}{\partial\beta}=\langle E\rangle, \qquad \frac{\partial^2\log Z}{\partial\beta^2} =\operatorname{Var}(E), \qquad -\frac{\partial\langle E\rangle}{\partial\beta} =\operatorname{Var}(E).$$

For $\beta\Delta\to\infty$, $p_1\to0$, so both the mean energy and its variance vanish. For $\beta\Delta\to0$, the two states become equally weighted, giving

$$p_1\to\frac12, \qquad \langle E\rangle\to\frac{\Delta}{2}, \qquad \operatorname{Var}(E)\to\frac{\Delta^2}{4}.$$

The canonical equilibrium hypothesis, fixed $\beta$, spectrum, and absence of degeneracy are physical assumptions. Normalization, moments, derivatives, and limits follow mathematically after the law is fixed. No estimator or finite dataset appears, so there is no inference choice in this task.

In an exact microcanonical description, probability is supported only on the fixed-energy shell. For this nondegenerate two-state model, selecting energy $0$ or $\Delta$ selects one state and gives zero energy variance; there is no canonical mixture with Boltzmann weights. Relating microcanonical and canonical answers would require an appropriate many-body or thermodynamic limit and additional hypotheses, none of which are supplied here.

Interpret the work on this task as follows:

• Demonstrated: the canonical assumption is stated separately from the probability calculations; the response and direct variance agree; both limits are interpreted; and the microcanonical contrast respects the finite model.
• Uncertain: the formulas are correct, but equilibrium, fixed temperature, degeneracy, or the distinction between an ensemble and one configuration is left unstated.
• Not yet demonstrated: a partition function is taken to prove equilibrium, $\langle E^2\rangle$ is confused with the variance, or finite-system equivalence of ensembles is asserted without a limiting argument.

Correlated samples, estimators, and uncertainty

Let $X_1,\ldots,X_N$ be a stationary sequence with

$$\mathbb E[X_i]=\mu, \qquad \operatorname{Cov}(X_i,X_j)=\sigma^2\rho^{\lvert i-j\rvert}, \qquad 0\leq\rho<1,$$

and define the sample mean

$$\widehat\mu=\frac1N\sum_{i=1}^N X_i.$$

1. State the estimand and show whether $\widehat\mu$ is biased.
2. Derive its exact variance, retaining every covariance term.
3. Recover the independent-sample result at $\rho=0$.
4. Define a finite-$N$ variance-equivalent effective sample size by $\operatorname{Var}(\widehat\mu)=\sigma^2/N_{\mathrm{eff}}$. Find it exactly and give its large-$N$ form for fixed $\rho$.
5. Distinguish the fluctuation of one draw, the standard error of the estimator, and a confidence interval. State what extra information an interval requires.
6. For a fixed constant $a$, consider the shrinkage estimator $\widetilde\mu=a\widehat\mu$. Find its bias, variance, and mean squared error. Explain why a smaller variance alone does not establish a better estimator.
7. If $\rho$ and $\sigma^2$ are unknown, name a correlation-aware method you could use and state what must be checked or estimated.

Answer guide

The estimand is the stationary mean $\mu$. Linearity of expectation gives $\mathbb E[\widehat\mu]=\mu$, so correlation does not by itself bias this sample mean under the stated common-mean assumption.

Grouping the double covariance sum by lag gives

$$\begin{aligned} \operatorname{Var}(\widehat\mu) &=\frac{1}{N^2}\sum_{i,j=1}^N \operatorname{Cov}(X_i,X_j) \\ &=\frac{\sigma^2}{N^2} \left[N+2\sum_{k=1}^{N-1}(N-k)\rho^k\right]. \end{aligned}$$

At $\rho=0$, this becomes $\sigma^2/N$. Matching the correlated variance to $\sigma^2/N_{\mathrm{eff}}$ gives

$$N_{\mathrm{eff}} =\frac{N^2}{N+2\sum_{k=1}^{N-1}(N-k)\rho^k}.$$

For fixed $0\leq\rho<1$ and large $N$,

$$\operatorname{Var}(\widehat\mu) \sim \frac{\sigma^2}{N}\frac{1+\rho}{1-\rho}, \qquad N_{\mathrm{eff}} \sim N\frac{1-\rho}{1+\rho}.$$

Thus positive correlation increases the estimator’s variance and reduces its variance-equivalent sample size. The single-draw physical fluctuation is $\operatorname{Var}(X_i)=\sigma^2$; the estimator’s standard error is $\sqrt{\operatorname{Var}(\widehat\mu)}$. A confidence interval additionally needs a justified sampling distribution or asymptotic theorem and a defensible estimate of the unknown covariance structure. Covariance stationarity alone does not supply a Gaussian interval.

Writing $V=\operatorname{Var}(\widehat\mu)$, the fixed-$a$ estimator has

$$\operatorname{Bias}(\widetilde\mu)=(a-1)\mu, \qquad \operatorname{Var}(\widetilde\mu)=a^2V,$$

and

$$\operatorname{MSE}(\widetilde\mu) =a^2V+(a-1)^2\mu^2.$$

Shrinking toward zero can reduce variance while introducing bias, so the MSE and scientific target—not variance alone—govern the comparison.

With unknown dependence, one could estimate an autocovariance sum with a declared window, use batch means or blocks long relative to the observed correlation scale, or apply another justified time-series method. The work must address stationarity, the observable-specific correlation tail, the window or block choice, and stability under reasonable changes of that choice. An estimated effective sample size summarizes variance for this estimator and observable; it is not a literal count of independent configurations.

Interpret the work on this task as follows:

• Demonstrated: the estimand and common-mean assumption are explicit; the off-diagonal covariance terms remain; the independent limit is recovered; effective sample size is derived rather than asserted; and bias, variance, MSE, and interval assumptions stay distinct.
• Uncertain: the formulas are plausible, but stationarity, the variance convention, correlation estimation, or the assumptions behind an interval remain implicit.
• Not yet demonstrated: correlated observations receive the independent-sample standard error, estimator uncertainty is confused with the spread of individual draws, or a lower variance is claimed to remove bias.

Read the pattern without a total score

Keep the three task results separate:

probability and conditioning: ...
ensemble and fluctuation: ...
correlated estimation: ...

The work demonstrates the full capability only when all three results are Demonstrated, because each layer is needed to interpret the others. If no task is Not yet demonstrated but at least one is Uncertain, treat the capability evidence as uncertain and make the missing assumption explicit. If any task is Not yet demonstrated, review that layer before relying on it in a later path. Do not add the labels, convert them to points, or average them.

For focused review, use Statistical ensembles and probability repair. It revisits the same three distinctions and ends with a fresh work-based check. After review, change the probability parameter, replace the two-state system with at least three energies, and repeat the estimator analysis with a new correlation model or blocked synthetic series. Reassess only the new work.

Scope and limitations

These tasks test finite probability calculations, one declared equilibrium ensemble, and one stationary covariance model. They do not establish measure-theoretic probability, equilibration of observed data, correctness of a sampling algorithm, a central limit theorem, or equivalence of ensembles. They also do not make statistical readiness a prerequisite for every QFT route; it is recommended preparation where ensembles, Monte Carlo data, or uncertainty estimates matter.

For deeper treatment, continue to Probability Spaces, Random Variables, and Conditional Expectation, Statistical Ensembles and Field Configurations, or Markov Generators, Semigroups, Ergodicity, and Correlated-Sample Error. If this check was preparation for numerical work, continue to Diagnose computational and evidence readiness only after the correlated-estimation task is stable. Otherwise, return to Readiness or to the path that sent you here.

References

• Rick Durrett, Probability: Theory and Examples, fifth edition, Cambridge University Press, 2019, official open PDF.
• Mehran Kardar, Statistical Physics of Particles, Cambridge University Press, 2007, doi:10.1017/CBO9780511815898.
• Gareth O. Roberts and Jeffrey S. Rosenthal, “General State Space Markov Chains and MCMC Algorithms,” Probability Surveys 1 (2004), 20–71, doi:10.1214/154957804100000024.
• Ulli Wolff, “Monte Carlo Errors with Less Errors,” Computer Physics Communications 156 (2004), 143–153, doi:10.1016/S0010-4655(03)00467-3.