Skip to content

Random sample

Distinguish an iid random sample from finite-population sampling without replacement, and see why random selection need not produce a representative realised sample.

A random sample, in the usual mathematical-statistics sense, is a collection X1,…,XnX_1,\ldots,X_n of mutually independent random variables with the same probability distribution. This is the iid model: independent and identically distributed. It lets an analyst describe how an estimate would change if the data collection were repeated.
In survey sampling, the phrase also has a broader meaning: observations selected by a specified random mechanism. A simple random sample without replacement from a finite population is a legitimate probability sample, but its draws are generally dependent. Before using a sampling formula, identify which meaning and which design apply.

What the iid definition promises

Let FF be the common cumulative distribution function. For an iid sample,
P(X1≤x1,…,Xn≤xn)=∏i=1nF(xi).\mathbb{P}(X_1\le x_1,\ldots,X_n\le x_n) =\prod_{i=1}^{n}F(x_i).
There are two separate requirements. Identically distributed means every observation has the same distribution; it does not mean the realised values are equal. Independent means learning the other observations does not change an observation's distribution. Equal individual distributions alone do not establish independence.

The capital letters describe random variables before observation; x1,…,xnx_1,\ldots,x_n denote the realised data. The definition requires neither normality nor a particular sample size. Finite moments are extra assumptions needed for the mean and variance formulas below.

For an iid sample with finite population mean μ\mu and variance σ2\sigma^2, the sample mean satisfies

Xˉ=1n∑i=1nXi,E[Xˉ]=μ,Var(Xˉ)=σ2n.\bar X=\frac1n\sum_{i=1}^{n}X_i, \qquad \mathbb{E}[\bar X]=\mu, \qquad \text{Var}(\bar X)=\frac{\sigma^2}{n}.
The first equality about expectation comes from linearity. The variance result uses independence to eliminate cross-covariances. Its square root, σ/n\sigma/\sqrt n, is the standard error of the mean when the population standard deviation is known. It measures repeated-sample uncertainty, not the dispersion of individual trades.

A two-trade benchmark with an imperfect realised sample

Suppose a researcher is testing a sampler against a deliberately simple synthetic P&L model. Each fixed-size trade produces USD 0 or USD 10, each with probability 1/21/2. The target is the expected P&L per trade under this specified model, USD 5. The population variance is 25 USD225\ \text{USD}^2.

Generate two observations independently from that distribution. Equivalently, draw twice from two labelled tickets carrying 0 and 10, replacing the ticket and randomising independently each time. There are four equally likely ordered outcomes:

Ordered P&Ls, USDProbabilitySample mean, USD per trade
(0,0)(0,0)1/41/40
(0,10)(0,10)1/41/45
(10,0)(10,0)1/41/45
(10,10)(10,10)1/41/410
The sampling distribution of the mean therefore has probabilities 1/4,1/2,1/41/4,1/2,1/4 at 0, 5 and 10. Directly,
E[Xˉ]=14(0)+12(5)+14(10)=5 USD,Var(Xˉ)=14(0−5)2+12(5−5)2+14(10−5)2=12.5 USD2.\begin{aligned} \mathbb{E}[\bar X]&=\tfrac14(0)+\tfrac12(5)+\tfrac14(10)=5\ \text{USD},\\ \text{Var}(\bar X)&=\tfrac14(0-5)^2+\tfrac12(5-5)^2+\tfrac14(10-5)^2 =12.5\ \text{USD}^2. \end{aligned}
This matches σ2/n=25/2\sigma^2/n=25/2. Yet half of all two-observation samples have a mean different from the target. Random selection does not promise a perfectly representative realised sample. Unbiasedness is a statement about the average estimate over repetitions, not a guarantee about one dataset.
Two synthetic P&L observations each have marginal probabilities one half at USD 0 and USD 10. IID sampling with replacement gives sample means USD 0, USD 5 and USD 10 with probabilities 0.25, 0.50 and 0.25, and variance 12.5 USD squared. Without replacement from two fixed records, the mean is always USD 5 and its variance is zero.
Two synthetic P&L observations each have marginal probabilities one half at USD 0 and USD 10. IID sampling with replacement gives sample means USD 0, USD 5 and USD 10 with probabilities 0.25, 0.50 and 0.25, and variance 12.5 USD squared. Without replacement from two fixed records, the mean is always USD 5 and its variance is zero.
Original exact enumeration, not simulated market data. Both designs have individual mean USD 5 and variance 25 USD squared; their sample-mean distributions differ.

Remove replacement, and the question changes

Now treat those two tickets as a fixed finite population of two trade records, one with P&L USD 0 and one with P&L USD 10. The target is this finite population's average P&L, USD 5 per record. Select both records without replacement, randomising the order.

Only (0,10)(0,10) and (10,0)(10,0) can occur, each with probability 1/21/2. Each draw still has the same marginal distribution as before. But after observing the first value, the second is certain: if X1=0X_1=0, then X2=10X_2=10. These draws are dependent. Their mean is always USD 5, so its sampling variance is zero.

For a finite population of size N>1N>1, a simple random sample without replacement gives every subset of nn distinct units the same probability. Writing the fixed values as y1,…,yNy_1,\ldots,y_N, define
μN=1N∑j=1Nyj,σN2=1N∑j=1N(yj−μN)2.\mu_N=\frac1N\sum_{j=1}^{N}y_j, \qquad \sigma_N^2=\frac1N\sum_{j=1}^{N}(y_j-\mu_N)^2.

Under that design,

E[Xˉ]=μN,Var(Xˉ)=σN2nN−nN−1.\mathbb{E}[\bar X]=\mu_N, \qquad \text{Var}(\bar X)=\frac{\sigma_N^2}{n}\frac{N-n}{N-1}.
The variance convention here uses divisor NN. The last factor is the finite-population adjustment to the iid variance. When n=Nn=N, the entire finite population has been observed and its mean has no sampling uncertainty. When n=1n=1, the variance is σN2\sigma_N^2. If nn is small relative to NN, the adjustment is close to one, which can justify an approximation for this variance; it does not make the draws exactly independent.
A census of historical trade records removes sampling error about those records. It does not remove uncertainty about future trades or prove that the historical model will persist.

What a return dataset needs before calling it iid

A file of consecutive daily returns is a sample of observations, but calendar order does not establish the iid model. Dependence, changing volatility or a changing return distribution can invalidate its assumptions. Shuffling the rows changes their presentation; it does not establish the iid model or justify removing covariance terms.

With finite second moments, the general identity is

Var(Xˉ)=1n2(∑i=1nVar(Xi)+2∑i<jCov(Xi,Xj)).\text{Var}(\bar X)=\frac1{n^2} \left(\sum_{i=1}^{n}\text{Var}(X_i) +2\sum_{i<j}\text{Cov}(X_i,X_j)\right).
The covariance terms explain why plugging a daily standard deviation into σ/n\sigma/\sqrt n can misstate uncertainty. Even zero pairwise covariances do not, by themselves, prove independence.
Finally, specify the population before choosing observations. Randomly sampling from a list containing only surviving funds cannot recover funds missing from that list. Randomisation operates on the sampling frame supplied to it; it does not automatically repair sampling bias from an incomplete frame or selective measurement.
For a first pass at sample summaries and their assumptions, see Basic Statistics and Regression (public preview; full lesson free with a signed-in account). Populations, Samples, and Sampling Distributions develops the iid mean, variance and normal-family results in detail (public preview; full lesson requires membership).

Sources

  • Casella, G. and Berger, R. L., Statistical Inference, 2nd ed., §5.1, especially Definition 5.1.1 and the discussion of finite-population sampling; §5.2 for sample-mean properties.
  • Rice, J. A., Mathematical Statistics and Data Analysis, 3rd ed., §§7.2–7.3: finite populations, simple random sampling and the sample mean's expectation and variance.
  • Shao, J., University of Wisconsin–Madison, Stat 609 Lecture 17: Population, Random Sample, and Statistics, Definition 5.1.1 and Theorem 5.2.6.
  • Statistics Canada, Probability sampling, especially simple random sampling with and without replacement.
  • Stark, P. B., University of California, Berkeley, Standard Error, section on the sum and mean of a simple random sample.
Last updated
  • sampling
  • iid
  • finite population
  • sampling distribution
  • statistical inference
Keep connecting the ideas
← Back to the glossaryBack to top ↑