Sampling Bias, Precision, and Independence

Playback speed

Swipe, scroll or use ← →

Separate bias from variation

Precision describes random fluctuation, while bias describes systematic displacement. Keeping them separate prevents a stable estimate from being mistaken for an accurate one. Sampling variation is random change in an estimate across repeated samples, whereas bias is systematic displacement from the target. Precision concerns random variation in an estimate, whereas systematic sampling error confounds the effect of interest with another explanatory factor.

Repeated estimates	Centred on the target	Displaced from the target
Tightly clustered	Precise and accurate	Precise and wrong every time
Widely scattered	Noisy, but right on average	Noisy and wrong on average

A scale that always reads two kilograms high sits in the top right cell: repeating the weighing does not remove that offset. Random fluctuation between readings is different, because averaging independent readings reduces its effect. Check calibration as well as repeatability.

1 / 16

Separate bias from variation

A factory measures ten items from each of five production runs. Treating all fifty items as independent exaggerates information because items within a run share conditions. Randomizing measurement order can protect against instrument drift, while stratifying by run can improve coverage and precision. Increasing independent sample size generally improves precision but does not automatically remove selection or measurement bias. Independent replication adds information about variation; repeated readings of the same unit do not create the same amount of independent information. Sampling variation is the random change in an estimate from one sample to the next, while bias is a systematic shift away from the target.

2 / 16
Quick check

Across repeated samples an estimate fluctuates around a shifted value. Which terms describe the fluctuation and the shift?
1. ABias is random movement across repeated samples and precision is systematic displacement, so a narrow interval diagnoses low bias even under undercoverage.
  
  The definitions are reversed, and interval width does not measure systematic error.
2. BSampling variation is random change in an estimate across repeated samples, whereas bias is systematic displacement from the target.
  
  Random fluctuation and systematic displacement require different diagnoses and cannot be summarized by one notion of error.
3. CRandom variation and bias are the same error source; both can always be removed simply by increasing the number of rows from the existing frame.
  
  More independent data can reduce random variation but not systematic selection or measurement bias.
3 / 16
Recognize independent information

A smaller standard error represents a more precise estimate, but it does not by itself demonstrate low bias. Increasing independent sample size generally improves precision but does not automatically remove selection or measurement bias. Measuring the same person five times does not give the same independent information as measuring five different people. Within-unit measurements share sources of variation and are not equivalent to independent units.

4 / 16
Recognize independent information

Map the source of uncertainty: inherent variation, measurement error, selection, clustering, time order, and sample size. Use genuine independent replication as the basis for uncertainty and address systematic error through design rather than a smaller standard error. Randomization can protect against systematic order effects by preventing treatment or condition from being aligned with time. Stratification can improve representation and sampling efficiency when strata capture important, known sources of variation. A smaller standard error means a more precise estimate, but it does not show whether the estimate is biased.

5 / 16
Quick check

A reported standard error is very small. What does that establish, and what does it leave unresolved?
1. AA smaller standard error shows a more precise estimate, but on its own it does not demonstrate low bias.
  
  Standard error describes how much the estimate moves across repeated samples, so it leaves systematic displacement unresolved.
2. BA smaller standard error proves the estimator is centered on the target, even when the sampling frame leaves out a whole shift.
  
  An estimate can repeat closely and still sit away from the target when the frame excludes part of the population.
3. CA smaller standard error means the raw observations have little variability.
  
  Spread among the observations and uncertainty in an estimate are different quantities.
6 / 16
Design for precision

Independent replication adds information about variation; repeated readings of the same unit do not create the same amount of independent information. Precision depends on inherent variability, measurement error, independent replication, and the efficiency of the sampling scheme. A common error is to assume that a larger sample automatically corrects any sampling-frame defect. More observations from the same defective frame can leave systematic bias unchanged.

7 / 16
Design for precision

Duplicating measurements or sampling more heavily from the same biased mechanism may produce many rows without comparable gains in information. Precision cannot certify that the estimate is centered on the correct target. When rows inside a group resemble each other, counting each row as fresh information overstates the real sample size. A defensible report distinguishes accuracy concerns caused by bias from precision concerns caused by random variation.

8 / 16
Quick check

A laboratory measures the same specimen ten times. Why is this not equivalent to ten independently sampled specimens?
1. ATen repeat readings of one item add exactly as much information about between-item variation as one reading from ten independently produced items.
  
  Repeat readings share item-level variation and cannot replace independent units.
2. BRepeated readings never add information of any kind, even about measurement repeatability.
  
  They can characterize measurement error, though not independent between-unit variation.
3. CIndependent replication adds information about variation; repeated readings of the same unit do not create the same amount of independent information.
  
  New independent units reveal between-unit variation; extra readings of one unit largely reuse the same unit-level information.
9 / 16
Keep your progress in the app

That’s 3 of 6 quick checks. In the app they stay answered, and every lesson remembers where you left off.

Keep reading here
Control order and strata

Randomization can protect against systematic order effects by preventing treatment or condition from being aligned with time. Stratification can improve representation and sampling efficiency when strata capture important, known sources of variation. Pseudo-precision results from claiming that randomization guarantees a representative sample from the target population. Randomization addresses assignment or order effects but cannot repair every coverage problem.

10 / 16
Quick check

Treatments would otherwise be applied in a fixed time order. How does randomization help?
1. ARandomization removes every kind of selection bias and guarantees that the sampled factories represent the complete target population under study.
  
  Random order or assignment does not repair frame coverage.
2. BRandomization can protect against systematic order effects by preventing treatment or condition from being aligned with time.
  
  Random order breaks a systematic link between condition and time, protecting the comparison from order-related confounding.
3. CRandomization cannot protect against time trends, because treatment order has no connection to systematic measurement drift.
  
  Breaking alignment between condition and time is precisely how random order protects the comparison.
11 / 16
Avoid pseudo-replication

Positive dependence within clusters reduces effective information compared with the same number of independent observations. Duplicating measurements or sampling more heavily from the same biased mechanism may produce many rows without comparable gains in information. Precision cannot certify that the estimate is centered on the correct target. Randomization addresses assignment or order effects but cannot repair every coverage problem. Positive within-cluster association makes observations partly redundant, reducing information relative to an independent sample of equal size.

12 / 16
Quick check

A survey records 50 pupils in each of 10 schools, with positive similarity inside schools, and reports precision as if all 500 rows were independent. What happens to the effective information?
1. AUse 500 as the independent sample size because every pupil is a different person.
  
  Different people can still share school-level conditions, so clustering remains in their outcomes.
2. BIncrease effective information above 500 because similar pupils corroborate one another.
  
  Similarity within schools duplicates shared information and cannot create more independent evidence than rows.
3. CPositive dependence within clusters reduces effective information compared with the same number of independent observations.
  
  Positive within-cluster association makes observations partly redundant, reducing information relative to an independent sample of equal size.
13 / 16
Key takeaways

A defensible report distinguishes accuracy concerns caused by bias from precision concerns caused by random variation. Map the source of uncertainty: inherent variation, measurement error, selection, clustering, time order, and sample size. Use genuine independent replication as the basis for uncertainty and address systematic error through design rather than a smaller standard error. Standard error measures random precision, not systematic accuracy.

14 / 16
Key takeaways
Sampling Bias, Precision, and Independence review lens:
- Bias: Sampling variation is random change in an estimate across repeated samples, whereas bias is systematic displacement from the target.
- Replication: Increasing independent sample size generally improves precision but does not automatically remove selection or measurement bias.
- Design: Randomization can protect against systematic order effects by preventing treatment or condition from being aligned with time.
- Disclosure: A defensible report distinguishes accuracy concerns caused by bias from precision concerns caused by random variation.
15 / 16
Quick check

A voluntary online poll has 100,000 responses and a tiny standard error, but its frame excludes offline workers and its wording may bias answers. What distinction must the report make?
1. AA defensible report distinguishes accuracy concerns caused by bias from precision concerns caused by random variation.
  
  Separating the two tells readers whether the main threat is systematic inaccuracy, random imprecision, or both.
2. BA defensible report combines bias and precision into a single accuracy score.
  
  The two threats have different causes and remedies.
3. CIt reports only standard error because systematic error is already represented in that number and no separate assessment of bias or coverage is necessary.
  
  Standard error omits systematic displacement from the target.
16 / 16
6 quick checks · then the test

In the app, finishing the quick checks opens this lesson’s 10-question test, and the ones you miss come back exactly when you’re about to forget them.

Take this lesson’s test →

← Previous

Next card →

The whole course, on your phone

Lessons you can read, audio you can listen to on the way to work, and practice that remembers what you got wrong.

“This app is the ‘One Lake’ for preparing the certification exams. Really worth it.”

Abdelhak Y. Google Play review

Sampling Bias, Precision, and Independence

Separate bias from variation

Recognize independent information

Design for precision

Keep your progress in the app

Control order and strata

Avoid pseudo-replication

Key takeaways

6 quick checks · then the test

The whole course, on your phone