Material Insights · Understanding the Material World

How science knows: observation, models, measurement and testing: Uncertainty: how sure can a measurement be

All courses / Understanding Science / Module 1: How science knows: observation, models, measurement and testing Part 5 of 6

Uncertainty: how sure can a measurement be

No measurement is exact. A good scientist reports a number together with how far off it could be, and designs the work so that the doubt shrinks. This part separates precision from accuracy, shows how to write and shrink uncertainty, and explains how scientists tell a real effect from luck.

5.1 Every measurement has an error, and the two kinds behave differently

Weigh the same packet of tea five times on a kitchen scale and you may read 252, 249, 251, 250 and 253 grams. The packet did not change. The readings vary because the scale, your reading and the surface are never quite steady. This scatter is called random error. It pushes readings above and below the true value by chance, and it can be reduced by repeating and averaging.

Now suppose the scale was set up with a small offset, so that it reads 6 grams too high on every packet. All your readings are off by the same amount in the same direction. This is systematic error, a bias built into the instrument or the method. Averaging does not remove it. A thousand readings will cluster beautifully around the wrong value.

This difference explains the two words that people often swap. Precision describes how tightly repeated readings cluster, which is the size of the random error. Accuracy describes how close the average is to the true value, which depends on the systematic error. A scale can be precise but inaccurate, as with the offset above, or accurate on average but imprecise, as with a shaky one. The best instrument is both, and the two are checked in different ways. Precision is judged by repeating. Accuracy is judged by comparing with a trusted standard, such as a certified 500 g weight.

Systematic errors are the more dangerous, because nothing in the data warns you. In 2011 a badly connected fibre-optic cable in the OPERA neutrino experiment made particles seem faster than light. The error was found and the result withdrawn. A clock that runs fast, a thermometer in direct sun, a questionnaire with leading wording and a survey that only reaches people with phones are all of this type.

5.2 Reporting a result honestly means giving a range and not too many digits

A measurement is incomplete without its uncertainty. 'The table is 1.50 m long' leaves open whether you can trust the second decimal place. 'The table is 1.50 plus or minus 0.01 m' says that the true length is probably between 1.49 and 1.51. The plus-or-minus figure is not an apology. It is information, and without it one cannot say whether two measurements agree.

Take the packet of tea. The five readings average 251 g, and the individual readings spread by about 2 g. The average of five is better than any single reading. A standard result of statistics says that the uncertainty in the mean shrinks roughly with the square root of the number of readings. With 5 readings the improvement is about 2.2 times, with 100 readings it is 10 times. Taking 100 times the readings gives only 10 times the precision, which is why the search for very high precision is so costly. And it works only for random error. If the scale is biased by 6 grams, 10,000 readings will not help.

Significant figures are a short-hand for the same honesty. If a measurement is good to three digits, write three. Writing 251.0000 g because the calculator displayed it would claim a precision you do not have. When you multiply or divide, the answer cannot be more reliable than the least reliable input. The engineering module Thinking in estimates makes the same point: keep only the digits the data can justify.

In science, whether two numbers disagree depends on their uncertainties. A measured 9.78 plus or minus 0.02 m/s^2 for gravity in your town and a standard 9.81 differ by 0.03, about one and a half times the uncertainty, which is unremarkable. If the uncertainty were 0.002, the same gap would signal that something real, such as latitude or altitude, was going on.

Data Worked estimate: five readings of a packet of tea (grams)
StepResultWorking
Readings252, 249, 251, 250, 253Five weighings
Mean251 g1255 / 5 (calculated)
Spread of a single reading (standard deviation)about 1.6 gSquare root of (1+4+0+1+4) / 4 (calculated)
Uncertainty of the meanabout 0.7 g1.6 / square root of 5 = 1.6 / 2.2 (calculated)
Report251.0 plus or minus 0.7 gOne decimal is enough
Readings for plus or minus 0.1 gabout 250(1.6 / 0.1)^2 = 256 (calculated)

Averaging cures random scatter only. A 6 g offset would remain.

Source: Hewitt, Lyons, Suchocki and Yeh, Conceptual Integrated Science, 2nd edition (Pearson, 2013), ch. 1, 'About Science'
  • Hewitt, Lyons, Suchocki and Yeh, Conceptual Integrated Science, 2nd edition (Pearson, 2013) · ch. 1, 'About Science'

5.3 Statistics separates a real effect from luck, and correlation from cause

A coin lands heads six times in a row. Is it biased? Not necessarily: the chance of six heads is 1 in 64, and in a long evening of flipping you would see it. The same question arises whenever data are noisy. A new medicine seems to help 60 of 100 patients and a dummy helps 52. Is the medicine working, or is that gap the sort of luck you would see anyway? Statistics supplies a way to put a number on it, by asking how often chance alone would produce a gap that large.

A common measure is a count of 'sigma', the number of standard widths of random scatter by which the result sits from what chance expects. Particle physicists demand five sigma, which corresponds to a chance of about 1 in 3.5 million of a false alarm, before they announce a discovery. The LIGO detection of gravitational waves on 14 September 2015, at more than 5.1 sigma, and the Higgs particle in 2012 both met this bar. In medicine and social science the bar is often lower, so more published claims fail to hold up when repeated.

Opinion polls give a feel for how size matters. A properly random poll of n people has an uncertainty of roughly one divided by the square root of n. For 1,000 people that is 1 / 32, about 3 percentage points. To halve the margin you need four times the people. Polls in India's huge, varied electorate show why the sample must be random and not simply large: a million replies from an online form would be biased towards people with internet access.

Last, correlation is not cause. Ice cream sales and drowning rise together, because both rise in summer. A third factor, heat, drives both. Data can show that two things move together. To argue that one causes the other, you need an experiment, a natural experiment or a mechanism, and you need to rule out a hidden third factor.

See it in the table

Au GoldTea Tea

Check your understanding

A bathroom scale reads 2 kg too high every time, with readings very close together. This scale is

The readings cluster tightly, so it is precise. The constant offset means they miss the true value, so it is not accurate. Averaging cannot remove this kind of bias.

You average 100 readings in place of 4. Roughly how much smaller is the random uncertainty of the mean?

Random uncertainty falls as the square root of the number of readings. The ratio of the roots, 10 to 2, gives a factor of 5.

A random poll of 2,500 people has a margin of about

One over the square root of 2,500 is 1/50, which is 0.02, or 2 percentage points.
← Part 4: Measurement, units and powers of tenMark read · Part 6: Limits of science, and science with technology and society →