Material Insights · Understanding the Material World

Information theory

Information can be measured in bits, and every channel has a maximum rate for error-free messages.

Mathematics and information · Information theory · No. 126

  • Electronic age
  • Science
Mathematics and information · The science behind technology

Claude Shannon showed in 1948 that every noisy channel has a top speed for error-free messages, and that clever coding can come close to it.

Chart: Illustration: Information theory

Where it comes from

Chart: Map of where it comes from

Shannon, a mathematician at Bell Labs, published his theory in two parts in July and October 1948. It began with the problem of reproducing at one point a message chosen at another.

Three things to know
  1. The 1948 paper appears to contain the first published use of the word bit.
  2. Shannon's rule says a message can be sent with as few errors as you like at any rate below capacity, but not above it.
  3. Gallager invented low-density parity-check codes by 1962, and they were largely forgotten for more than 30 years.
The journey
  1. 1 BitInformation measured as a choice between two options
  2. 2 CapacityA noisy channel has a top rate for reliable messages, 1948
  3. 3 CodesHamming builds the first practical error-correcting codes, 1950
  4. 4 Closing inTurbo codes come close to the limit, 1993
  5. 5 TodayPhones, fibre, Wi-Fi, streaming and the web
Covered in Heat, temperature and the laws of thermodynamicsScience
Go deeper pick a tab to open it in full