Material Insights · Understanding the Material World

Information theory: The idea

In plain words

The idea explained without jargon

Information can be counted. One bit is the amount needed to settle a choice between two equally likely options, like a coin toss. A message full of patterns, such as English text, carries less per symbol than its alphabet could, which is why it can be squeezed into fewer bits. Every channel, whether a cable, a radio link or a fibre, also has a capacity: a ceiling on the bits per second it can carry, set by its bandwidth and its noise. Below that ceiling, clever coding lets a message arrive with as few errors as you like. Above it, errors cannot be avoided [1]. The theory proves the ceiling is real but does not say how to build codes that reach it; that took decades [5].

as of 2026-11-12

Key equation

The equation that sums it up, and what each symbol means

Equation
C = W log2(1 + S/N)
Symbols
{"C":"the capacity: the most bits per second that can be sent with as few errors as you like","W":"the bandwidth of the channel, in hertz","S":"the average signal power received","N":"the noise power in the same band"}
Note
Shannon wrote it as C = W log((P + N) / N) for a channel with white thermal noise, where P is the transmitter's average power; with logarithms to base 2 it gives bits per second, and it is the same as the form above with S = P [1]. Letting the bandwidth grow without limit, the least energy per bit needed for reliable transmission, relative to the noise, falls to ln 2, or -1.59 dB, the 'ultimate Shannon limit' [5].

as of 2026-11-12

An analogy

A picture from everyday life that makes the idea easier to hold

Think of a water pipe. A pipe of a given width can carry only so many litres a second however hard you push, and good packing, such as filling it with no gaps and no air, lets you get close to that figure. Capacity works like the pipe's width. The picture stops being true because noise, not leaks, sets the limit, because the theory counts surprise rather than meaning, and because the limit can be approached only by coding long blocks of data.

as of 2026-11-12

Sources

  1. A Mathematical Theory of Communication The Bell System Technical Journal (C. E. Shannon), 1948
  2. Channel Coding: The Road to Channel Capacity Proceedings of the IEEE, on arXiv (D. J. Costello Jr. and G. D. Forney Jr.), 2006