Material Insights · Understanding the Material World

Web search engine

Programs crawl and index the web and rank pages for any query.

Computing and AI · AI and data · No. 289

  • Digital age
  • Readiness 9 of 9
  • Platform
Computing and AI · Technology

WebCrawler, a student project of 1994, indexed about 50,000 pages in its first year; one engine has long handled about 90% of US searches.

Chart: Illustration: Web search engine

Where it comes from

Chart: Map of where it comes from

Early search engines were student and university projects: JumpStation at Stirling in December 1993 and WebCrawler at the University of Washington, live in April 1994. Google began as a Stanford research project and raised its first 100,000 dollars in 1998.

Three things to know
  1. WebCrawler, which went live on 20 April 1994, was among the first to search the full text of pages, not just their titles.
  2. Brin and Page at Stanford ranked pages by the web's link graph: a page linked from many important pages ranks higher.
  3. In August 2024 a US federal court found that Google had acted unlawfully to maintain its search monopoly, and remedies followed in 2025.
The journey
  1. 1 CrawlRobot programs follow links from page to page and fetch each one
  2. 2 IndexEvery word is listed with the pages it appears on
  3. 3 RankPages are scored by their words and by the links pointing to them
  4. 4 QueryA user types words and gets the best-scoring pages in a fraction of a second
  5. 5 AdvertisingPaid adverts beside the results pay for the service
Go deeper pick a tab to open it in full