How does a search engine work?

Search engine technology relies on three different elements:

• Crawling – First, search engines deploy a series of automated agents, known as spiders or crawlers. These spiders quickly travel from page to page, scanning content and cataloging the images, text, links and meta data from each.
• Index – The information obtained by the spiders are organized into an index that stores a wide variety of information about each page visited. This index represents the entire universe of pages that can appear in a search engines’ result set. The index for a typical general search engine such as Google or Yahoo! contains information on billions of documents.
• Algorithm – Each engine deploys a complex formula, or algorithm, which is used to determine the relevance of each individual page. When a user conducts a search, the search engine retrieves all pages that match it and then uses the algorithm to rank the pages in order of relevance. Search engines continually update their algorithms in an attempt to further refine the relevance of their search results.

More information on how search engines find and rank pages from Wikipedia.