How Search Engines Work
Most likely, you have probably used a web search engine such as Google, AltaVista or Yahoo! to find specific information on the Internet. Did you ever stop and wonder how exactly do search engines find that information? We will explain that to you in this section. In so doing, you will begin to discover how to efficiently structure your web site to obtain maximum brand strength from search engines.
It’s a very common misconception that when a user enters a query into any search engine, the engine interrogates the Web to find pages that match the query. That is NOT how it works at all! The search engine searches its own copy of the Web. Every search engine actually creates its own copy of the Internet. This copy is called an index. The size of a search engine’s index varies from search engine to search engine, but it is always smaller than the Web as a whole. For example, as of August 2003, it is currently estimated that the Web presently consists of approximately 3.4 Billion pages, whereas the largest search engine index is currently about 600 million pages.
The search engine builds a list of pages to add to its index using a special piece of software known as a crawler or spider. The spider crawls across the Web, adding pages it visits to the list of pages to its index. The spider is capable of reading a Web page and finding links to other pages to visit. In this fashion, the spider can often travel across the Web finding new or updated pages to add or update to its index.
Some time after a page has been “spidered” (visited by a crawler), the search engine’s software effectively adds a copy of new or altered pages to the search engine’s index. When a user enters a query into a search engine, the search’s engine software searches its index to find the pages matching the search query. It then sorts those pages into ranking order. Each search engine uses its own “secret technology” to find and rank pages, but most base their technology on the frequency and location of the search term on the page.
This, as briefly as it sounds, is how search engines actually work. With that minimal knowledge and a little thought, you can reach the following conclusions:
An engine may not have a copy of every page on your site
If it does have it, that copy may not be up to date
An engine can have a copy of a page that no longer exists on your site
Also, remember that in order for a page on your site to be listed in response to a user query, words in that query must match words in the search engine’s copy of the page. The professional image of your company or your brand could be, to a certain degree, damaged if you don’t take these facts into consideration. What we mean by this is, if the search engine looks for a page that still happens to be on its index but is no longer available on your web site, it will generate what is called an HTTP error 404, wich means the page the engine and thus the user are looking for at your site does not exist. This fully underscores the importance of updating your web site at regular intervals and making certain there are no pages missing or any broken links.
Author:
Serge Thibodeau of Rank For Sales
