For more information contact Professor Dan Remenyi at dan.remenyi@mcil.co.uk

The Top Ten Search Engines

Based on a survey by StatMarketRatings April 3 2000. 

Search Engine Referrals.  

Yahoo, Snap and Lycos are directories. Ask Jeeves is an Answer Service and is more like a directory as human beings compile its listing. Users pose questions in plain English and receive links to Web sites containing relevant information, services and products. 

Search Engines and Directories Explained

It is important to understand the differences between search engines and directories. It is also important to understand the way in which search engines work before discussing how a web site can be found effectively through them.

Search Engines: 

Search engines create their listings automatically. They crawl the web and users search through what they have found. When a web page is changed, search engines find these changes and this can affect the web page listing. Page titles, site content and other elements all play a role.

Directories: 

Descriptions of sites are submitted to the directory or sites are reviewed by editors. The search then looks for matches in the descriptions submitted. Changing web pages has no effect on the listing. Factors affecting directory listing are different to those affecting search engine listings. Sites with extensive and original content are more likely to get reviewed than a poor site. Therefore the key to a good directory listing is to have a well-designed site with extensive and original content.

Hybrid Search Engines: 

Some search engines maintain an associated directory. Being included in a search engine’s directory is usually a combination of luck and quality. Sometimes you can “submit” your site for review, but there is no guarantee that it will be included.

Three major elements of a Search Engine:

1. The spider: Also called the crawler. The spider visits a web page, reads it, and then follows links to other pages within the site. Hence the term a site being “spidered” or “crawled”. The spider returns to the site on a regular basis, such as every month or two, to look for changes. This can have an effect on website statistics, boosting the number of website hits, when in reality neither surfers nor users are visiting the site.

2. The Index: The spider puts everything it finds into the index. The index, also called the catalog, contains a copy of every web page that the spider finds. If a web page changes, the index is updated with new information. Sometimes a new page or a page that has been changed will have been spidered but not yet indexed. Until it is indexed it will not be available to those searching with the search engine.

3. Search Engine Software: This is the program that sifts through the millions of pages recorded in the index, finding matches and ranking in order of relevance.

How Search Engines Rank Web Pages

Search engines rank web pages on the basis of how relevant they find them in relation to certain topics. The primary rules that search engines use for determining relevancy are position and frequency.

Position

Search engines assume pages with keywords appearing in the title are more relevant to the topic than pages without. Search engines also check to see if the keywords appear near the top of a web page, such as in the headline or in the first few paragraphs of text. They assume that any page relevant to the topic will mention those words right from the beginning.

Frequency

Search engines analyze how often keywords appear in relation to other words in a web page. Those with a higher frequency are often deemed more relevant than other web pages.

Different Search Engines

A search on different search engines may produce different results. Search engines vary in the number of pages indexed and in the frequency that they index web pages. Therefore no search engine has the exact same collection of web pages to search through.

Search engines may also improve a pages ranking for certain reasons. For example the search engine Excite uses link popularity as part of its ranking method. The search engine can tell which pages in its index has a large number of links pointing at them. Such a page would have its ranking improved as it is probably well regarded on the Internet.

Hybrid search engines may give relevancy boosts to sites that they’ve reviewed. Their logic is that because a site was good enough to review it’s more relevant than an unreviewed site.

Search Engine Spamming

Web page designers may repeat a word hundreds of times on a page to increase the frequency and improve the pages ranking. However search engines watch for this and may detect it easily. If a search engine detects search engine spamming it may penalize or exclude the page from its index. Search engines are becoming more adept at recognizing spamming techniques.

The importance of top Search Engine Ranking

Although search engines often return thousands of search results, usually only the 10 most “relevant” results are initially displayed. Quite often a user may find a site that satisfies their requirements in the first 10 results. Such a user may not see any need to look through the remaining results. Therefore it is essential that a website is in the “top ten” results.

Tips to Improve Search Engine Ranking