How Search Engine Spiders Work

Search Engine Optimization

By: Stephen S Alison 

There are hundreds of search engines available today, but some are far more complex than others. This article will give you an overview of how some of the most popular ones work.

Let’s start with a smaller engine: InfoSeek. They only index about 200 words of your web page, so it’s important to make sure that you have meta tags on your site, and that the most important things are listed first. The information you put in your meta tags will be used to display a description of your site, and most meta tags can contain about 200 characters of text. The keywords meta tag, however, can have up to 1,000 characters.

Excite used to be a search powerhouse, but has now been dropped as the provider of AOL and Netscape search, so it’s less important than it once was. The algorithm it uses to determine keyword relevance is very complicated: it indexes your pages and then attempts to summarize them by selecting only the most relevant sentences. Expect to have your pages reviewed roughly once every two weeks. Keep in mind, though, that with meta tags have no meaning to Excite when it comes to rankings, even though it will use your description tags as long as the words are relevant to your pages’ content.

Let’s move on to Lycos. Lycos has fully integrated the Open Directory Project (ODP) into their mainstream results pages, and they also use search results from AllTheWeb. Lycos also runs click-throughs to their sister site HotBot. Lycos is one of the harder search engines to understand, as their submission pages say one thing but then they index your site in a completely different way. As a general rule of thumb, your site will be indexed in Lycos in due time as long as you get indexed in ODP and AllTheWeb.

Even though WebCrawler is owned by Excite, it still has its own search engine and indexer. If you happen to be listed with WebCrawler, you should try to stay listed with them, as it isn’t the easiest search engine to get listed with. Its hit-and-miss standards combined with the sporadic indexing methods makes the submission process tough, although not impossible.

The biggest player is, of course, Google, who use a page ranking system as the central basis of their index. It was once nearly impossible to manipulate this page ranking system to drive up your rankings, but people quickly figured out that the more links they could generate to their site on the rest of the net, the better Google ranked them. Google is not thought to be using context-sensitive rankings. Context-sensitive information is used at Yahoo, Looksmart and the ODP, however, and Google regularly spiders those sites when it re-indexes its own database.

MSN is another important search engine. The holy trinity of search engines at the moment is Google, Yahoo!, and MSN. These three search engines combine to provide you with the vast majority of the traffic that you will receive from search engines. MSN will generally be the first search engine to index your site and it will almost certainly list the most pages the fastest.

Although no-one can tell you exactly when you will be indexed on any search engine, it’s best to check back at least weekly. Whatever you do, though, don’t re-submit your site more often than every two months or so – you might not get indexed at all if you do this.

Search Engine Optimization Techniques -  The Basic SEO Rules for All Websites. Search engine optimization isn’t rocket science, but it can feel complicated at times. To have a successful campaign, just follow these simple rules detailed here – they apply to all websites.

The Grand Daddy of all Seo Software! Get A Top 5 Google Ranking In Under 30 Days!

The days of the dinosaurs are over! Generate massive keyword lists and spy on your Adwords competition!