How Do The Search Engine Spiders Work?

When it comes to the subject of search engine optimization, there’s a lot of confusion. One common area of confusion is about search engine spiders, commonly called search engine robots.

If you want your site to be found or indexed in the search engines, there’s several facts you need to understand. First, you need to know what Search Engine Robots are all about. What is a Search Engine Robot? Search Engine Robots are computer programs that visit your web page, gather information about your site and then provide this information back to the search engines.

How do search engines view your site when the robots arrive at your site? Do they view your site as one site comprised of a group of pages or do they compile information about your site as many individual pages?
The answer to this question may surprise you since the search engine robot neither sees your site as one web site or a bunch of pages.

Sometimes when a search engine spider comes to your site, it gets confused and cannot understand the organization of your site. This situation is often caused by what is termed a spider trap. What’s a spider trap? A spider trap is a situation where the spider thinks it is looking at a bunch of unique pages when it is really looking at the some page. This is caused by variable strings at the end of urls that are session identifiers. The spiders often get lost in a situation like this.

What determines what page your site will rank on in a search engine and how intelligent are the search engine spiders? They are actually rather dumb and can’t really think.

Another problem with the spiders is that they become outdated and the approach the spider software uses is no longer current and up-to-date. Some spiders are better than others in crawling more complex web sites but the spidering approach all the spiders take is somewhat identical.
Here’s some tips to consider:

  1. Your objective should be to keep the spiders returning often to your site because new content usually ranks higher. Since “content is king” on the Internet, you need to constantly add new material to your site such as articles, videos, and good high quality information. This new content should be consistently added over time as opposed to all at once. For example, you might add one new piece of content per day.
  2. Contrary to what many people think, it is not a good idea to include a robots revisit tag in your HTML code. This simply doesn’t work. You want the robots to keep coming back as you add new content. With new content they will stay longer and examine more of your site pages.
  3. Everyone knows blogs get crawled often by the spiders. The reason is simple- blogs constantly add new content- sometimes as often as daily.
  4. Although some website owners submit their site to the search engines using software. This is not a recommended approach. A better approach is for the spiders to find you from links from other sites which are already indexed and are pointing to you. Submittal is advisable, however, when it comes to directories humans examine these entries and decide how to handle them.
  5. Another tip is to use a site map. A site map is exactly as it sounds- a map of the relationship of your pages to each other and how they are linked. To maximize the use of a site map, The best use of a site map is to link every page on your site to the Stieglitz, a task which most people forget, according to Matt Cutts. The reason for this is that the spiders can find one of your pages (not necessarily the home page) and you want the spiders to find the other pages from any page. A site map set up properly will accomplish this.
    What do you think the best approach is for having Google index your site- right away or after pages off your site link to you?

The best approach is to get other parts of the web such as directories linked to you first because these off page factors can help your rank in a major way. The rewards of off page factors is huge!

Want to learn more about SEO and get hundreds of tips for FREE? A tip a day will make you an expert. Sign up for a Free Tip of the Week on the right side of this page. You can unsubscribe at any time.

Certified Master SEO Instructor by the Search Engine Academy and CEO and founder of SEO Training SW.
Roy Reyer
Roy Reyer
View all posts by Roy Reyer
Roy's website
VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

18th February 2009 No Comments

Leave a Reply

Your email address will not be published. Required fields are marked *






Webinar Replay How to Increase Conversions and Traffic to Your Website

October 16th, 2014


Warning: Use of undefined constant php - assumed 'php' (this will throw an Error in a future version of PHP) in /home/customer/www/ on line 55