One of the more confusing aspects of Search Engine Optimization (SEO) is dealing with the files that control the search engine spiders or crawlers. This article will show you the uses and basic structure of each file.
A robots.txt file tells search engines what you do not want them to index. It can include commands that pertain to all search engines, or targeted to specific crawlers.
Example:
User-agent: *
Disallow: /cgi-bin
Disallow: /eklog
Disallow: /images
User-Agent: MSNbot
Crawl-Delay: 10
The first 4 lines are directed to all crawlers. The last 2 lines tell MSNbot to delay indexing 10 seconds between pages to prevent overloading your site.
In contrast to a ..
Source: crawlers
I mentioned in my second poll post that the Stompernet guys would be coming out with a new tool soon, and that also they would be limiting availability of some of their goodies to those that subscribe. That will include this new SEO tool.
First off, I have worked out how to get their videos to syndicated correctly, so click through to watch the video showing off this new tool.
The bad news is that the first piece of totally new content to be locked is a video on keyword research by Dan Thies.
Now Dan is known by professional SEOs as “The ..
Source: seo