robots.txt and sitemap files defined
One of the more confusing aspects of Search Engine Optimization (SEO) is dealing with the files that control the search engine spiders or crawlers. This article will show you the uses and basic structure of each file.
A robots.txt file tells search engines what you do not want them to index. It can include commands that pertain to all search engines, or targeted to specific crawlers.
Example:
User-agent: *
Disallow: /cgi-bin
Disallow: /eklog
Disallow: /images
User-Agent: MSNbot
Crawl-Delay: 10
The first 4 lines are directed to all crawlers. The last 2 lines tell MSNbot to delay indexing 10 seconds between pages to prevent overloading your site.
In contrast to a ..
Source: crawlers
Leave a Reply