|
Save Googlebot From Crawling Infinite Space
By Navneet Kaushal
Expert Author
Article Date: 2008-08-06
Over at the Google Webmaster Central Blog, Google has put up an interesting post about how Googlebot can get stuck in the loop of 'infinite space' and what measures can the Webmasters take to avoid such a glitch.
Infinite Space as defined by Google: "These are very large numbers of links that usually provide little or no new content for Googlebot to index. If this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in Googlebot failing to completely index the real content on your site."
However, with the increase in such incidents, Google has now started to inform the Webmasters via the Webmaster Tools message center, whenever such an anomaly is discovered. Because the issue of infinite space can cause inconvenience to the Webmasters, therefore it is imperative that they get their websites verified with Webmaster Tools. Frequent check of the Webmaster Tools message center would also serve as an added advantage.
Google Webmaster Tools Message Center:
Example of an Infinite Space:
Google has taken the example of a calendar with a 'next month' link. As per Google, "It may be possible to keep following those "Next Month" links forever! Of course, that's not what you want Googlebot to do. Googlebot is smart enough to figure out some of those on its own, but there are a lot of ways to create an infinite space and we may not detect all of them."
How to Avoid/Correct the Infinite Space Issue:
However, while explaining the problem of Infinite Space, Google has also provided some tips as to how Webmasters can avoid and correct this anomaly.
- Go through the Webmaster Tools Help article, where you can know more about the steps that can be taken to avoid infinite space.
- Eliminate whole categories of dynamically generated links using the robots.txt file. You can learn more about the usage of the robots.txt file for the elimination of infinite space here.
- You can also block these useless links with a "nofollow" link attribute. You can learn more about the usage of the "nofollow" link attribute here.
Comments
About the Author:
Nav is the founder and CEO of Page Traffic, a premier search engine company
known for its assured SEO service, web design and development, copywriting and full time SEO professionals.
Navneet has wide experience in natural search engine optimization, internet
marketing and PPC campaigns. He is a prolific writer and his articles can be
found in the "Best Articles" section of many websites and article banks. As
a search engine analyst , he has over 9 years of experience and his
knowledge is in application here.
|
|
