If it’s the Googlebot you really want to say something meaningful. “Take me to your leader” (position) would be nice wouldn’t it? But seriously, your website is visited on a regular basis by it and other ‘bots or crawlers and that leads to your site being catalogued. You can facilitate or impede this process fairly simply if you know what to do. That is the basis of the search engine optimization process.
Several of our clients have come to us recently with problems related to establishing proper communications with the Googlebot and the others. Your website’s robot conversation happens via a text file on your website called robots.txt and even not having one can cause problems.
We found one misconfigured example that resulted in Google’s description of the site being blocked and Google said so publicly. This is not a result you want clients and prospects to see. This misconfiguration had also had a negative effect on the search returns for this site. We corrected the problem and within several weeks the description appeared correctly and the site resumed moving upwards in search rankings.
This robots.txt file has a variety of uses. It can block access to all or parts of your site. It can block some or all ‘bots. One of our clients is using an older ecommerce system that changes expired product listings into pages that look like an error to Google. When Google indicates that your site is exhibiting hundreds of errors it will negatively affect your search rankings. We fixed this problem by using the robots.txt file to instruct the Googlebot and other ‘bots to not catalogue the problem pages related to expired listings from the crawl. Once again this had a positive effect on the site’s search rankings.
If you have questions about robots or anything to do with making your website a more effective business tool call me, text me, email me or tweet me.