There is not need to write html in Robot text file, Some special codes are required to write there which you can find early in Google and this text file placed in Root folder of your project, Simple..........
Printable View
There is not need to write html in Robot text file, Some special codes are required to write there which you can find early in Google and this text file placed in Root folder of your project, Simple..........
They will also give you statistics of how many of your pages are open to Google's bot as well as how many you have limited. If you have set up a Google Sitemap you can enforce their tools to test your robots.txt file.
A robots.txt file is a short text file that is in your home directory. Before search engines spider your site, look at this file to see which files / file types and / or directories that are not authorized to see.
Thanks a lots to All of you for this information coz as it concern to search engine spider in matter to crawl so it very important to know and also how to implement in our site.
As I am newbie in this field so for me I have learn today something new and very much important.