Results 1 to 5 of 5
  1. #1

    Protect Your SEO With This Robots Txt Generator

    A few days ago I found an awesome tool that all of us can use. Whether you're designing mobile sites, web sites or even just own a site you can use this robots.txt generator. Websites use a robots text file to let the crawlers know about their site. The crawler visits your site and looks for the.txt file. If that file says not to crawl, the robot will not crawl your site. It's that simple.

    Why wouldn't I want a robot to crawl my site? Give Your Opinion?

  2. #2
    Join Date
    Oct 2011
    Because you don't want to crawl over search engines.

  3. #3
    Join Date
    Jan 2005
    Idaho USA
    Some webmasters do not want certain pages indexed and do not want them displayed in SERPs. An example is the checkout page of an e-commerce site.

    There's little or no point in having your checkout page come up in SRERPs. You want people to find and visit your product pages. They'll get to the checkout page when they buy. There's no need to search for it.

    Another example may be certain pages belonging to a university or scientific organization. Some of their pages may contain information/data useful only to insiders and they may not want outside/uninvited visitors on the page.
    The Old Sarge

  4. #4
    simple reason is that your page is not read as per SEO requirements or you're using duplicate content which may harm your site reputation.. that's why we use robots.txt file to hide these pages.

  5. #5
    Join Date
    May 2012
    Kuala Lumpur, Malaysia
    Quote Originally Posted by shalyn View Post
    SEO requirements
    The original idea of robots.txt is to avoid search engine crawl privacy page. Page such as admin login, checkout page.

    Due to Google panda update, webmaster is realize the important of quality content. Up trend, webmaster implement robots.txt to avoid duplicate content.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts