Results 1 to 12 of 12

Thread: Robot.txt File

  1. #1
    Join Date
    Mar 2014
    Location
    Dubai, United Arab Emirates
    Posts
    52

    Robot.txt File

    I want to discuss about the role of robot.txt file in SEO. What is the main purpose of this file when we submit sitemap to Google? What is the difference between sitemap and robot.txt file?

  2. #2
    Join Date
    May 2013
    Location
    Delhi India
    Posts
    88
    This one is very simple ! Robot.txt is Basically used for not to Index website Pages By Google ! If you don't wanna to index some pages of a website than you have to use Robot.txt !

    Sitemap is used to index website pages fast on Google, So this is differance between robot.txt file & sitemap

  3. #3
    Join Date
    Oct 2012
    Posts
    28
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them to visit and not to visit. Robots.txt is not mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

    The structure of a robots.txt is pretty simple – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:

    User-agent:

    Disallow:

    “User-agent” are search engines' crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line:

  4. #4
    Join Date
    Jan 2014
    Location
    India
    Posts
    38
    Xml Sitemap for Index the Web Pages(website) in Major Search Engines.

    Robots.txt for Block to Indexing Some Confidential File in Search Engines.

  5. #5
    Join Date
    Dec 2014
    Location
    Torrence
    Posts
    17
    The file robot.txt is basically a text file. You need to add some text on the file to tell Google that you do not want some pages of the website to be crawled by it. Just like sitemap.xml is for making Google crawl over the pages on your website, robot.txt is to prevent Google from visiting pages on your website.

  6. #6
    Join Date
    Jan 2015
    Posts
    44
    it is require file to indicate the crawlers about which pages to index and which to not. if you dont put robots.txt file cralwer will index all pages of your site.it is not require for all websites.

  7. #7
    Join Date
    Feb 2014
    Posts
    34
    Hi Dear Friends Robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

  8. #8
    Join Date
    Mar 2015
    Posts
    10
    Robots.txt file is created, typically with notepad, to tell the good robots-'crawlers' that you don't wish them to crawl/index certain sections or pages in your site. A robots.txt file is not really needed, but I would put one in that allows all crawling.

  9. #9
    Join Date
    Mar 2015
    Posts
    4
    Sitemaps are used for let SE know what to crawl and Robots.txt is to restrict form crawling or indexing.

  10. #10
    Join Date
    Jan 2015
    Posts
    3
    Robots.txt is file through which you can allow Google bots, crawl or not to crawl your website. Robots.txt is a text file with a code User-agent: * and you can disallow Google bots to crawl your website by command Disallow: /wp-admin/.

  11. #11
    Join Date
    Apr 2015
    Posts
    72
    Robot.txt file is used to allow and disallow web pages for indexing and crawling. It informs to Google.

  12. #12
    Join Date
    May 2015
    Posts
    240
    Robots.txt is a plain text file that you upload to the root directory of website. Bots look at this text file and process it for indexing and decides which pages not to crawl.

    Sitemaps tells search engines which pages are available for crawling. Sitemap is xml file with a list of URLs for a site along with metadata about each URL.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •