Page 2 of 2 12
Results 16 to 30 of 30

Thread: What is robots.txt?

  1. #16
    Join Date
    Nov 2019
    Posts
    60
    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. ... The slash after “Disallow” tells the robot to not visit any pages on the site.

  2. #17
    Join Date
    Dec 2019
    Posts
    3
    Robots. txt is a text file webmasters create to instruct web crawler (typically search engine robots) how to crawl pages on their website

  3. #18
    Join Date
    Dec 2019
    Posts
    29
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  4. #19
    Join Date
    Nov 2019
    Posts
    13
    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

  5. #20
    Join Date
    Nov 2019
    Location
    India
    Posts
    20
    Robot.txt is a file that allows the Google crawler to crawl pages that allowed the Robot.txt file. If you want to hidden your pages to Google crawler then you can not give permission.

  6. #21
    Join Date
    Jun 2019
    Location
    Dubai
    Posts
    14
    Impressive guys, agree with your answers.

  7. #22
    Join Date
    Mar 2020
    Posts
    11
    The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

  8. #23
    Join Date
    Mar 2020
    Location
    India
    Posts
    8
    The robot.txt file allows Google crawler which pages it needs to crawl. The crawler will not visit your website unless your website has a robots.txt file.

  9. #24
    Join Date
    Apr 2020
    Location
    Ghatkopar, Mumbai
    Posts
    2
    Robots.txt is a file that tells bots of every search engine what to do with your webpages. Should webpages index or not. follow links on the webpage or not. It helps bots to crawl your website effectively.

  10. #25
    Join Date
    May 2020
    Location
    Vietnam
    Posts
    48
    Have you tried, I don't know, Google it?

  11. #26
    Join Date
    Aug 2020
    Posts
    37
    The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

  12. #27
    Join Date
    Aug 2020
    Posts
    73
    A robots. txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.

  13. #28
    Join Date
    Apr 2021
    Posts
    116
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.

  14. #29
    Join Date
    Jun 2021
    Posts
    85
    Robots. txt is a text file is the user created file to instruct web robots

  15. #30
    Join Date
    Jul 2021
    Location
    USA
    Posts
    22
    Hi,

    With the help of robots. txt file search engine crawler can access a URLs on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page away of Google, block indexing with no index or password-protect the page.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •