share

Robots.txt

  • https://www.unminify.com
  • :
  • TOOL TO UNMINIFY JAVASCRIPT

Robots.txt

The robots.txt file is part of the the robots exclusion protocol (REP), which regulate crawler site access to files/sections/directories of your web server.

Why robots.txt Matters?

  • Advises search engine crawlers which pages or files the crawler can or can't access from your site.
  • Keeps web robots at bay so not to overload your servers with request (if they, respect them)

How To Fix Your Robot.txt Issues

  • Edit your file to be as simple as possible, targeting whats blocked
  • Upload your changes to your server and test
  • Page specific blocks can be used in the head portion of your page as follows:
    <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">

Overview