share

Robots.txt

  • https://www.diyschools.com
  • :
  • BEST SCHOOL NEAR ME, SCHOOL NEAR ME, TUITION NEAR ME, COACHING NEAR ME

Robots.txt

The robots.txt file is part of the the robots exclusion protocol (REP), which regulate crawler site access to files/sections/directories of your web server.

Why robots.txt Matters?

  • Advises search engine crawlers which pages or files the crawler can or can't access from your site.
  • Keeps web robots at bay so not to overload your servers with request (if they, respect them)

How To Fix Your Robot.txt Issues

  • Edit your file to be as simple as possible, targeting whats blocked
  • Upload your changes to your server and test
  • Page specific blocks can be used in the head portion of your page as follows:
    <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">

Overview