The robots.txt file is part of the the robots exclusion protocol (REP), which regulate
crawler site access to files/sections/directories of your web server.
Why robots.txt Matters?
Advises search engine crawlers which pages or files the crawler can or can't
access from your site.
Keeps web robots at bay so not to overload your servers with request (if they,
respect them)
How To Fix Your Robot.txt Issues
Edit your file to be as simple as possible, targeting whats blocked
Upload your changes to your server and test
Page specific blocks can be used in the head portion of your page as follows: <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
