![]() |
Webmaster information |
How can I prevent the DepSpid spider from visiting certain places of a domain?The DepSpid spider follows the Robots Exclusion Standard (robots.txt). A good place for finding information about it are the Web Robots Pages.
If you want to prevent all robots from accessing certain places, you should specify a User-Agent: * section in your robots.txt file. If you want to only prevent the DepSpid spider from accessing certain places, you will need to use a User-Agent: DepSpid section in your robots.txt file. |