Package com.panforge.robotstxt

Provides support for reading and parsing robots.txt

See: Description

Interface Summary

Interface	Description
Grant	Grant.
RobotsTxt	Represents access policy from a single "robots.txt" file.

Package com.panforge.robotstxt Description

Provides support for reading and parsing robots.txt

Standard for Robots Exclusion is a mechanism allowing servers to communicate with web crawlers about it's access policy. This implementation follows recommendations found in the following sources:

http://www.robotstxt.org/orig.html
http://www.robotstxt.org/norobots-rfc.txt
https://en.wikipedia.org/wiki/Robots_exclusion_standard
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt