com.panforge.robotstxt

Interface RobotsTxt

public interface RobotsTxt

Represents access policy from a single "robots.txt" file.

Use read(java.io.InputStream) to read and parse robots.txt.

Method Summary

Modifier and Type	Method and Description
default Grant	ask(String userAgent, String path) Asks for grant.
Integer	getCrawlDelay() Deprecated. use ask(java.lang.String, java.lang.String) to get Grant from which Grant.getCrawlDelay() might be invoked.
List<String>	getDisallowList(String userAgent) Gets a list of disallowed resources.
String	getHost() Gets host.
List<String>	getSitemaps() Gets site maps.
boolean	query(String userAgent, String path) Checks access to the given HTTP path.
static RobotsTxt	read(InputStream input) Reads robots.txt available at the URL.

Method Detail

query

boolean query(String userAgent,
              String path)

Checks access to the given HTTP path.

Parameters:

userAgent - user agent to be used evaluate authorization

path - path to access

Returns:

true if there is an access to the requested path

ask

default Grant ask(String userAgent,
                  String path)

Asks for grant.

Parameters:

userAgent - user agent to be used evaluate authorization

path - path to access

Returns:

grant (never null)

getCrawlDelay

@Deprecated
Integer getCrawlDelay()

Deprecated. use ask(java.lang.String, java.lang.String) to get Grant from which Grant.getCrawlDelay() might be invoked.

Gets crawl delay.

Returns:

crawl delay in seconds or 0 if no delay declared

getHost

String getHost()

Gets host.

Returns:

host or null if no host declared

getSitemaps

List<String> getSitemaps()

Gets site maps.

Returns:

list of site map URL's.

getDisallowList

List<String> getDisallowList(String userAgent)

Gets a list of disallowed resources.

Parameters:

userAgent - user agent

Returns:

list of disallowed resources

read

static RobotsTxt read(InputStream input)
               throws IOException

Reads robots.txt available at the URL.

Parameters:

input - stream of content

Returns:

parsed robots.txt object

Throws:

IOException - if unable to read content.