All Implemented Interfaces:

java.lang.Comparable
```
public final class WebPage
 implements Comparable<WebPage>
                    
```
Every WebPage is a seperate execution context
Notice: Use a build-in java string or a Utf8 to serialize strings?
see org.apache.gora.hbase.util.HBaseByteInterface#fromBytes
In serialization phrase, a byte array created by s.getBytes(UTF8_CHARSET) is serialized, and in deserialization phrase, every string are wrapped to be a Utf8
So both build-in string and a Utf8 wrap is OK to serialize, and Utf8 is always returned

Field Summary

Fields
Modifier and Type	Field	Description
`public final static Logger`	`LOG`
`public static AtomicInteger`	`sequencer`
`public static WebPage`	`NIL`
`private String`	`url`
`private String`	`reversedUrl`
`private VolatileConfig`	`conf`
`private final Variables`	`variables`
`private boolean`	`isCached`
`private boolean`	`isLoaded`
`private boolean`	`isFetched`
`private volatile boolean`	`isContentUpdated`
`private volatile ByteBuffer`	`tmpContent`
`private String`	`args`

Method Summary

Modifier and Type	Method	Description
`String`	`getUrl()`	page.location is the last working address, and page.
`String`	`getReversedUrl()`	Getter for the field `reversedUrl`.
`VolatileConfig`	`getConf()`
`void`	`setConf(@NotNull() VolatileConfig conf)`
`Variables`	`getVariables()`	***************************************************************************Common fields****************************************************************************
`boolean`	`isCached()`
`boolean`	`isLoaded()`
`boolean`	`isFetched()`
`boolean`	`isContentUpdated()`
`ByteBuffer`	`getTmpContent()`	Get the cached content
`void`	`setTmpContent(ByteBuffer tmpContent)`	Set the cached content, keep the page content unmodified
`String`	`getArgs()`	The load arguments is variant task by task, so the local version is the first choice,while the persisted version is used for historical check only
`void`	`setArgs(@NotNull() String args)`	Set the local args variable and the persist version, and also clear the load options.
`static WebPage`	`newWebPage(@NotNull() String url, @NotNull() VolatileConfig conf)`	newWebPage.
`static WebPage`	`newTestWebPage(@NotNull() String url)`	newWebPage.
`static WebPage`	`newWebPage(@NotNull() String url, @NotNull() VolatileConfig conf, @Nullable() String href)`	newWebPage.
`static WebPage`	`newInternalPage(@NotNull() String url)`	newInternalPage.
`static WebPage`	`newInternalPage(@NotNull() String url, @NotNull() String title)`	newInternalPage.
`static WebPage`	`newInternalPage(@NotNull() String url, @NotNull() String title, @NotNull() String content)`	newInternalPage.
`static WebPage`	`newInternalPage(@NotNull() String url, int id, @NotNull() String title, @NotNull() String content)`	newInternalPage.
`static WebPage`	`box(@NotNull() String url, @NotNull() String reversedUrl, @NotNull() GWebPage page, @NotNull() VolatileConfig conf)`	Initialize a WebPage with the underlying GWebPage instance.
`static WebPage`	`box(@NotNull() String url, @NotNull() GWebPage page, @NotNull() VolatileConfig conf)`	Initialize a WebPage with the underlying GWebPage instance.
`static WebPage`	`box(@NotNull() String url, @NotNull() GWebPage page, boolean urlReversed, @NotNull() VolatileConfig conf)`	Initialize a WebPage with the underlying GWebPage instance.
`static Utf8`	`wrapKey(@NotNull() Mark mark)`	***************************************************************************Other****************************************************************************
`static Utf8`	`u8(@Nullable() String value)`	What's the difference between String and Utf8?
`String`	`getKey()`	getKey.
`String`	`getHref()`	Get The hypertext reference of this page.
`void`	`setHref(@Nullable() String href)`	Set The hypertext reference of this page.
`boolean`	`isNil()`	isNil.
`boolean`	`isNotNil()`	isNotNil.
`boolean`	`isInternal()`	isInternal.
`boolean`	`isNotInternal()`	isNotInternal.
`GWebPage`	`unbox()`	unbox.
`void`	`unsafeSetGPage(@NotNull() GWebPage page)`
`void`	`unsafeCloneGPage(WebPage page)`
`boolean`	`hasVar(@NotNull() String name)`	Check if the page scope temporary variable with name {@name} exist
`Object`	`getVar(@NotNull() String name)`	Get a page scope temporary variable
`Object`	`removeVar(@NotNull() String name)`	getAndRemoveVar.
`void`	`setVar(@NotNull() String name, @NotNull() Object value)`	Get a page scope temporary variable
`void`	`setCached(boolean cached)`
`void`	`setLoaded(boolean loaded)`
`void`	`setFetched(boolean fetched)`
`int`	`getMaxRetries()`
`Metadata`	`getMetadata()`
`CrawlMarks`	`getMarks()`
`boolean`	`hasMark(Mark mark)`
`String`	`getConfiguredUrl()`	getConfiguredUrl.
`int`	`getFetchedLinkCount()`
`void`	`setFetchedLinkCount(int count)`
`ZoneId`	`getZoneId()`
`void`	`setZoneId(@NotNull() ZoneId zoneId)`
`String`	`getBatchId()`
`void`	`setBatchId(String value)`
`void`	`markSeed()`
`void`	`unmarkSeed()`
`boolean`	`isSeed()`
`int`	`getDistance()`
`void`	`setDistance(int newDistance)`
`void`	`updateDistance(int newDistance)`
`FetchMode`	`getFetchMode()`
`void`	`setFetchMode(@NotNull() FetchMode mode)`	Fetch mode is used to determine the protocol before fetch, so it shall be set before fetch
`BrowserType`	`getLastBrowser()`
`void`	`setLastBrowser(@NotNull() BrowserType browser)`
`HtmlIntegrity`	`getHtmlIntegrity()`
`void`	`setHtmlIntegrity(@NotNull() HtmlIntegrity integrity)`
`int`	`getFetchPriority()`
`void`	`setFetchPriority(int priority)`
`int`	`sniffFetchPriority()`
`Instant`	`getCreateTime()`
`void`	`setCreateTime(@NotNull() Instant createTime)`
`Instant`	`getGenerateTime()`
`void`	`setGenerateTime(@NotNull() Instant generateTime)`
`Instant`	`getModelSyncTime()`
`void`	`setModelSyncTime(@Nullable() Instant modelSyncTime)`
`int`	`getFetchCount()`
`void`	`setFetchCount(int count)`
`void`	`updateFetchCount()`
`CrawlStatus`	`getCrawlStatus()`
`void`	`setCrawlStatus(@NotNull() CrawlStatus crawlStatus)`
`void`	`setCrawlStatus(int value)`
`String`	`getBaseUrl()`	The baseUrl is as the same as LocationA baseUrl has the same semantic with Jsoup.
`String`	`getLocation()`	WebPage.url is the permanent internal address, it might not still available to access the target.And WebPage.location or WebPage.baseUrl is the last working address, it might redirect to url,or it might have additional random parameters.WebPage.location may be different from url, it's generally normalized.
`void`	`setLocation(@NotNull() String value)`	The url is the permanent internal address, it might not still available to access the target.
`Instant`	`getFetchTime()`	The latest fetch time
`void`	`setFetchTime(@NotNull() Instant time)`	The latest fetch time
`Instant`	`getPrevFetchTime()`	The previous fetch time, updated at the fetch stage
`void`	`setPrevFetchTime(@NotNull() Instant time)`
`Instant`	`getPrevCrawlTime1()`	The previous crawl time, used for fat link crawl, which means both the page itself and out pages are fetched
`void`	`setPrevCrawlTime1(@NotNull() Instant time)`	The previous crawl time, used for fat link crawl, which means both the page itself and out pages are fetched
`Duration`	`getFetchInterval()`
`void`	`setFetchInterval(@NotNull() Duration duration)`
`void`	`setFetchInterval(long interval)`
`void`	`setFetchInterval(float interval)`
`ProtocolStatus`	`getProtocolStatus()`
`void`	`setProtocolStatus(@NotNull() ProtocolStatus protocolStatus)`
`ProtocolHeaders`	`getHeaders()`	Header information returned from the web server used to server the content which is subsequently fetched from.This includes keys such asTRANSFER_ENCODING,CONTENT_ENCODING,CONTENT_LANGUAGE,CONTENT_LENGTH,CONTENT_LOCATION,CONTENT_DISPOSITION,CONTENT_MD5,CONTENT_TYPE,LAST_MODIFIEDand LOCATION.
`String`	`getReprUrl()`
`void`	`setReprUrl(@NotNull() String value)`
`int`	`getFetchRetries()`
`void`	`setFetchRetries(int value)`
`Duration`	`getLastTimeout()`
`Instant`	`getModifiedTime()`
`void`	`setModifiedTime(@NotNull() Instant value)`
`Instant`	`getPrevModifiedTime()`
`void`	`setPrevModifiedTime(@NotNull() Instant value)`
`Instant`	`sniffModifiedTime()`
`String`	`getFetchTimeHistory(@NotNull() String defaultValue)`
`void`	`updateFetchTimeHistory(@NotNull() Instant fetchTime)`	***************************************************************************Parsing****************************************************************************
`Instant`	`getFirstFetchTime()`	Get the first fetch time
`String`	`getNamespace()`	namespace : metadata, seed, wwwreserved
`void`	`setNamespace(String ns)`	reserved
`PageCategory`	`getPageCategory()`	getPageCategory.
`void`	`setPageCategory(@NotNull() PageCategory pageCategory)`	category : index, detail, review, media, search, etc
`void`	`setPageCategory(@NotNull() OpenPageCategory pageCategory)`
`String`	`getEncoding()`	getEncoding.
`void`	`setEncoding(@Nullable() String encoding)`	setEncoding.
`String`	`getEncodingOrDefault(@NotNull() String defaultEncoding)`	Get content encodingContent encoding is detected just before it's parsed
`String`	`getEncodingClues()`	getEncodingClues.
`void`	`setEncodingClues(@NotNull() String clues)`	setEncodingClues.
`ByteBuffer`	`getContent()`	The entire raw document content e.g.
`ByteBuffer`	`getPersistContent()`	Get the uncached content
`Array<byte>`	`getContentAsBytes()`	Get content as bytes, the underling buffer is duplicated
`String`	`getContentAsString()`	TODO: Encoding is always UTF-8?
`ByteArrayInputStream`	`getContentAsInputStream()`	Get the page content as input stream
`InputSource`	`getContentAsSaxInputSource()`	Get the page content as sax input source
`void`	`setContent(@Nullable() String value)`	Set the page content
`void`	`setContent(@Nullable() Array<byte> value)`	Set the page content
`void`	`setContent(@Nullable() ByteBuffer value)`	Set the page content
`void`	`clearPersistContent()`
`long`	`getContentLength()`	TODO: check consistency with HttpHeaders.
`long`	`getPersistContentLength()`
`long`	`getLastContentBytes()`
`long`	`getAveContentBytes()`	getAveContentBytes.
`String`	`getContentType()`	getContentType.
`void`	`setContentType(String value)`	setContentType.
`ByteBuffer`	`getPrevSignature()`	getPrevSignature.
`void`	`setPrevSignature(@Nullable() ByteBuffer value)`	setPrevSignature.
`String`	`getPrevSignatureAsString()`	getPrevSignatureAsString.
`String`	`getProxy()`	getProxy.
`void`	`setProxy(@Nullable() String proxy)`	setProxy.
`ActiveDomStatus`	`getActiveDomStatus()`
`void`	`setActiveDomStatus(ActiveDomStatus s)`
`Map<String, ActiveDomStat>`	`getActiveDomStats()`
`void`	`setActiveDomStats(@NotNull() Map<String, ActiveDomStat> stats)`
`ActiveDomUrls`	`getActiveDomUrls()`
`void`	`setActiveDomUrls(@NotNull() ActiveDomUrls urls)`	setActiveDomUrls.
`ByteBuffer`	`getSignature()`	An implementation of a WebPage's signature from which it can be identified and referenced at any point in time.This is essentially the WebPage's fingerprint representing its state for any point in time.
`void`	`setSignature(Array<byte> value)`	setSignature.
`String`	`getSignatureAsString()`	getSignatureAsString.
`String`	`getPageTitle()`
`void`	`setPageTitle(String pageTitle)`
`String`	`getContentTitle()`
`void`	`setContentTitle(String contentTitle)`
`String`	`sniffTitle()`
`String`	`getPageText()`
`void`	`setPageText(String value)`
`String`	`getContentText()`
`void`	`setContentText(String textContent)`
`int`	`getContentTextLen()`
`void`	`setTextCascaded(String text)`
`ParseStatus`	`getParseStatus()`
`void`	`setParseStatus(ParseStatus parseStatus)`
`Map<CharSequence, GHypeLink>`	`getLiveLinks()`
`Collection<String>`	`getSimpleLiveLinks()`
`void`	`setLiveLinks(Iterable<HyperlinkPersistable> liveLinks)`	TODO: Remove redundant url to reduce space
`void`	`setLiveLinks(Map<CharSequence, GHypeLink> links)`
`void`	`addLiveLink(HyperlinkPersistable hyperLink)`
`Map<CharSequence, CharSequence>`	`getVividLinks()`
`Collection<String>`	`getSimpleVividLinks()`
`void`	`setVividLinks(Map<CharSequence, CharSequence> links)`
`List<CharSequence>`	`getDeadLinks()`
`void`	`setDeadLinks(List<CharSequence> deadLinks)`
`List<CharSequence>`	`getLinks()`
`void`	`setLinks(List<CharSequence> links)`
`void`	`addHyperlinks(Iterable<HyperlinkPersistable> hypeLinks)`	Record all links appeared in a pageThe links are in FIFO order, for each time we fetch and parse a page,we push newly discovered links to the queue, if the queue is full, we drop out some old ones,usually they do not appears in the page any more.
`void`	`addLinks(Iterable<CharSequence> hypeLinks)`	addLinks.
`int`	`getImpreciseLinkCount()`	getImpreciseLinkCount.
`void`	`setImpreciseLinkCount(int count)`	setImpreciseLinkCount.
`void`	`increaseImpreciseLinkCount(int count)`	increaseImpreciseLinkCount.
`Map<CharSequence, CharSequence>`	`getInlinks()`	getInlinks.
`CharSequence`	`getAnchor()`	getAnchor.
`void`	`setAnchor(CharSequence anchor)`	Anchor can be used to sniff article title
`Array<String>`	`getInlinkAnchors()`
`void`	`setInlinkAnchors(Collection<CharSequence> anchors)`
`int`	`getAnchorOrder()`
`void`	`setAnchorOrder(int order)`
`Instant`	`getContentPublishTime()`
`void`	`setContentPublishTime(Instant publishTime)`
`boolean`	`updateContentPublishTime(Instant newPublishTime)`
`Instant`	`getPrevContentPublishTime()`
`void`	`setPrevContentPublishTime(Instant publishTime)`
`Instant`	`getRefContentPublishTime()`
`void`	`setRefContentPublishTime(Instant publishTime)`
`Instant`	`getContentModifiedTime()`
`void`	`setContentModifiedTime(Instant modifiedTime)`
`Instant`	`getPrevContentModifiedTime()`
`void`	`setPrevContentModifiedTime(Instant modifiedTime)`
`boolean`	`updateContentModifiedTime(Instant newModifiedTime)`
`Instant`	`getPrevRefContentPublishTime()`
`void`	`setPrevRefContentPublishTime(Instant publishTime)`
`boolean`	`updateRefContentPublishTime(Instant newRefPublishTime)`
`String`	`getReferrer()`	getReferrer.
`void`	`setReferrer(String referrer)`	setReferrer.
`PageModel`	`getPageModel()`	***************************************************************************Page Model****************************************************************************
`float`	`getScore()`	***************************************************************************Scoring****************************************************************************
`void`	`setScore(float value)`	setScore.
`float`	`getContentScore()`	getContentScore.
`void`	`setContentScore(float score)`	setContentScore.
`String`	`getSortScore()`	getSortScore.
`void`	`setSortScore(String score)`	setSortScore.
`float`	`getCash()`	getCash.
`void`	`setCash(float cash)`	setCash.
`PageCounters`	`getPageCounters()`	getPageCounters.
`String`	`getIndexTimeHistory(String defaultValue)`	***************************************************************************Index****************************************************************************
`void`	`putIndexTimeHistory(Instant indexTime)`	putIndexTimeHistory.
`Instant`	`getFirstIndexTime(Instant defaultValue)`	getFirstIndexTime.
`int`	`hashCode()`
`int`	`compareTo(@NotNull() WebPage o)`
`boolean`	`equals(Object other)`
`String`	`toString()`

Methods inherited from class java.lang.Comparable
compareTo
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Method Detail
  - getUrl
```
@NotNull() String getUrl()
```
    page.location is the last working address, and page.url is the permanent internal address
  - getReversedUrl
```
@NotNull() String getReversedUrl()
```
    Getter for the field reversedUrl.
  - getConf
```
@NotNull() VolatileConfig getConf()
```
  - setConf
```
 void setConf(@NotNull() VolatileConfig conf)
```
  - getVariables
```
@NotNull() Variables getVariables()
```
    *****************************************************************************Common fields******************************************************************************
  - isCached
```
 boolean isCached()
```
  - isLoaded
```
 boolean isLoaded()
```
  - isFetched
```
 boolean isFetched()
```
  - isContentUpdated
```
 boolean isContentUpdated()
```
  - getTmpContent
```
@Nullable() ByteBuffer getTmpContent()
```
    Get the cached content
  - setTmpContent
```
 void setTmpContent(ByteBuffer tmpContent)
```
    Set the cached content, keep the page content unmodified
  - getArgs
```
@NotNull() String getArgs()
```
    The load arguments is variant task by task, so the local version is the first choice,while the persisted version is used for historical check only
  - setArgs
```
 void setArgs(@NotNull() String args)
```
    Set the local args variable and the persist version, and also clear the load options.
  - newWebPage
```
@NotNull() static WebPage newWebPage(@NotNull() String url, @NotNull() VolatileConfig conf)
```
    newWebPage.
    
    Parameters:
    
    url - a java.lang.String object.
    
    conf - a ai.platon.pulsar.common.config.VolatileConfig object.
  - newTestWebPage
```
@NotNull() static WebPage newTestWebPage(@NotNull() String url)
```
    newWebPage.
    
    Parameters:
    
    url - a java.lang.String object.
  - newWebPage
```
@NotNull() static WebPage newWebPage(@NotNull() String url, @NotNull() VolatileConfig conf, @Nullable() String href)
```
    newWebPage.
    
    Parameters:
    
    url - a java.lang.String object.
    
    conf - a ai.platon.pulsar.common.config.VolatileConfig object.
  - newInternalPage
```
@NotNull() static WebPage newInternalPage(@NotNull() String url)
```
    newInternalPage.
    
    Parameters:
    
    url - a java.lang.String object.
  - newInternalPage
```
@NotNull() static WebPage newInternalPage(@NotNull() String url, @NotNull() String title)
```
    newInternalPage.
    
    Parameters:
    
    url - a java.lang.String object.
    
    title - a java.lang.String object.
  - newInternalPage
```
@NotNull() static WebPage newInternalPage(@NotNull() String url, @NotNull() String title, @NotNull() String content)
```
    newInternalPage.
    
    Parameters:
    
    url - a java.lang.String object.
    
    title - a java.lang.String object.
    
    content - a java.lang.String object.
  - newInternalPage
```
@NotNull() static WebPage newInternalPage(@NotNull() String url, int id, @NotNull() String title, @NotNull() String content)
```
    newInternalPage.
    
    Parameters:
    
    url - a java.lang.String object.
    
    id - a int.
    
    title - a java.lang.String object.
    
    content - a java.lang.String object.
  - box
```
@NotNull() static WebPage box(@NotNull() String url, @NotNull() String reversedUrl, @NotNull() GWebPage page, @NotNull() VolatileConfig conf)
```
    Initialize a WebPage with the underlying GWebPage instance.
    
    Parameters:
    
    url - a java.lang.String object.
    
    reversedUrl - a java.lang.String object.
    
    page - a ai.platon.pulsar.persist.gora.generated.GWebPage object.
  - box
```
@NotNull() static WebPage box(@NotNull() String url, @NotNull() GWebPage page, @NotNull() VolatileConfig conf)
```
    Initialize a WebPage with the underlying GWebPage instance.
    
    Parameters:
    
    url - a java.lang.String object.
    
    page - a ai.platon.pulsar.persist.gora.generated.GWebPage object.
  - box
```
@NotNull() static WebPage box(@NotNull() String url, @NotNull() GWebPage page, boolean urlReversed, @NotNull() VolatileConfig conf)
```
    Initialize a WebPage with the underlying GWebPage instance.
    
    Parameters:
    
    url - a java.lang.String object.
    
    page - a ai.platon.pulsar.persist.gora.generated.GWebPage object.
    
    urlReversed - a boolean.
  - wrapKey
```
@NotNull() static Utf8 wrapKey(@NotNull() Mark mark)
```
    *****************************************************************************Other******************************************************************************
    
    Parameters:
    
    mark - a ai.platon.pulsar.persist.metadata.Mark object.
  - u8
```
@Nullable() static Utf8 u8(@Nullable() String value)
```
    What's the difference between String and Utf8?
    
    Parameters:
    
    value - a java.lang.String object.
  - getKey
```
@NotNull() String getKey()
```
    getKey.
  - getHref
```
@Nullable() String getHref()
```
    Get The hypertext reference of this page.It defines the address of the document, which this time is linked from
    TODO: use a seperate field to hold href
  - setHref
```
 void setHref(@Nullable() String href)
```
    Set The hypertext reference of this page.It defines the address of the document, which this time is linked from
    
    Parameters:
    
    href - The hypertext reference
  - isNil
```
 boolean isNil()
```
    isNil.
  - isNotNil
```
 boolean isNotNil()
```
    isNotNil.
  - isInternal
```
 boolean isInternal()
```
    isInternal.
  - isNotInternal
```
 boolean isNotInternal()
```
    isNotInternal.
  - unbox
```
@NotNull() GWebPage unbox()
```
    unbox.
  - unsafeSetGPage
```
 void unsafeSetGPage(@NotNull() GWebPage page)
```
  - unsafeCloneGPage
```
 void unsafeCloneGPage(WebPage page)
```
  - hasVar
```
 boolean hasVar(@NotNull() String name)
```
    Check if the page scope temporary variable with name {@name} exist
    
    Parameters:
    
    name - The variable name to check
  - getVar
```
 Object getVar(@NotNull() String name)
```
    Get a page scope temporary variable
    
    Parameters:
    
    name - a String object.
  - removeVar
```
 Object removeVar(@NotNull() String name)
```
    getAndRemoveVar.
    
    Parameters:
    
    name - a java.lang.String object.
  - setVar
```
 void setVar(@NotNull() String name, @NotNull() Object value)
```
    Get a page scope temporary variable
    
    Parameters:
    
    name - The variable name.
    
    value - The variable value.
  - setCached
```
 void setCached(boolean cached)
```
  - setLoaded
```
 void setLoaded(boolean loaded)
```
  - setFetched
```
 void setFetched(boolean fetched)
```
  - getMaxRetries
```
 int getMaxRetries()
```
  - getMetadata
```
 Metadata getMetadata()
```
  - getMarks
```
 CrawlMarks getMarks()
```
  - hasMark
```
 boolean hasMark(Mark mark)
```
  - getConfiguredUrl
```
@NotNull() String getConfiguredUrl()
```
    getConfiguredUrl.
  - getFetchedLinkCount
```
 int getFetchedLinkCount()
```
  - setFetchedLinkCount
```
 void setFetchedLinkCount(int count)
```
  - getZoneId
```
@NotNull() ZoneId getZoneId()
```
  - setZoneId
```
 void setZoneId(@NotNull() ZoneId zoneId)
```
  - getBatchId
```
 String getBatchId()
```
  - setBatchId
```
 void setBatchId(String value)
```
  - markSeed
```
 void markSeed()
```
  - unmarkSeed
```
 void unmarkSeed()
```
  - isSeed
```
 boolean isSeed()
```
  - getDistance
```
 int getDistance()
```
  - setDistance
```
 void setDistance(int newDistance)
```
  - updateDistance
```
 void updateDistance(int newDistance)
```
  - getFetchMode
```
@NotNull() FetchMode getFetchMode()
```
  - setFetchMode
```
 void setFetchMode(@NotNull() FetchMode mode)
```
    Fetch mode is used to determine the protocol before fetch, so it shall be set before fetch
  - getLastBrowser
```
@NotNull() BrowserType getLastBrowser()
```
  - setLastBrowser
```
 void setLastBrowser(@NotNull() BrowserType browser)
```
  - getHtmlIntegrity
```
@NotNull() HtmlIntegrity getHtmlIntegrity()
```
  - setHtmlIntegrity
```
 void setHtmlIntegrity(@NotNull() HtmlIntegrity integrity)
```
  - getFetchPriority
```
 int getFetchPriority()
```
  - setFetchPriority
```
 void setFetchPriority(int priority)
```
  - sniffFetchPriority
```
 int sniffFetchPriority()
```
  - getCreateTime
```
@NotNull() Instant getCreateTime()
```
  - setCreateTime
```
 void setCreateTime(@NotNull() Instant createTime)
```
  - getGenerateTime
```
@NotNull() Instant getGenerateTime()
```
  - setGenerateTime
```
 void setGenerateTime(@NotNull() Instant generateTime)
```
  - getModelSyncTime
```
@Nullable() Instant getModelSyncTime()
```
  - setModelSyncTime
```
 void setModelSyncTime(@Nullable() Instant modelSyncTime)
```
  - getFetchCount
```
 int getFetchCount()
```
  - setFetchCount
```
 void setFetchCount(int count)
```
  - updateFetchCount
```
 void updateFetchCount()
```
  - getCrawlStatus
```
@NotNull() CrawlStatus getCrawlStatus()
```
  - setCrawlStatus
```
 void setCrawlStatus(@NotNull() CrawlStatus crawlStatus)
```
  - setCrawlStatus
```
 void setCrawlStatus(int value)
```
  - getBaseUrl
```
 String getBaseUrl()
```
    The baseUrl is as the same as Location
    A baseUrl has the same semantic with Jsoup.parse:
  - getLocation
```
 String getLocation()
```
    WebPage.url is the permanent internal address, it might not still available to access the target.And WebPage.location or WebPage.baseUrl is the last working address, it might redirect to url,or it might have additional random parameters.WebPage.location may be different from url, it's generally normalized.
  - setLocation
```
 void setLocation(@NotNull() String value)
```
    The url is the permanent internal address, it might not still available to access the target.
    Location is the last working address, it might redirect to url, or it might have additional random parameters.
    Location may be different from url, it's generally normalized.
    
    Parameters:
    
    value - The location.
  - getFetchTime
```
@NotNull() Instant getFetchTime()
```
    The latest fetch time
  - setFetchTime
```
 void setFetchTime(@NotNull() Instant time)
```
    The latest fetch time
    
    Parameters:
    
    time - The latest fetch time
  - getPrevFetchTime
```
@NotNull() Instant getPrevFetchTime()
```
    The previous fetch time, updated at the fetch stage
  - setPrevFetchTime
```
 void setPrevFetchTime(@NotNull() Instant time)
```
  - getPrevCrawlTime1
```
@NotNull() Instant getPrevCrawlTime1()
```
    The previous crawl time, used for fat link crawl, which means both the page itself and out pages are fetched
  - setPrevCrawlTime1
```
 void setPrevCrawlTime1(@NotNull() Instant time)
```
    The previous crawl time, used for fat link crawl, which means both the page itself and out pages are fetched
  - getFetchInterval
```
@NotNull() Duration getFetchInterval()
```
  - setFetchInterval
```
 void setFetchInterval(@NotNull() Duration duration)
```
  - setFetchInterval
```
 void setFetchInterval(long interval)
```
  - setFetchInterval
```
 void setFetchInterval(float interval)
```
  - getProtocolStatus
```
@NotNull() ProtocolStatus getProtocolStatus()
```
  - setProtocolStatus
```
 void setProtocolStatus(@NotNull() ProtocolStatus protocolStatus)
```
  - getHeaders
```
@NotNull() ProtocolHeaders getHeaders()
```
    Header information returned from the web server used to server the content which is subsequently fetched from.This includes keys such asTRANSFER_ENCODING,CONTENT_ENCODING,CONTENT_LANGUAGE,CONTENT_LENGTH,CONTENT_LOCATION,CONTENT_DISPOSITION,CONTENT_MD5,CONTENT_TYPE,LAST_MODIFIEDand LOCATION.
  - getReprUrl
```
@NotNull() String getReprUrl()
```
  - setReprUrl
```
 void setReprUrl(@NotNull() String value)
```
  - getFetchRetries
```
 int getFetchRetries()
```
  - setFetchRetries
```
 void setFetchRetries(int value)
```
  - getLastTimeout
```
@NotNull() Duration getLastTimeout()
```
  - getModifiedTime
```
@NotNull() Instant getModifiedTime()
```
  - setModifiedTime
```
 void setModifiedTime(@NotNull() Instant value)
```
  - getPrevModifiedTime
```
@NotNull() Instant getPrevModifiedTime()
```
  - setPrevModifiedTime
```
 void setPrevModifiedTime(@NotNull() Instant value)
```
  - sniffModifiedTime
```
@NotNull() Instant sniffModifiedTime()
```
  - getFetchTimeHistory
```
@NotNull() String getFetchTimeHistory(@NotNull() String defaultValue)
```
  - updateFetchTimeHistory
```
 void updateFetchTimeHistory(@NotNull() Instant fetchTime)
```
    *****************************************************************************Parsing******************************************************************************
  - getFirstFetchTime
```
@Nullable() Instant getFirstFetchTime()
```
    Get the first fetch time
  - getNamespace
```
@Nullable() String getNamespace()
```
    namespace : metadata, seed, wwwreserved
  - setNamespace
```
 void setNamespace(String ns)
```
    reserved
    
    Parameters:
    
    ns - a java.lang.String object.
  - getPageCategory
```
@NotNull() PageCategory getPageCategory()
```
    getPageCategory.
  - setPageCategory
```
 void setPageCategory(@NotNull() PageCategory pageCategory)
```
    category : index, detail, review, media, search, etc
    
    Parameters:
    
    pageCategory - a ai.platon.pulsar.persist.metadata.PageCategory object.
  - setPageCategory
```
 void setPageCategory(@NotNull() OpenPageCategory pageCategory)
```
  - getEncoding
```
@Nullable() String getEncoding()
```
    getEncoding.
  - setEncoding
```
 void setEncoding(@Nullable() String encoding)
```
    setEncoding.
    
    Parameters:
    
    encoding - a java.lang.String object.
  - getEncodingOrDefault
```
@NotNull() String getEncodingOrDefault(@NotNull() String defaultEncoding)
```
    Get content encodingContent encoding is detected just before it's parsed
    
    Parameters:
    
    defaultEncoding - a java.lang.String object.
  - getEncodingClues
```
@NotNull() String getEncodingClues()
```
    getEncodingClues.
  - setEncodingClues
```
 void setEncodingClues(@NotNull() String clues)
```
    setEncodingClues.
    
    Parameters:
    
    clues - a java.lang.String object.
  - getContent
```
@Nullable() ByteBuffer getContent()
```
    The entire raw document content e.g. raw XHTML
  - getPersistContent
```
@Nullable() ByteBuffer getPersistContent()
```
    Get the uncached content
  - getContentAsBytes
```
@NotNull() Array<byte> getContentAsBytes()
```
    Get content as bytes, the underling buffer is duplicated
  - getContentAsString
```
@NotNull() String getContentAsString()
```
    TODO: Encoding is always UTF-8?
    Get the page content as a string
  - getContentAsInputStream
```
@NotNull() ByteArrayInputStream getContentAsInputStream()
```
    Get the page content as input stream
  - getContentAsSaxInputSource
```
@NotNull() InputSource getContentAsSaxInputSource()
```
    Get the page content as sax input source
  - setContent
```
 void setContent(@Nullable() String value)
```
    Set the page content
  - setContent
```
 void setContent(@Nullable() Array<byte> value)
```
    Set the page content
  - setContent
```
 void setContent(@Nullable() ByteBuffer value)
```
    Set the page content
    
    Parameters:
    
    value - a ByteBuffer.
  - clearPersistContent
```
 void clearPersistContent()
```
  - getContentLength
```
 long getContentLength()
```
    TODO: check consistency with HttpHeaders.CONTENT_LENGTH
  - getPersistContentLength
```
 long getPersistContentLength()
```
  - getLastContentBytes
```
 long getLastContentBytes()
```
  - getAveContentBytes
```
 long getAveContentBytes()
```
    getAveContentBytes.
  - getContentType
```
@NotNull() String getContentType()
```
    getContentType.
  - setContentType
```
 void setContentType(String value)
```
    setContentType.
    
    Parameters:
    
    value - a java.lang.String object.
  - getPrevSignature
```
@Nullable() ByteBuffer getPrevSignature()
```
    getPrevSignature.
  - setPrevSignature
```
 void setPrevSignature(@Nullable() ByteBuffer value)
```
    setPrevSignature.
    
    Parameters:
    
    value - a java.nio.ByteBuffer object.
  - getPrevSignatureAsString
```
@NotNull() String getPrevSignatureAsString()
```
    getPrevSignatureAsString.
  - getProxy
```
@Nullable() String getProxy()
```
    getProxy.
  - setProxy
```
 void setProxy(@Nullable() String proxy)
```
    setProxy.
    
    Parameters:
    
    proxy - a java.lang.String object.
  - getActiveDomStatus
```
@Nullable() ActiveDomStatus getActiveDomStatus()
```
  - setActiveDomStatus
```
 void setActiveDomStatus(ActiveDomStatus s)
```
  - getActiveDomStats
```
@NotNull() Map<String, ActiveDomStat> getActiveDomStats()
```
  - setActiveDomStats
```
 void setActiveDomStats(@NotNull() Map<String, ActiveDomStat> stats)
```
  - getActiveDomUrls
```
@NotNull() ActiveDomUrls getActiveDomUrls()
```
  - setActiveDomUrls
```
 void setActiveDomUrls(@NotNull() ActiveDomUrls urls)
```
    setActiveDomUrls.
    
    Parameters:
    
    urls - a ai.platon.pulsar.persist.model.ActiveDomUrls object.
  - getSignature
```
@Nullable() ByteBuffer getSignature()
```
    An implementation of a WebPage's signature from which it can be identified and referenced at any point in time.This is essentially the WebPage's fingerprint representing its state for any point in time.
  - setSignature
```
 void setSignature(Array<byte> value)
```
    setSignature.
    
    Parameters:
    
    value - an array of objects.
  - getSignatureAsString
```
@NotNull() String getSignatureAsString()
```
    getSignatureAsString.
  - getPageTitle
```
@NotNull() String getPageTitle()
```
  - setPageTitle
```
 void setPageTitle(String pageTitle)
```
  - getContentTitle
```
@NotNull() String getContentTitle()
```
  - setContentTitle
```
 void setContentTitle(String contentTitle)
```
  - sniffTitle
```
@NotNull() String sniffTitle()
```
  - getPageText
```
@NotNull() String getPageText()
```
  - setPageText
```
 void setPageText(String value)
```
  - getContentText
```
@NotNull() String getContentText()
```
  - setContentText
```
 void setContentText(String textContent)
```
  - getContentTextLen
```
 int getContentTextLen()
```
  - setTextCascaded
```
 void setTextCascaded(String text)
```
  - getParseStatus
```
@NotNull() ParseStatus getParseStatus()
```
  - setParseStatus
```
 void setParseStatus(ParseStatus parseStatus)
```
  - getLiveLinks
```
 Map<CharSequence, GHypeLink> getLiveLinks()
```
  - getSimpleLiveLinks
```
 Collection<String> getSimpleLiveLinks()
```
  - setLiveLinks
```
 void setLiveLinks(Iterable<HyperlinkPersistable> liveLinks)
```
    TODO: Remove redundant url to reduce space
    
    Parameters:
    
    liveLinks - a java.lang.Iterable object.
  - setLiveLinks
```
 void setLiveLinks(Map<CharSequence, GHypeLink> links)
```
  - addLiveLink
```
 void addLiveLink(HyperlinkPersistable hyperLink)
```
  - getVividLinks
```
 Map<CharSequence, CharSequence> getVividLinks()
```
  - getSimpleVividLinks
```
 Collection<String> getSimpleVividLinks()
```
  - setVividLinks
```
 void setVividLinks(Map<CharSequence, CharSequence> links)
```
  - getDeadLinks
```
 List<CharSequence> getDeadLinks()
```
  - setDeadLinks
```
 void setDeadLinks(List<CharSequence> deadLinks)
```
  - getLinks
```
 List<CharSequence> getLinks()
```
  - setLinks
```
 void setLinks(List<CharSequence> links)
```
  - addHyperlinks
```
 void addHyperlinks(Iterable<HyperlinkPersistable> hypeLinks)
```
    Record all links appeared in a pageThe links are in FIFO order, for each time we fetch and parse a page,we push newly discovered links to the queue, if the queue is full, we drop out some old ones,usually they do not appears in the page any more.
    TODO: compress linksTODO: HBase seems not modify any nested array
    
    Parameters:
    
    hypeLinks - a java.lang.Iterable object.
  - addLinks
```
 void addLinks(Iterable<CharSequence> hypeLinks)
```
    addLinks.
    
    Parameters:
    
    hypeLinks - a java.lang.Iterable object.
  - getImpreciseLinkCount
```
 int getImpreciseLinkCount()
```
    getImpreciseLinkCount.
  - setImpreciseLinkCount
```
 void setImpreciseLinkCount(int count)
```
    setImpreciseLinkCount.
    
    Parameters:
    
    count - a int.
  - increaseImpreciseLinkCount
```
 void increaseImpreciseLinkCount(int count)
```
    increaseImpreciseLinkCount.
    
    Parameters:
    
    count - a int.
  - getInlinks
```
 Map<CharSequence, CharSequence> getInlinks()
```
    getInlinks.
  - getAnchor
```
@NotNull() CharSequence getAnchor()
```
    getAnchor.
  - setAnchor
```
 void setAnchor(CharSequence anchor)
```
    Anchor can be used to sniff article title
    
    Parameters:
    
    anchor - a java.lang.CharSequence object.
  - getInlinkAnchors
```
 Array<String> getInlinkAnchors()
```
  - setInlinkAnchors
```
 void setInlinkAnchors(Collection<CharSequence> anchors)
```
  - getAnchorOrder
```
 int getAnchorOrder()
```
  - setAnchorOrder
```
 void setAnchorOrder(int order)
```
  - getContentPublishTime
```
 Instant getContentPublishTime()
```
  - setContentPublishTime
```
 void setContentPublishTime(Instant publishTime)
```
  - updateContentPublishTime
```
 boolean updateContentPublishTime(Instant newPublishTime)
```
  - getPrevContentPublishTime
```
 Instant getPrevContentPublishTime()
```
  - setPrevContentPublishTime
```
 void setPrevContentPublishTime(Instant publishTime)
```
  - getRefContentPublishTime
```
 Instant getRefContentPublishTime()
```
  - setRefContentPublishTime
```
 void setRefContentPublishTime(Instant publishTime)
```
  - getContentModifiedTime
```
 Instant getContentModifiedTime()
```
  - setContentModifiedTime
```
 void setContentModifiedTime(Instant modifiedTime)
```
  - getPrevContentModifiedTime
```
 Instant getPrevContentModifiedTime()
```
  - setPrevContentModifiedTime
```
 void setPrevContentModifiedTime(Instant modifiedTime)
```
  - updateContentModifiedTime
```
 boolean updateContentModifiedTime(Instant newModifiedTime)
```
  - getPrevRefContentPublishTime
```
 Instant getPrevRefContentPublishTime()
```
  - setPrevRefContentPublishTime
```
 void setPrevRefContentPublishTime(Instant publishTime)
```
  - updateRefContentPublishTime
```
 boolean updateRefContentPublishTime(Instant newRefPublishTime)
```
  - getReferrer
```
@NotNull() String getReferrer()
```
    getReferrer.
  - setReferrer
```
 void setReferrer(String referrer)
```
    setReferrer.
    
    Parameters:
    
    referrer - a java.lang.String object.
  - getPageModel
```
@NotNull() PageModel getPageModel()
```
    *****************************************************************************Page Model******************************************************************************
  - getScore
```
 float getScore()
```
    *****************************************************************************Scoring******************************************************************************
  - setScore
```
 void setScore(float value)
```
    setScore.
    
    Parameters:
    
    value - a float.
  - getContentScore
```
 float getContentScore()
```
    getContentScore.
  - setContentScore
```
 void setContentScore(float score)
```
    setContentScore.
    
    Parameters:
    
    score - a float.
  - getSortScore
```
@NotNull() String getSortScore()
```
    getSortScore.
  - setSortScore
```
 void setSortScore(String score)
```
    setSortScore.
    
    Parameters:
    
    score - a java.lang.String object.
  - getCash
```
 float getCash()
```
    getCash.
  - setCash
```
 void setCash(float cash)
```
    setCash.
    
    Parameters:
    
    cash - a float.
  - getPageCounters
```
@NotNull() PageCounters getPageCounters()
```
    getPageCounters.
  - getIndexTimeHistory
```
 String getIndexTimeHistory(String defaultValue)
```
    *****************************************************************************Index******************************************************************************
    
    Parameters:
    
    defaultValue - a java.lang.String object.
  - putIndexTimeHistory
```
 void putIndexTimeHistory(Instant indexTime)
```
    putIndexTimeHistory.
    
    Parameters:
    
    indexTime - a java.time.Instant object.
  - getFirstIndexTime
```
 Instant getFirstIndexTime(Instant defaultValue)
```
    getFirstIndexTime.
    
    Parameters:
    
    defaultValue - a java.time.Instant object.
  - hashCode
```
 int hashCode()
```
  - compareTo
```
 int compareTo(@NotNull() WebPage o)
```
  - equals
```
 boolean equals(Object other)
```
  - toString
```
 String toString()
```

Class WebPage

Field Summary

Method Summary

Methods inherited from class java.lang.Comparable

Methods inherited from class java.lang.Object

Method Detail

getUrl

getReversedUrl

getConf

setConf

getVariables

isCached

isLoaded

isFetched

isContentUpdated

getTmpContent

setTmpContent

getArgs

setArgs

newWebPage

newTestWebPage

newWebPage

newInternalPage

newInternalPage

newInternalPage

newInternalPage

box

box

box

wrapKey

u8

getKey

getHref

setHref

isNil

isNotNil

isInternal

isNotInternal

unbox

unsafeSetGPage

unsafeCloneGPage

hasVar

getVar

removeVar

setVar

setCached

setLoaded

setFetched

getMaxRetries

getMetadata

getMarks

hasMark

getConfiguredUrl

getFetchedLinkCount

setFetchedLinkCount

getZoneId

setZoneId

getBatchId

setBatchId

markSeed

unmarkSeed

isSeed

getDistance

setDistance

updateDistance

getFetchMode

setFetchMode

getLastBrowser

setLastBrowser

getHtmlIntegrity

setHtmlIntegrity

getFetchPriority

setFetchPriority

sniffFetchPriority

getCreateTime

setCreateTime

getGenerateTime

setGenerateTime

getModelSyncTime

setModelSyncTime