-
- All Implemented Interfaces:
-
ai.platon.pulsar.common.collect.collector.DataCollector,ai.platon.pulsar.common.collect.collector.PriorityDataCollector,kotlin.Comparable
public class UrlCacheCollector extends AbstractPriorityDataCollector<UrlAware>
-
-
Field Summary
Fields Modifier and Type Field Description private Stringnameprivate final Integersizeprivate final IntegerexternalSizeprivate final IntegerestimatedExternalSizeprivate final IntegerestimatedSizeprivate final UrlCacheurlCacheprivate final Integercapacityprivate final Integerpriorityprivate final Integeridprivate final Set<String>labelsprivate Stringlangprivate Stringcountryprivate Stringdistrictprivate IntegercollectCountprivate IntegercollectedCountprivate final InstantcreateTimeprivate InstantfirstCollectTimeprivate InstantlastCollectedTimeprivate InstantdeadTimeprivate final DurationcollectTimeprivate final BooleanisDead
-
Constructor Summary
Constructors Constructor Description UrlCacheCollector(UrlCache urlCache)
-
Method Summary
Modifier and Type Method Description StringgetName()The collector name UnitsetName(String name)The collector name IntegergetSize()IntegergetExternalSize()IntegergetEstimatedExternalSize()IntegergetEstimatedSize()final UrlCachegetUrlCache()IntegergetCapacity()The capacity IntegergetPriority()IntegergetId()The collector id Set<String>getLabels()The task labels StringgetLang()Required website language UnitsetLang(String lang)Required website language StringgetCountry()Required website country UnitsetCountry(String country)Required website country StringgetDistrict()Required website district UnitsetDistrict(String district)Required website district IntegergetCollectCount()The total count of collect attempt UnitsetCollectCount(Integer collectCount)The total count of collect attempt IntegergetCollectedCount()The total collected count UnitsetCollectedCount(Integer collectedCount)The total collected count InstantgetCreateTime()The create time of this collector InstantgetFirstCollectTime()The first collect time UnitsetFirstCollectTime(Instant firstCollectTime)The first collect time InstantgetLastCollectedTime()The last time to collect an item successfully UnitsetLastCollectedTime(Instant lastCollectedTime)The last time to collect an item successfully InstantgetDeadTime()The dead time of this collector, if the collector is dead, all items should be dropped UnitsetDeadTime(Instant deadTime)The dead time of this collector, if the collector is dead, all items should be dropped DurationgetCollectTime()BooleangetIsDead()BooleanhasMore()If the fetch cache is a LoadingurlCache, the items can be both in memory or in external source, so even if all queues are empty, hasMore can return true IntegercollectTo(List<UrlAware> sink)List<String>dump()Unitclear()UnitdeepClear()-
Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractPriorityDataCollector
collectTo, collectTo, collectTo, compareTo -
Methods inherited from class ai.platon.pulsar.common.collect.collector.UrlCacheCollector
toString -
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
-
Constructor Detail
-
UrlCacheCollector
UrlCacheCollector(UrlCache urlCache)
-
-
Method Detail
-
getExternalSize
Integer getExternalSize()
-
getEstimatedExternalSize
Integer getEstimatedExternalSize()
-
getEstimatedSize
Integer getEstimatedSize()
-
getUrlCache
final UrlCache getUrlCache()
-
getCapacity
Integer getCapacity()
The capacity
-
getPriority
Integer getPriority()
-
getCountry
String getCountry()
Required website country
-
setCountry
Unit setCountry(String country)
Required website country
-
getDistrict
String getDistrict()
Required website district
-
setDistrict
Unit setDistrict(String district)
Required website district
-
getCollectCount
Integer getCollectCount()
The total count of collect attempt
-
setCollectCount
Unit setCollectCount(Integer collectCount)
The total count of collect attempt
-
getCollectedCount
Integer getCollectedCount()
The total collected count
-
setCollectedCount
Unit setCollectedCount(Integer collectedCount)
The total collected count
-
getCreateTime
Instant getCreateTime()
The create time of this collector
-
getFirstCollectTime
Instant getFirstCollectTime()
The first collect time
-
setFirstCollectTime
Unit setFirstCollectTime(Instant firstCollectTime)
The first collect time
-
getLastCollectedTime
Instant getLastCollectedTime()
The last time to collect an item successfully
-
setLastCollectedTime
Unit setLastCollectedTime(Instant lastCollectedTime)
The last time to collect an item successfully
-
getDeadTime
Instant getDeadTime()
The dead time of this collector, if the collector is dead, all items should be dropped
-
setDeadTime
Unit setDeadTime(Instant deadTime)
The dead time of this collector, if the collector is dead, all items should be dropped
-
getCollectTime
Duration getCollectTime()
-
hasMore
@Synchronized() Boolean hasMore()
If the fetch cache is a LoadingurlCache, the items can be both in memory or in external source, so even if all queues are empty, hasMore can return true
-
collectTo
@Synchronized() Integer collectTo(List<UrlAware> sink)
-
dump
@Synchronized() List<String> dump()
-
clear
@Synchronized() Unit clear()
-
deepClear
@Synchronized() Unit deepClear()
-
-
-
-