Package ai.platon.pulsar.common.collect
Interface ExternalUrlLoader
-
- All Implemented Interfaces:
public interface ExternalUrlLoaderA url loader loads urls from external sources, such as files, databases.
-
-
Method Summary
Modifier and Type Method Description abstract Unitreset()Force the loading time to expire abstract Unitsave(UrlAware url, UrlTopic topic)Save the url to the external repository abstract UnitsaveAll(Iterable<UrlAware> urls, UrlTopic topic)Save all the url to the external repository abstract BooleanhasMore()If there are more items in the source abstract BooleanhasMore(UrlTopic topic)If there are more items with topic in the source abstract IntegercountRemaining()Count remaining size, this operation might be slow for a large url database, use estimateRemaining for fast counting. abstract IntegercountRemaining(UrlTopic topic)Count remaining size with topic, this operation might be slow for a large url database, use estimateRemaining for fast counting. abstract IntegerestimateRemaining()Estimate the size of remaining items, this operation should be very fast abstract IntegerestimateRemaining(UrlTopic topic)Estimate the size of remaining items, this operation should be very fast abstract Collection<UrlAware>loadToNow(Collection<UrlAware> sink, Integer size, UrlTopic topic)Load items from the source to the sink immediately abstract <T extends Any> Collection<T>loadToNow(Collection<T> sink, Integer size, UrlTopic topic, Function1<UrlAware, T> transformer)Load items from the source to the sink immediately abstract UnitloadTo(Collection<UrlAware> sink, Integer size, UrlTopic topic)Load items from the source to the sink safely abstract <T extends Any> UnitloadTo(Collection<T> sink, Integer size, UrlTopic topic, Function1<UrlAware, T> transformer)Load items from the source to the sink safely abstract LongdeleteAll(UrlTopic topic)Delete all items with topic from the source -
-
Method Detail
-
saveAll
abstract Unit saveAll(Iterable<UrlAware> urls, UrlTopic topic)
Save all the url to the external repository
-
countRemaining
abstract Integer countRemaining()
Count remaining size, this operation might be slow for a large url database, use estimateRemaining for fast counting.
-
countRemaining
abstract Integer countRemaining(UrlTopic topic)
Count remaining size with topic, this operation might be slow for a large url database, use estimateRemaining for fast counting.
-
estimateRemaining
abstract Integer estimateRemaining()
Estimate the size of remaining items, this operation should be very fast
-
estimateRemaining
abstract Integer estimateRemaining(UrlTopic topic)
Estimate the size of remaining items, this operation should be very fast
-
loadToNow
abstract Collection<UrlAware> loadToNow(Collection<UrlAware> sink, Integer size, UrlTopic topic)
Load items from the source to the sink immediately
-
loadToNow
abstract <T extends Any> Collection<T> loadToNow(Collection<T> sink, Integer size, UrlTopic topic, Function1<UrlAware, T> transformer)
Load items from the source to the sink immediately
-
loadTo
abstract Unit loadTo(Collection<UrlAware> sink, Integer size, UrlTopic topic)
Load items from the source to the sink safely
-
loadTo
abstract <T extends Any> Unit loadTo(Collection<T> sink, Integer size, UrlTopic topic, Function1<UrlAware, T> transformer)
Load items from the source to the sink safely
-
-
-
-