All Implemented Interfaces:

kotlin.collections.Iterable
```
public final class UrlFeeder
 implements Iterable<UrlAware>
                    
```
The url feeder collects urls from the url pool and feed them to the crawlers.
The url feed collect urls using DataCollector, each DataCollector collect urls from exactly one UrlCache.
The user can register multiple UrlCaches and DataCollectors for different type of tasks.

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description

Field Summary

Fields
Modifier and Type	Field	Description
`private final Collection<PriorityDataCollector<UrlAware>>`	`openCollectors`
`private final List<PriorityDataCollector<UrlAware>>`	`collectors`
`private final ConcurrentLoadingIterable<UrlAware>`	`loadingIterable`
`private final Integer`	`cacheSize`
`private final Integer`	`size`
`private final Integer`	`estimatedSize`
`private final String`	`abstract`
`private final String`	`report`
`private final UrlPool`	`urlPool`
`private final Integer`	`lowerCacheSize`
`private final Boolean`	`enableDefaults`

Constructor Summary

Constructors
Constructor	Description
`UrlFeeder(UrlPool urlPool, Integer lowerCacheSize, Boolean enableDefaults)`

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`final Collection<PriorityDataCollector<UrlAware>>`	`getOpenCollectors()`
`final List<PriorityDataCollector<UrlAware>>`	`getCollectors()`
`final ConcurrentLoadingIterable<UrlAware>`	`getLoadingIterable()`	The loading iterable is a concurrent loading iterable, which loads urls from the url pool and feeds them to the crawlers.
`final Integer`	`getCacheSize()`
`final Integer`	`getSize()`
`final Integer`	`getEstimatedSize()`
`final String`	`getAbstract()`
`final String`	`getReport()`
`final UrlPool`	`getUrlPool()`
`final Integer`	`getLowerCacheSize()`
`final Boolean`	`getEnableDefaults()`
`final Boolean`	`isEmpty()`
`final Boolean`	`isNotEmpty()`
`final Unit`	`addFirst(UrlAware url)`	Add a hyperlink to the very beginning of the fetch queue, so it will be served first
`final Unit`	`addLast(UrlAware url)`	Add a hyperlink to the end of the fetch queue, so it will be served last
`Iterator<UrlAware>`	`iterator()`
`final Integer`	`estimatedOrder(Integer priority)`	Estimate the order to fetch for the next task to add with priority priority.
`final UrlFeeder`	`addDefaultCollectors()`
`final UrlFeeder`	`addCollector(PriorityDataCollector<UrlAware> collector)`
`final UrlFeeder`	`addCollectors(Iterable<PriorityDataCollector<UrlAware>> collectors)`
`final List<PriorityDataCollector<UrlAware>>`	`findByName(String name)`
`final List<PriorityDataCollector<UrlAware>>`	`findByName(Iterable<String> names)`
`final List<PriorityDataCollector<UrlAware>>`	`findByName(Regex regex)`
`final List<PriorityDataCollector<UrlAware>>`	`findByNameLike(String name)`
`final Boolean`	`remove(DataCollector<UrlAware> collector)`
`final Boolean`	`removeAll(Collection<DataCollector<UrlAware>> collectors)`
`final Unit`	`clear()`

Methods inherited from class kotlin.collections.Iterable
forEach, spliterator
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UrlFeeder

UrlFeeder(UrlPool urlPool, Integer lowerCacheSize, Boolean enableDefaults)

Method Detail

getOpenCollectors

 final Collection<PriorityDataCollector<UrlAware>> getOpenCollectors()

getCollectors

 final List<PriorityDataCollector<UrlAware>> getCollectors()

getLoadingIterable
```
 final ConcurrentLoadingIterable<UrlAware> getLoadingIterable()
```
The loading iterable is a concurrent loading iterable, which loads urls from the url pool and feeds them to the crawlers.

getCacheSize
```
 final Integer getCacheSize()
```

getSize
```
 final Integer getSize()
```

getEstimatedSize
```
 final Integer getEstimatedSize()
```

getAbstract
```
 final String getAbstract()
```

getReport
```
 final String getReport()
```

getUrlPool
```
 final UrlPool getUrlPool()
```

getLowerCacheSize
```
 final Integer getLowerCacheSize()
```

getEnableDefaults
```
 final Boolean getEnableDefaults()
```

isEmpty
```
 final Boolean isEmpty()
```

isNotEmpty
```
 final Boolean isNotEmpty()
```

addFirst
```
 final Unit addFirst(UrlAware url)
```
Add a hyperlink to the very beginning of the fetch queue, so it will be served first

addLast
```
 final Unit addLast(UrlAware url)
```
Add a hyperlink to the end of the fetch queue, so it will be served last

iterator
```
 Iterator<UrlAware> iterator()
```

estimatedOrder
```
 final Integer estimatedOrder(Integer priority)
```
Estimate the order to fetch for the next task to add with priority priority.

addDefaultCollectors
```
 final UrlFeeder addDefaultCollectors()
```

addCollector

 final UrlFeeder addCollector(PriorityDataCollector<UrlAware> collector)

addCollectors

 final UrlFeeder addCollectors(Iterable<PriorityDataCollector<UrlAware>> collectors)

findByName

 final List<PriorityDataCollector<UrlAware>> findByName(String name)

findByName

 final List<PriorityDataCollector<UrlAware>> findByName(Iterable<String> names)

findByName

 final List<PriorityDataCollector<UrlAware>> findByName(Regex regex)

findByNameLike

 final List<PriorityDataCollector<UrlAware>> findByNameLike(String name)

remove

 final Boolean remove(DataCollector<UrlAware> collector)

removeAll

 final Boolean removeAll(Collection<DataCollector<UrlAware>> collectors)

clear
```
 final Unit clear()
```

Class UrlFeeder

Nested Class Summary

Field Summary

Constructor Summary

Enum Constant Summary

Method Summary

Methods inherited from class kotlin.collections.Iterable

Methods inherited from class java.lang.Object

Constructor Detail

UrlFeeder

Method Detail

getOpenCollectors

getCollectors

getLoadingIterable

getCacheSize

getSize

getEstimatedSize

getAbstract

getReport

getUrlPool

getLowerCacheSize

getEnableDefaults

isEmpty

isNotEmpty

addFirst

addLast

iterator

estimatedOrder

addDefaultCollectors

addCollector

addCollectors

findByName

findByName

findByName

findByNameLike

remove

removeAll

clear