Package com.digitalpebble.stormcrawler.opensearch.persistence

Class AggregationSpout

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichSpout

com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

com.digitalpebble.stormcrawler.opensearch.persistence.AbstractSpout

com.digitalpebble.stormcrawler.opensearch.persistence.AggregationSpout

All Implemented Interfaces:

Serializable, org.apache.storm.spout.ISpout, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichSpout, org.opensearch.action.ActionListener<org.opensearch.action.search.SearchResponse>

Direct Known Subclasses:

HybridSpout

public class AggregationSpout
extends AbstractSpout
implements org.opensearch.action.ActionListener<org.opensearch.action.search.SearchResponse>

Spout which pulls URL from an ES index. Use a single instance unless you use 'es.status.routing' with the StatusUpdaterBolt, in which case you need to have exactly the same number of spout instances as ES shards. Guarantees a good mix of URLs by aggregating them by an arbitrary field e.g. key.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

AbstractQueryingSpout.InProcessMap<K extends Object,V extends Object>

Field Summary

Fields

Modifier and Type	Field	Description
protected Set<String>	currentBuckets

Fields inherited from class com.digitalpebble.stormcrawler.opensearch.persistence.AbstractSpout

bucketSortField, client, filterQueries, indexName, logIdprefix, maxBucketNum, maxURLsPerBucket, OSBoltType, OSStatusBucketFieldParamName, OSStatusBucketSortFieldParamName, OSStatusFilterParamName, OSStatusGlobalSortFieldParamName, OSStatusIndexNameParamName, OSStatusMaxBucketParamName, OSStatusMaxURLsParamName, OSStatusQueryTimeoutParamName, partitionField, queryDate, queryTimeout, shardID, totalSortField

Fields inherited from class com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

_collector, beingProcessed, buffer, eventCounter, isInQuery, lastTimeResetToNOW, maxDelayBetweenQueries, minDelayBetweenQueries, queryTimes, resetFetchDateAfterNSecs, resetFetchDateParamName, StatusMaxDelayParamName, StatusMinDelayParamName, StatusTTLPurgatory

Constructor Summary

Constructors

Constructor	Description
AggregationSpout()

Method Summary

Modifier and Type	Method	Description
void	onFailure(Exception arg0)
void	onResponse(org.opensearch.action.search.SearchResponse response)
void	open(Map<String,Object> stormConf, org.apache.storm.task.TopologyContext context, org.apache.storm.spout.SpoutOutputCollector collector)
protected void	populateBuffer()	Builds a query and use it retrieve the results from OS *
protected void	sortValuesForKey(String key, Object[] sortValues)

Methods inherited from class com.digitalpebble.stormcrawler.opensearch.persistence.AbstractSpout

ack, addHitToBuffer, close, fail, fromKeyValues

Methods inherited from class com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

activate, deactivate, declareOutputFields, getTimeLastQuerySent, markQueryReceivedNow, nextTuple

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Field Details

currentBuckets

protected Set<String> currentBuckets

Constructor Details

AggregationSpout

public AggregationSpout()

Method Details

open

public void open(Map<String,Object> stormConf,
 org.apache.storm.task.TopologyContext context,
 org.apache.storm.spout.SpoutOutputCollector collector)

Specified by:

open in interface org.apache.storm.spout.ISpout

Overrides:

open in class AbstractSpout

populateBuffer

protected void populateBuffer()

Description copied from class: AbstractSpout

Builds a query and use it retrieve the results from OS *

Specified by:

populateBuffer in class AbstractSpout

onFailure

public void onFailure(Exception arg0)

Specified by:

onFailure in interface org.opensearch.action.ActionListener<org.opensearch.action.search.SearchResponse>

onResponse

public void onResponse(org.opensearch.action.search.SearchResponse response)

Specified by:

onResponse in interface org.opensearch.action.ActionListener<org.opensearch.action.search.SearchResponse>

sortValuesForKey

protected void sortValuesForKey(String key,
 Object[] sortValues)