Package com.digitalpebble.stormcrawler.opensearch.persistence

Class HybridSpout

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichSpout

com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

com.digitalpebble.stormcrawler.opensearch.persistence.AbstractSpout

com.digitalpebble.stormcrawler.opensearch.persistence.AggregationSpout

com.digitalpebble.stormcrawler.opensearch.persistence.HybridSpout

All Implemented Interfaces:

EmptyQueueListener, Serializable, org.apache.storm.spout.ISpout, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichSpout, org.opensearch.action.ActionListener<org.opensearch.action.search.SearchResponse>

public class HybridSpout
extends AggregationSpout
implements EmptyQueueListener

Uses collapsing spouts to get an initial set of URLs and keys to query for and gets emptyQueue notifications from the URLBuffer to query OpenSearch for a specific key.

Since:

1.15

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

AbstractQueryingSpout.InProcessMap<K extends Object,V extends Object>

Field Summary

Fields

Modifier and Type	Field	Description
protected static final String	RELOADPARAMNAME

Fields inherited from class com.digitalpebble.stormcrawler.opensearch.persistence.AggregationSpout

currentBuckets

Fields inherited from class com.digitalpebble.stormcrawler.opensearch.persistence.AbstractSpout

bucketSortField, client, filterQueries, indexName, logIdprefix, maxBucketNum, maxURLsPerBucket, OSBoltType, OSStatusBucketFieldParamName, OSStatusBucketSortFieldParamName, OSStatusFilterParamName, OSStatusGlobalSortFieldParamName, OSStatusIndexNameParamName, OSStatusMaxBucketParamName, OSStatusMaxURLsParamName, OSStatusQueryTimeoutParamName, partitionField, queryDate, queryTimeout, shardID, totalSortField

Fields inherited from class com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

_collector, beingProcessed, buffer, eventCounter, isInQuery, lastTimeResetToNOW, maxDelayBetweenQueries, minDelayBetweenQueries, queryTimes, resetFetchDateAfterNSecs, resetFetchDateParamName, StatusMaxDelayParamName, StatusMinDelayParamName, StatusTTLPurgatory

Constructor Summary

Constructors

Constructor	Description
HybridSpout()

Method Summary

Modifier and Type	Method	Description
void	emptyQueue(String queueName)
void	onResponse(org.opensearch.action.search.SearchResponse response)
void	open(Map<String,Object> stormConf, org.apache.storm.task.TopologyContext context, org.apache.storm.spout.SpoutOutputCollector collector)
protected void	sortValuesForKey(String key, Object[] sortValues)

Methods inherited from class com.digitalpebble.stormcrawler.opensearch.persistence.AggregationSpout

onFailure, populateBuffer

Methods inherited from class com.digitalpebble.stormcrawler.opensearch.persistence.AbstractSpout

ack, addHitToBuffer, close, fail, fromKeyValues

Methods inherited from class com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

activate, deactivate, declareOutputFields, getTimeLastQuerySent, markQueryReceivedNow, nextTuple

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Field Details

RELOADPARAMNAME

protected static final String RELOADPARAMNAME

See Also:

• Constant Field Values

Constructor Details

HybridSpout

public HybridSpout()

Method Details

open

public void open(Map<String,Object> stormConf,
 org.apache.storm.task.TopologyContext context,
 org.apache.storm.spout.SpoutOutputCollector collector)

Specified by:

open in interface org.apache.storm.spout.ISpout

Overrides:

open in class AggregationSpout

emptyQueue

public void emptyQueue(String queueName)

Specified by:

emptyQueue in interface EmptyQueueListener

onResponse

public void onResponse(org.opensearch.action.search.SearchResponse response)

Specified by:

onResponse in interface org.opensearch.action.ActionListener<org.opensearch.action.search.SearchResponse>

Overrides:

onResponse in class AggregationSpout

sortValuesForKey

protected void sortValuesForKey(String key,
 Object[] sortValues)

Overrides:

sortValuesForKey in class AggregationSpout