Package com.digitalpebble.stormcrawler.opensearch.bolt

Class IndexerBolt

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichBolt

com.digitalpebble.stormcrawler.indexing.AbstractIndexerBolt

com.digitalpebble.stormcrawler.opensearch.bolt.IndexerBolt

All Implemented Interfaces:

com.github.benmanes.caffeine.cache.RemovalListener<String,List<org.apache.storm.tuple.Tuple>>, Serializable, org.apache.storm.task.IBolt, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichBolt, org.opensearch.action.bulk.BulkProcessor.Listener

public class IndexerBolt
extends AbstractIndexerBolt
implements com.github.benmanes.caffeine.cache.RemovalListener<String,List<org.apache.storm.tuple.Tuple>>, org.opensearch.action.bulk.BulkProcessor.Listener

Sends documents to opensearch. Indexes all the fields from the tuples or a Map <String,Object> from a named field.

See Also:

• Serialized Form

Field Summary

Fields inherited from class com.digitalpebble.stormcrawler.indexing.AbstractIndexerBolt

canonicalMetadataParamName, ignoreEmptyFieldValueParamName, metadata2fieldParamName, metadataFilterParamName, textFieldParamName, textLengthParamName, urlFieldParamName

Constructor Summary

Constructors

Constructor	Description
IndexerBolt()
IndexerBolt(String indexName)	Sets the index name instead of taking it from the configuration.

Method Summary

Modifier and Type	Method	Description
void	afterBulk(long executionId, org.opensearch.action.bulk.BulkRequest request, Throwable failure)
void	afterBulk(long executionId, org.opensearch.action.bulk.BulkRequest request, org.opensearch.action.bulk.BulkResponse response)
void	beforeBulk(long executionId, org.opensearch.action.bulk.BulkRequest request)
void	cleanup()
void	execute(org.apache.storm.tuple.Tuple tuple)
protected String	getIndexName(Metadata m)	Must be overridden for implementing custom index names based on some metadata information By Default, indexName coming from config is used
void	onRemoval(@Nullable String key, @Nullable List<org.apache.storm.tuple.Tuple> value, @NotNull com.github.benmanes.caffeine.cache.RemovalCause cause)
void	prepare(Map<String,Object> conf, org.apache.storm.task.TopologyContext context, org.apache.storm.task.OutputCollector collector)

Methods inherited from class com.digitalpebble.stormcrawler.indexing.AbstractIndexerBolt

declareOutputFields, fieldNameForText, fieldNameForURL, filterDocument, filterMetadata, getDocumentID, ignoreEmptyFields, trimText, valueForURL

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Constructor Details

IndexerBolt

public IndexerBolt()

IndexerBolt

public IndexerBolt(String indexName)

Sets the index name instead of taking it from the configuration. *

Method Details

prepare

public void prepare(Map<String,Object> conf,
 org.apache.storm.task.TopologyContext context,
 org.apache.storm.task.OutputCollector collector)

Specified by:

prepare in interface org.apache.storm.task.IBolt

Overrides:

prepare in class AbstractIndexerBolt

onRemoval

public void onRemoval(@Nullable
 @Nullable String key,
 @Nullable
 @Nullable List<org.apache.storm.tuple.Tuple> value,
 @NotNull
 @NotNull com.github.benmanes.caffeine.cache.RemovalCause cause)

Specified by:

onRemoval in interface com.github.benmanes.caffeine.cache.RemovalListener<String,List<org.apache.storm.tuple.Tuple>>

cleanup

public void cleanup()

Specified by:

cleanup in interface org.apache.storm.task.IBolt

Overrides:

cleanup in class org.apache.storm.topology.base.BaseRichBolt

execute

public void execute(org.apache.storm.tuple.Tuple tuple)

Specified by:

execute in interface org.apache.storm.task.IBolt

getIndexName

protected String getIndexName(Metadata m)

Must be overridden for implementing custom index names based on some metadata information By Default, indexName coming from config is used

beforeBulk

public void beforeBulk(long executionId,
 org.opensearch.action.bulk.BulkRequest request)

Specified by:

beforeBulk in interface org.opensearch.action.bulk.BulkProcessor.Listener

afterBulk

public void afterBulk(long executionId,
 org.opensearch.action.bulk.BulkRequest request,
 org.opensearch.action.bulk.BulkResponse response)

Specified by:

afterBulk in interface org.opensearch.action.bulk.BulkProcessor.Listener

afterBulk

public void afterBulk(long executionId,
 org.opensearch.action.bulk.BulkRequest request,
 Throwable failure)

Specified by:

afterBulk in interface org.opensearch.action.bulk.BulkProcessor.Listener