org.apache.nifi.processor.util.listen

Class AbstractListenEventBatchingProcessor<E extends Event>

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processor.util.listen.AbstractListenEventProcessor<E>

org.apache.nifi.processor.util.listen.AbstractListenEventBatchingProcessor<E>

Type Parameters:

E - the type of Event

All Implemented Interfaces:

ConfigurableComponent, Processor

public abstract class AbstractListenEventBatchingProcessor<E extends Event>
extends AbstractListenEventProcessor<E>

An abstract processor that extends from AbstractListenEventProcessor and adds common functionality for batching events into a single FlowFile.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	AbstractListenEventBatchingProcessor.FlowFileEventBatch Wrapper to hold a FlowFile and the events that have been appended to it.

Field Summary

Fields

Modifier and Type	Field and Description
static PropertyDescriptor	MAX_BATCH_SIZE
static PropertyDescriptor	MESSAGE_DELIMITER
protected byte[]	messageDemarcatorBytes

Fields inherited from class org.apache.nifi.processor.util.listen.AbstractListenEventProcessor

charset, CHARSET, descriptors, dispatcher, errorEvents, events, MAX_CONNECTIONS, MAX_MESSAGE_QUEUE_SIZE, MAX_SOCKET_BUFFER_SIZE, POLL_TIMEOUT_MS, port, PORT, RECV_BUFFER_SIZE, REL_SUCCESS, relationships

Constructor Summary

Constructors

Constructor and Description
AbstractListenEventBatchingProcessor()

Method Summary

Modifier and Type	Method and Description
protected abstract Map<String,String>	getAttributes(AbstractListenEventBatchingProcessor.FlowFileEventBatch batch) Creates the attributes for the FlowFile of the given batch.
protected Map<String,AbstractListenEventBatchingProcessor.FlowFileEventBatch>	getBatches(ProcessSession session, int totalBatchSize, byte[] messageDemarcatorBytes) Batches together up to the batchSize events.
protected String	getBatchKey(E event)
protected abstract String	getTransitUri(AbstractListenEventBatchingProcessor.FlowFileEventBatch batch) Creates the transit uri to be used when reporting a provenance receive event for the given batch.
protected void	init(ProcessorInitializationContext context)
void	onScheduled(ProcessContext context)
void	onTrigger(ProcessContext context, ProcessSession session)
protected void	postProcess(ProcessContext context, ProcessSession session, List<E> events) Called at the end of onTrigger to allow sub-classes to take post processing action on the events

Methods inherited from class org.apache.nifi.processor.util.listen.AbstractListenEventProcessor

createBufferPool, createDispatcher, getAdditionalProperties, getAdditionalRelationships, getDispatcherPort, getErrorQueueSize, getLongPollTimeout, getMessage, getQueueSize, getRelationships, getSupportedPropertyDescriptors, onUnscheduled

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

customValidate, equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

Field Detail

MAX_BATCH_SIZE

public static final PropertyDescriptor MAX_BATCH_SIZE

MESSAGE_DELIMITER

public static final PropertyDescriptor MESSAGE_DELIMITER

messageDemarcatorBytes

protected volatile byte[] messageDemarcatorBytes

Constructor Detail

AbstractListenEventBatchingProcessor

public AbstractListenEventBatchingProcessor()

Method Detail

init

protected void init(ProcessorInitializationContext context)

Overrides:

init in class AbstractListenEventProcessor<E extends Event>

onScheduled

@OnScheduled
public void onScheduled(ProcessContext context)
                              throws IOException

Overrides:

onScheduled in class AbstractListenEventProcessor<E extends Event>

Throws:

IOException

onTrigger

public void onTrigger(ProcessContext context,
                      ProcessSession session)
               throws ProcessException

Specified by:

onTrigger in class AbstractProcessor

Throws:

ProcessException

getAttributes

protected abstract Map<String,String> getAttributes(AbstractListenEventBatchingProcessor.FlowFileEventBatch batch)

Creates the attributes for the FlowFile of the given batch.

Parameters:

batch - the current batch

Returns:

the Map of FlowFile attributes

getTransitUri

protected abstract String getTransitUri(AbstractListenEventBatchingProcessor.FlowFileEventBatch batch)

Creates the transit uri to be used when reporting a provenance receive event for the given batch.

Parameters:

batch - the current batch

Returns:

the transit uri string

postProcess

protected void postProcess(ProcessContext context,
                           ProcessSession session,
                           List<E> events)

Called at the end of onTrigger to allow sub-classes to take post processing action on the events

Parameters:

context - the current context

session - the current session

events - the list of all events processed by the current execution of onTrigger

getBatches

protected Map<String,AbstractListenEventBatchingProcessor.FlowFileEventBatch> getBatches(ProcessSession session,
                                                                                         int totalBatchSize,
                                                                                         byte[] messageDemarcatorBytes)

Batches together up to the batchSize events. Events are grouped together based on a batch key which by default is the sender of the event, but can be override by sub-classes. This method will return when batchSize has been reached, or when no more events are available on the queue.

Parameters:

session - the current session

totalBatchSize - the total number of events to process

messageDemarcatorBytes - the demarcator to put between messages when writing to a FlowFile

Returns:

a Map from the batch key to the FlowFile and events for that batch, the size of events in all the batches will be <= batchSize

getBatchKey

protected String getBatchKey(E event)

Parameters:

event - an event that was pulled off the queue

Returns:

a key to use for batching events together, by default this uses the sender of the event, but sub-classes should override this to batch by something else