org.apache.nifi.processor.util.list

Class AbstractListProcessor<T extends ListableEntity>

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processor.util.list.AbstractListProcessor<T>

All Implemented Interfaces:

ConfigurableComponent, Processor

@TriggerSerially
 @Stateful(scopes={LOCAL,CLUSTER},
          description="After a listing of resources is performed, the latest timestamp of any of the resources is stored in the component\'s state. The scope used depends on the implementation.")
public abstract class AbstractListProcessor<T extends ListableEntity>
extends AbstractProcessor

An Abstract Processor that is intended to simplify the coding required in order to perform Listing operations of remote or local resources. Those resources may be files, "objects", "messages", or any other sort of entity that may need to be listed in such a way that we identity the entity only once. Each of these objects, messages, etc. is referred to as an "entity" for the scope of this Processor.

This class is responsible for triggering the listing to occur, filtering the results returned such that only new (unlisted) entities or entities that have been modified will be emitted from the Processor.

In order to make use of this abstract class, the entities listed must meet the following criteria:

• Entity must have a timestamp associated with it. This timestamp is used to determine if entities are "new" or not. Any entity that is returned by the listing will be considered "new" if the timestamp is later than the latest timestamp pulled.
• If the timestamp of an entity is before OR equal to the latest timestamp pulled, then the entity is not considered new. If the timestamp is later than the last timestamp pulled, then the entity is considered new.
• With 'Tracking Entities' strategy, the size of entity content is also used to determine if an entity is "new". If the size changes the entity is considered "new".
• Entity must have a user-readable name that can be used for logging purposes.

This class persists state across restarts so that even if NiFi is restarted, duplicates will not be pulled from the target system given the above criteria. This is performed using the StateManager. This allows the system to be restarted and begin processing where it left off. The state that is stored is the latest timestamp that has been pulled (as determined by the timestamps of the entities that are returned). See the section above for information about how this information isused in order to determine new entities.

NOTE: This processor performs migrations of legacy state mechanisms inclusive of locally stored, file-based state and the optional utilization of the Distributed Cache Service property to the new StateManager functionality. Upon successful migration, the associated data from one or both of the legacy mechanisms is purged.

For each new entity that is listed, the Processor will send a FlowFile to the 'success' relationship. The FlowFile will have no content but will have some set of attributes (defined by the concrete implementation) that can be used to fetch those resources or interact with them in whatever way makes sense for the configured dataflow.

Subclasses are responsible for the following:

• Perform a listing of resources. The subclass will implement the performListing(ProcessContext, Long) method, which creates a listing of all entities on the target system that have timestamps later than the provided timestamp. If the entities returned have a timestamp before the provided one, those entities will be filtered out. It is therefore not necessary to perform the filtering of timestamps but is provided in order to give the implementation the ability to filter those resources on the server side rather than pulling back all of the information, if it makes sense to do so in the concrete implementation.
• Creating a Map of attributes that are applicable for an entity. The attributes that are assigned to each FlowFile are exactly those returned by the createAttributes(ListableEntity, ProcessContext).
• Returning the configured path. Many resources can be comprised of a "path" (or a "container" or "bucket", etc.) as well as name or identifier that is unique only within that path. The getPath(ProcessContext) method is responsible for returning the path that is currently being polled for entities. If this does concept does not apply for the concrete implementation, it is recommended that the concrete implementation return "." or "/" for all invocations of this method.
• Determining when the listing must be cleared. It is sometimes necessary to clear state about which entities have already been ingested, as the result of a user changing a property value. The isListingResetNecessary(PropertyDescriptor) method is responsible for determining when the listing needs to be reset by returning a boolean indicating whether or not a change in the value of the provided property should trigger the timestamp and identifier information to be cleared.
• Provide the target system timestamp precision. By either letting user to choose the right one by adding TARGET_SYSTEM_TIMESTAMP_PRECISION to the return value of getSupportedPropertyDescriptors method or, overriding getDefaultTimePrecision method in case the target system has a fixed time precision.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	AbstractListProcessor.StringSerDe

Field Summary

Fields

Modifier and Type	Field and Description
static AllowableValue	BY_ENTITIES
static AllowableValue	BY_TIMESTAMPS
static PropertyDescriptor	DISTRIBUTED_CACHE_SERVICE
(package private) static String	IDENTIFIER_PREFIX
private boolean	justElectedPrimaryNode
(package private) static String	LAST_PROCESSED_LATEST_ENTRY_TIMESTAMP_KEY
private Long	lastListedLatestEntryTimestampMillis Represents the timestamp of an entity which was the latest one within those listed at the previous cycle.
private Long	lastProcessedLatestEntryTimestampMillis Represents the timestamp of an entity which was the latest one within those picked up and written to the output relationship at the previous cycle.
private Long	lastRunTimeNanos
(package private) static String	LATEST_LISTED_ENTRY_TIMESTAMP_KEY
private List<String>	latestIdentifiersProcessed
private ListedEntityTracker<T>	listedEntityTracker
static Map<TimeUnit,Long>	LISTING_LAG_MILLIS
static PropertyDescriptor	LISTING_STRATEGY
static AllowableValue	PRECISION_AUTO_DETECT
static AllowableValue	PRECISION_MILLIS
static AllowableValue	PRECISION_MINUTES
static AllowableValue	PRECISION_SECONDS
static Relationship	REL_SUCCESS
private boolean	resetEntityTrackingState
private boolean	resetState
static PropertyDescriptor	TARGET_SYSTEM_TIMESTAMP_PRECISION

Constructor Summary

Constructors

Constructor and Description
AbstractListProcessor()

Method Summary

Modifier and Type	Method and Description
protected abstract Map<String,String>	createAttributes(T entity, ProcessContext context) Creates a Map of attributes that should be applied to the FlowFile to represent this entity.
protected ListedEntityTracker<T>	createListedEntityTracker()
protected Collection<ValidationResult>	customValidate(ValidationContext context) In order to add custom validation at sub-classes, implement customValidate(ValidationContext, Collection) method.
protected void	customValidate(ValidationContext validationContext, Collection<ValidationResult> validationResults) Sub-classes can add custom validation by implementing this method.
private EntityListing	deserialize(String serializedState)
protected String	getDefaultTimePrecision() This method is intended to be overridden by SubClasses those do not support TARGET_SYSTEM_TIMESTAMP_PRECISION property.
protected String	getKey(String directory)
protected abstract String	getPath(ProcessContext context) Returns the path to perform a listing on.
File	getPersistenceFile()
Set<Relationship>	getRelationships()
protected abstract Scope	getStateScope(PropertyContext context) Returns a Scope that specifies where the state should be managed for this Processor
void	initListedEntityTracker(ProcessContext context)
protected abstract boolean	isListingResetNecessary(PropertyDescriptor property) Determines whether or not the listing must be reset if the value of the given property is changed
private void	listByTrackingEntities(ProcessContext context, ProcessSession session)
void	listByTrackingTimestamps(ProcessContext context, ProcessSession session)
private void	migrateState(String path, DistributedMapCacheClient client, StateManager stateManager, Scope scope) This processor used to use the DistributedMapCacheClient in order to store cluster-wide state, before the introduction of the StateManager.
void	onPrimaryNodeChange(PrimaryNodeState newState)
void	onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue)
void	onTrigger(ProcessContext context, ProcessSession session)
protected abstract List<T>	performListing(ProcessContext context, Long minTimestamp) Performs a listing of the remote entities that can be pulled.
private void	persist(long latestListedEntryTimestampThisCycleMillis, long lastProcessedLatestEntryTimestampMillis, List<String> processedIdentifiesWithLatestTimestamp, StateManager stateManager, Scope scope)
private void	resetTimeStates()
void	updateState(ProcessContext context)

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, getSupportedPropertyDescriptors, hashCode, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, validate

Field Detail

DISTRIBUTED_CACHE_SERVICE

public static final PropertyDescriptor DISTRIBUTED_CACHE_SERVICE

PRECISION_AUTO_DETECT

public static final AllowableValue PRECISION_AUTO_DETECT

PRECISION_MILLIS

public static final AllowableValue PRECISION_MILLIS

PRECISION_SECONDS

public static final AllowableValue PRECISION_SECONDS

PRECISION_MINUTES

public static final AllowableValue PRECISION_MINUTES

TARGET_SYSTEM_TIMESTAMP_PRECISION

public static final PropertyDescriptor TARGET_SYSTEM_TIMESTAMP_PRECISION

REL_SUCCESS

public static final Relationship REL_SUCCESS

BY_TIMESTAMPS

public static final AllowableValue BY_TIMESTAMPS

BY_ENTITIES

public static final AllowableValue BY_ENTITIES

LISTING_STRATEGY

public static final PropertyDescriptor LISTING_STRATEGY

lastListedLatestEntryTimestampMillis

private volatile Long lastListedLatestEntryTimestampMillis

Represents the timestamp of an entity which was the latest one within those listed at the previous cycle. It does not necessary mean it has been processed as well. Whether it was processed or not depends on target system time precision and how old the entity timestamp was.

lastProcessedLatestEntryTimestampMillis

private volatile Long lastProcessedLatestEntryTimestampMillis

Represents the timestamp of an entity which was the latest one within those picked up and written to the output relationship at the previous cycle.

lastRunTimeNanos

private volatile Long lastRunTimeNanos

justElectedPrimaryNode

private volatile boolean justElectedPrimaryNode

resetState

private volatile boolean resetState

resetEntityTrackingState

private volatile boolean resetEntityTrackingState

latestIdentifiersProcessed

private volatile List<String> latestIdentifiersProcessed

listedEntityTracker

private volatile ListedEntityTracker<T extends ListableEntity> listedEntityTracker

LISTING_LAG_MILLIS

public static final Map<TimeUnit,Long> LISTING_LAG_MILLIS

LATEST_LISTED_ENTRY_TIMESTAMP_KEY

static final String LATEST_LISTED_ENTRY_TIMESTAMP_KEY

See Also:

Constant Field Values

LAST_PROCESSED_LATEST_ENTRY_TIMESTAMP_KEY

static final String LAST_PROCESSED_LATEST_ENTRY_TIMESTAMP_KEY

See Also:

Constant Field Values

IDENTIFIER_PREFIX

static final String IDENTIFIER_PREFIX

See Also:

Constant Field Values

Constructor Detail

AbstractListProcessor

public AbstractListProcessor()

Method Detail

getPersistenceFile

public File getPersistenceFile()

onPropertyModified

public void onPropertyModified(PropertyDescriptor descriptor,
                               String oldValue,
                               String newValue)

Specified by:

onPropertyModified in interface ConfigurableComponent

Overrides:

onPropertyModified in class AbstractConfigurableComponent

getRelationships

public Set<Relationship> getRelationships()

Specified by:

getRelationships in interface Processor

Overrides:

getRelationships in class AbstractSessionFactoryProcessor

customValidate

protected final Collection<ValidationResult> customValidate(ValidationContext context)

In order to add custom validation at sub-classes, implement customValidate(ValidationContext, Collection) method.

Overrides:

customValidate in class AbstractConfigurableComponent

customValidate

protected void customValidate(ValidationContext validationContext,
                              Collection<ValidationResult> validationResults)

Sub-classes can add custom validation by implementing this method.

Parameters:

validationContext - the validation context

validationResults - add custom validation result to this collection

onPrimaryNodeChange

@OnPrimaryNodeStateChange
public void onPrimaryNodeChange(PrimaryNodeState newState)

updateState

@OnScheduled
public final void updateState(ProcessContext context)
                                    throws IOException

Throws:

IOException

migrateState

private void migrateState(String path,
                          DistributedMapCacheClient client,
                          StateManager stateManager,
                          Scope scope)
                   throws IOException

This processor used to use the DistributedMapCacheClient in order to store cluster-wide state, before the introduction of the StateManager. This method will migrate state from that DistributedMapCacheClient, or from a local file, to the StateManager, if any state already exists. More specifically, this will extract out the relevant timestamp for when the processor last ran

Parameters:

path - the path to migrate state for

client - the DistributedMapCacheClient that is capable of obtaining the current state

stateManager - the StateManager to use in order to store the new state

scope - the scope to use

Throws:

IOException - if unable to retrieve or store the state

persist

private void persist(long latestListedEntryTimestampThisCycleMillis,
                     long lastProcessedLatestEntryTimestampMillis,
                     List<String> processedIdentifiesWithLatestTimestamp,
                     StateManager stateManager,
                     Scope scope)
              throws IOException

Throws:

IOException

getKey

protected String getKey(String directory)

deserialize

private EntityListing deserialize(String serializedState)
                           throws com.fasterxml.jackson.core.JsonParseException,
                                  com.fasterxml.jackson.databind.JsonMappingException,
                                  IOException

Throws:

com.fasterxml.jackson.core.JsonParseException

com.fasterxml.jackson.databind.JsonMappingException

IOException

onTrigger

public void onTrigger(ProcessContext context,
                      ProcessSession session)
               throws ProcessException

Specified by:

onTrigger in class AbstractProcessor

Throws:

ProcessException

listByTrackingTimestamps

public void listByTrackingTimestamps(ProcessContext context,
                                     ProcessSession session)
                              throws ProcessException

Throws:

ProcessException

getDefaultTimePrecision

protected String getDefaultTimePrecision()

This method is intended to be overridden by SubClasses those do not support TARGET_SYSTEM_TIMESTAMP_PRECISION property. So that it use return different precisions than PRECISION_AUTO_DETECT. If TARGET_SYSTEM_TIMESTAMP_PRECISION is supported as a valid Processor property, then PRECISION_AUTO_DETECT will be the default value when not specified by a user.

Returns:

resetTimeStates

private void resetTimeStates()

createAttributes

protected abstract Map<String,String> createAttributes(T entity,
                                                       ProcessContext context)

Creates a Map of attributes that should be applied to the FlowFile to represent this entity. This processor will emit a FlowFile for each "new" entity (see the documentation for this class for a discussion of how this class determines whether or not an entity is "new"). The FlowFile will contain no content. The attributes that will be included are exactly the attributes that are returned by this method.

Parameters:

entity - the entity represented by the FlowFile

context - the ProcessContext for obtaining configuration information

Returns:

a Map of attributes for this entity

getPath

protected abstract String getPath(ProcessContext context)

Returns the path to perform a listing on. Many resources can be comprised of a "path" (or a "container" or "bucket", etc.) as well as name or identifier that is unique only within that path. This method is responsible for returning the path that is currently being polled for entities. If this does concept does not apply for the concrete implementation, it is recommended that the concrete implementation return "." or "/" for all invocations of this method.

Parameters:

context - the ProcessContex to use in order to obtain configuration

Returns:

the path that is to be used to perform the listing, or null if not applicable.

performListing

protected abstract List<T> performListing(ProcessContext context,
                                          Long minTimestamp)
                                   throws IOException

Performs a listing of the remote entities that can be pulled. If any entity that is returned has already been "discovered" or "emitted" by this Processor, it will be ignored. A discussion of how the Processor determines those entities that have already been emitted is provided above in the documentation for this class. Any entity that is returned by this method with a timestamp prior to the minTimestamp will be filtered out by the Processor. Therefore, it is not necessary that implementations perform this filtering but can be more efficient if the filtering can be performed on the server side prior to retrieving the information.

Parameters:

context - the ProcessContex to use in order to pull the appropriate entities

minTimestamp - the minimum timestamp of entities that should be returned.

Returns:

a Listing of entities that have a timestamp >= minTimestamp

Throws:

IOException

isListingResetNecessary

protected abstract boolean isListingResetNecessary(PropertyDescriptor property)

Determines whether or not the listing must be reset if the value of the given property is changed

Parameters:

property - the property that has changed

Returns:

true if a change in value of the given property necessitates that the listing be reset, false otherwise.

getStateScope

protected abstract Scope getStateScope(PropertyContext context)

Returns a Scope that specifies where the state should be managed for this Processor

Parameters:

context - the ProcessContext to use in order to make a determination

Returns:

a Scope that specifies where the state should be managed for this Processor

initListedEntityTracker

@OnScheduled
public void initListedEntityTracker(ProcessContext context)

createListedEntityTracker

protected ListedEntityTracker<T> createListedEntityTracker()

listByTrackingEntities

private void listByTrackingEntities(ProcessContext context,
                                    ProcessSession session)
                             throws ProcessException

Throws:

ProcessException