org.apache.nifi.processors.hadoop

Class AbstractHadoopProcessor

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

All Implemented Interfaces:

ConfigurableComponent, Processor

@RequiresInstanceClassLoading(cloneAncestorResources=true)
public abstract class AbstractHadoopProcessor
extends AbstractProcessor

This is a base class that is helpful when building processors interacting with HDFS.

As of Apache NiFi 1.5.0, the Relogin Period property is no longer used in the configuration of a Hadoop processor. Due to changes made to SecurityUtil.loginKerberos(Configuration, String, String), which is used by this class to authenticate a principal with Kerberos, Hadoop components no longer attempt relogins explicitly. For more information, please read the documentation for SecurityUtil.loginKerberos(Configuration, String, String).

See Also:

SecurityUtil.loginKerberos(Configuration, String, String)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	AbstractHadoopProcessor.ExtendedConfiguration Extending Hadoop Configuration to prevent it from caching classes that can't be found.
protected static class	AbstractHadoopProcessor.HdfsResources
protected static class	AbstractHadoopProcessor.ValidationResources

Field Summary

Fields

Modifier and Type	Field and Description
static String	ABSOLUTE_HDFS_PATH_ATTRIBUTE
static PropertyDescriptor	ADDITIONAL_CLASSPATH_RESOURCES
private static String	ALLOW_EXPLICIT_KEYTAB
static PropertyDescriptor	COMPRESSION_CODEC
static PropertyDescriptor	DIRECTORY
private static AbstractHadoopProcessor.HdfsResources	EMPTY_HDFS_RESOURCES
static PropertyDescriptor	HADOOP_CONFIGURATION_RESOURCES
private AtomicReference<AbstractHadoopProcessor.HdfsResources>	hdfsResources
(package private) static PropertyDescriptor	KERBEROS_CREDENTIALS_SERVICE
static PropertyDescriptor	KERBEROS_RELOGIN_PERIOD
private File	kerberosConfigFile
protected KerberosProperties	kerberosProperties
protected List<PropertyDescriptor>	properties
private static Object	RESOURCES_LOCK
private AtomicReference<AbstractHadoopProcessor.ValidationResources>	validationResourceHolder

Constructor Summary

Constructors

Constructor and Description
AbstractHadoopProcessor()

Method Summary

Modifier and Type	Method and Description
void	abstractOnScheduled(ProcessContext context) If your subclass also has an @OnScheduled annotated method and you need hdfsResources in that method, then be sure to call super.abstractOnScheduled(context)
void	abstractOnStopped()
protected void	checkHdfsUriForTimeout(org.apache.hadoop.conf.Configuration config)
protected Collection<ValidationResult>	customValidate(ValidationContext validationContext)
protected org.apache.hadoop.io.compress.CompressionCodec	getCompressionCodec(ProcessContext context, org.apache.hadoop.conf.Configuration configuration) Returns the configured CompressionCodec, or null if none is configured.
protected org.apache.hadoop.conf.Configuration	getConfiguration()
private static org.apache.hadoop.conf.Configuration	getConfigurationFromResources(org.apache.hadoop.conf.Configuration config, String configResources)
protected org.apache.hadoop.fs.FileSystem	getFileSystem()
protected org.apache.hadoop.fs.FileSystem	getFileSystem(org.apache.hadoop.conf.Configuration config) This exists in order to allow unit tests to override it so that they don't take several minutes waiting for UDP packets to be received
protected org.apache.hadoop.fs.FileSystem	getFileSystemAsUser(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.security.UserGroupInformation ugi)
protected KerberosProperties	getKerberosProperties(File kerberosConfigFile)
static String	getPathDifference(org.apache.hadoop.fs.Path root, org.apache.hadoop.fs.Path child) Returns the relative path of the child that does not include the filename or the root path.
protected List<PropertyDescriptor>	getSupportedPropertyDescriptors()
protected org.apache.hadoop.security.UserGroupInformation	getUserGroupInformation()
protected void	init(ProcessorInitializationContext context)
private void	interruptStatisticsThread(org.apache.hadoop.fs.FileSystem fileSystem)
(package private) boolean	isAllowExplicitKeytab()
protected void	preProcessConfiguration(org.apache.hadoop.conf.Configuration config, ProcessContext context) This method will be called after the Configuration has been created, but before the FileSystem is created, allowing sub-classes to take further action on the Configuration before creating the FileSystem.
(package private) AbstractHadoopProcessor.HdfsResources	resetHDFSResources(String configResources, ProcessContext context)

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger, onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, getRelationships, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

Field Detail

ALLOW_EXPLICIT_KEYTAB

private static final String ALLOW_EXPLICIT_KEYTAB

See Also:

Constant Field Values

HADOOP_CONFIGURATION_RESOURCES

public static final PropertyDescriptor HADOOP_CONFIGURATION_RESOURCES

DIRECTORY

public static final PropertyDescriptor DIRECTORY

COMPRESSION_CODEC

public static final PropertyDescriptor COMPRESSION_CODEC

KERBEROS_RELOGIN_PERIOD

public static final PropertyDescriptor KERBEROS_RELOGIN_PERIOD

ADDITIONAL_CLASSPATH_RESOURCES

public static final PropertyDescriptor ADDITIONAL_CLASSPATH_RESOURCES

KERBEROS_CREDENTIALS_SERVICE

static final PropertyDescriptor KERBEROS_CREDENTIALS_SERVICE

ABSOLUTE_HDFS_PATH_ATTRIBUTE

public static final String ABSOLUTE_HDFS_PATH_ATTRIBUTE

See Also:

Constant Field Values

RESOURCES_LOCK

private static final Object RESOURCES_LOCK

EMPTY_HDFS_RESOURCES

private static final AbstractHadoopProcessor.HdfsResources EMPTY_HDFS_RESOURCES

kerberosProperties

protected KerberosProperties kerberosProperties

properties

protected List<PropertyDescriptor> properties

kerberosConfigFile

private volatile File kerberosConfigFile

hdfsResources

private final AtomicReference<AbstractHadoopProcessor.HdfsResources> hdfsResources

validationResourceHolder

private final AtomicReference<AbstractHadoopProcessor.ValidationResources> validationResourceHolder

Constructor Detail

AbstractHadoopProcessor

public AbstractHadoopProcessor()

Method Detail

init

protected void init(ProcessorInitializationContext context)

Overrides:

init in class AbstractSessionFactoryProcessor

getKerberosProperties

protected KerberosProperties getKerberosProperties(File kerberosConfigFile)

getSupportedPropertyDescriptors

protected List<PropertyDescriptor> getSupportedPropertyDescriptors()

Overrides:

getSupportedPropertyDescriptors in class AbstractConfigurableComponent

customValidate

protected Collection<ValidationResult> customValidate(ValidationContext validationContext)

Overrides:

customValidate in class AbstractConfigurableComponent

abstractOnScheduled

@OnScheduled
public final void abstractOnScheduled(ProcessContext context)
                                            throws IOException

If your subclass also has an @OnScheduled annotated method and you need hdfsResources in that method, then be sure to call super.abstractOnScheduled(context)

Throws:

IOException

abstractOnStopped

@OnStopped
public final void abstractOnStopped()

interruptStatisticsThread

private void interruptStatisticsThread(org.apache.hadoop.fs.FileSystem fileSystem)
                                throws NoSuchFieldException,
                                       IllegalAccessException

Throws:

NoSuchFieldException

IllegalAccessException

getConfigurationFromResources

private static org.apache.hadoop.conf.Configuration getConfigurationFromResources(org.apache.hadoop.conf.Configuration config,
                                                                                  String configResources)
                                                                           throws IOException

Throws:

IOException

resetHDFSResources

AbstractHadoopProcessor.HdfsResources resetHDFSResources(String configResources,
                                                         ProcessContext context)
                                                  throws IOException

Throws:

IOException

preProcessConfiguration

protected void preProcessConfiguration(org.apache.hadoop.conf.Configuration config,
                                       ProcessContext context)

This method will be called after the Configuration has been created, but before the FileSystem is created, allowing sub-classes to take further action on the Configuration before creating the FileSystem.

Parameters:

config - the Configuration that will be used to create the FileSystem

context - the context that can be used to retrieve additional values

getFileSystem

protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration config)
                                                 throws IOException

This exists in order to allow unit tests to override it so that they don't take several minutes waiting for UDP packets to be received

Parameters:

config - the configuration to use

Returns:

the FileSystem that is created for the given Configuration

Throws:

IOException - if unable to create the FileSystem

getFileSystemAsUser

protected org.apache.hadoop.fs.FileSystem getFileSystemAsUser(org.apache.hadoop.conf.Configuration config,
                                                              org.apache.hadoop.security.UserGroupInformation ugi)
                                                       throws IOException

Throws:

IOException

checkHdfsUriForTimeout

protected void checkHdfsUriForTimeout(org.apache.hadoop.conf.Configuration config)
                               throws IOException

Throws:

IOException

getCompressionCodec

protected org.apache.hadoop.io.compress.CompressionCodec getCompressionCodec(ProcessContext context,
                                                                             org.apache.hadoop.conf.Configuration configuration)

Returns the configured CompressionCodec, or null if none is configured.

Parameters:

context - the ProcessContext

configuration - the Hadoop Configuration

Returns:

CompressionCodec or null

getPathDifference

public static String getPathDifference(org.apache.hadoop.fs.Path root,
                                       org.apache.hadoop.fs.Path child)

Returns the relative path of the child that does not include the filename or the root path.

Parameters:

root - the path to relativize from

child - the path to relativize

Returns:

the relative path

getConfiguration

protected org.apache.hadoop.conf.Configuration getConfiguration()

getFileSystem

protected org.apache.hadoop.fs.FileSystem getFileSystem()

getUserGroupInformation

protected org.apache.hadoop.security.UserGroupInformation getUserGroupInformation()

isAllowExplicitKeytab

boolean isAllowExplicitKeytab()