org.apache.samza.system.hdfs.descriptors

Class HdfsSystemDescriptor

java.lang.Object

org.apache.samza.system.descriptors.SystemDescriptor<HdfsSystemDescriptor>

org.apache.samza.system.hdfs.descriptors.HdfsSystemDescriptor

public class HdfsSystemDescriptor
extends org.apache.samza.system.descriptors.SystemDescriptor<HdfsSystemDescriptor>

A HdfsSystemDescriptor can be used for specifying Samza and HDFS-specific properties of a HDFS input/output system. It can also be used for obtaining HdfsInputDescriptors and HdfsOutputDescriptors, which can be used for specifying Samza and system-specific properties of HDFS input/output streams.

System properties provided in configuration override corresponding properties specified using a descriptor.

Constructor Summary

Constructors

Constructor and Description
HdfsSystemDescriptor(java.lang.String systemName)

Method Summary

Modifier and Type	Method and Description
HdfsInputDescriptor	getInputDescriptor(java.lang.String streamId) Gets an HdfsInputDescriptor for the input stream of this system.
HdfsOutputDescriptor	getOutputDescriptor(java.lang.String streamId) Gets an HdfsOutputDescriptor for the output stream of this system.
java.util.Map<java.lang.String,java.lang.String>	toConfig()
HdfsSystemDescriptor	withConsumerBlackList(java.lang.String blackList) Black list used by directory partitioner to filter out unwanted files in a hdfs directory.
HdfsSystemDescriptor	withConsumerBufferCapacity(long bufferCapacity) The capacity of the hdfs consumer buffer - the blocking queue used for storing messages.
HdfsSystemDescriptor	withConsumerGroupPattern(java.lang.String groupPattern) Group pattern used by directory partitioner for advanced partitioning.
HdfsSystemDescriptor	withConsumerNumMaxRetries(long maxRetries) Number of max retries for the hdfs consumer readers per partition.
HdfsSystemDescriptor	withConsumerWhiteList(java.lang.String whiteList) White list used by directory partitioner to filter out unwanted files in a hdfs directory.
HdfsSystemDescriptor	withDatePathFormat(java.lang.String datePathFormat) In an HdfsWriter implementation that performs time-based output bucketing, the user may configure a date format (suitable for inclusion in a file path) using SimpleDateFormat formatting that the Bucketer implementation will use to generate HDFS paths and filenames.
HdfsSystemDescriptor	withOutputBaseDir(java.lang.String outputBaseDir) The base output directory into which all HDFS output for this job will be written.
HdfsSystemDescriptor	withReaderType(java.lang.String readerType) The type of the file reader for consumer (avro, plain, etc.)
HdfsSystemDescriptor	withStagingDirectory(java.lang.String stagingDirectory) Staging directory for storing partition description.
HdfsSystemDescriptor	withWriteBatchSizeBytes(long writeBatchSizeBytes) Split output files from all writer tasks based on # of bytes written to optimize MapReduce utilization for Hadoop jobs that will process the data later.
HdfsSystemDescriptor	withWriteBatchSizeRecords(long writeBatchSizeRecords) Split output files from all writer tasks based on # of bytes written to optimize MapReduce utilization for Hadoop jobs that will process the data later.
HdfsSystemDescriptor	withWriteCompressionType(java.lang.String writeCompressionType) Simple, human-readable label for various compression options.
HdfsSystemDescriptor	withWriterClassName(java.lang.String writerClassName) The fully-qualified class name of the HdfsWriter subclass that will write for this system.

Methods inherited from class org.apache.samza.system.descriptors.SystemDescriptor

getExpander, getSystemName, getTransformer, withDefaultStreamConfigs, withDefaultStreamOffsetDefault, withSystemConfigs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HdfsSystemDescriptor

public HdfsSystemDescriptor(java.lang.String systemName)

Method Detail

getInputDescriptor

public HdfsInputDescriptor getInputDescriptor(java.lang.String streamId)

Gets an HdfsInputDescriptor for the input stream of this system.

The message in the stream has no key and the value type is determined by reader type.

Parameters:

streamId - id of the input stream

Returns:

an HdfsInputDescriptor for the hdfs input stream

getOutputDescriptor

public HdfsOutputDescriptor getOutputDescriptor(java.lang.String streamId)

Gets an HdfsOutputDescriptor for the output stream of this system.

The message in the stream has no key and the value type is determined by writer class.

Parameters:

streamId - id of the output stream

Returns:

an HdfsOutputDescriptor for the hdfs output stream

withDatePathFormat

public HdfsSystemDescriptor withDatePathFormat(java.lang.String datePathFormat)

In an HdfsWriter implementation that performs time-based output bucketing, the user may configure a date format (suitable for inclusion in a file path) using SimpleDateFormat formatting that the Bucketer implementation will use to generate HDFS paths and filenames. The more granular this date format, the more often a bucketing HdfsWriter will begin a new date-path bucket when creating the next output file.

Parameters:

datePathFormat - date path format

Returns:

this system descriptor

withOutputBaseDir

public HdfsSystemDescriptor withOutputBaseDir(java.lang.String outputBaseDir)

The base output directory into which all HDFS output for this job will be written.

Parameters:

outputBaseDir - output base directory

Returns:

this system descriptor

withWriteBatchSizeBytes

public HdfsSystemDescriptor withWriteBatchSizeBytes(long writeBatchSizeBytes)

Split output files from all writer tasks based on # of bytes written to optimize MapReduce utilization for Hadoop jobs that will process the data later.

Parameters:

writeBatchSizeBytes - write batch size in bytes.

Returns:

this system descriptor

withWriteBatchSizeRecords

public HdfsSystemDescriptor withWriteBatchSizeRecords(long writeBatchSizeRecords)

Split output files from all writer tasks based on # of bytes written to optimize MapReduce utilization for Hadoop jobs that will process the data later.

Parameters:

writeBatchSizeRecords - write batch size in records.

Returns:

this system descriptor

withWriteCompressionType

public HdfsSystemDescriptor withWriteCompressionType(java.lang.String writeCompressionType)

Simple, human-readable label for various compression options. HdfsWriter implementations can choose how to handle these individually, or throw an exception. Example: "none", "gzip", ...

Parameters:

writeCompressionType - compression type for writer.

Returns:

this system descriptor

withWriterClassName

public HdfsSystemDescriptor withWriterClassName(java.lang.String writerClassName)

The fully-qualified class name of the HdfsWriter subclass that will write for this system.

Parameters:

writerClassName - writer class name.

Returns:

this system descriptor

withConsumerBufferCapacity

public HdfsSystemDescriptor withConsumerBufferCapacity(long bufferCapacity)

The capacity of the hdfs consumer buffer - the blocking queue used for storing messages.

Parameters:

bufferCapacity - the buffer capacity for HDFS consumer.

Returns:

this system descriptor

withConsumerNumMaxRetries

public HdfsSystemDescriptor withConsumerNumMaxRetries(long maxRetries)

Number of max retries for the hdfs consumer readers per partition.

Parameters:

maxRetries - number of max retires for HDFS consumer.

Returns:

this system descriptor

withConsumerWhiteList

public HdfsSystemDescriptor withConsumerWhiteList(java.lang.String whiteList)

White list used by directory partitioner to filter out unwanted files in a hdfs directory.

Parameters:

whiteList - white list for HDFS consumer inputs.

Returns:

this system descriptor

withConsumerBlackList

public HdfsSystemDescriptor withConsumerBlackList(java.lang.String blackList)

Black list used by directory partitioner to filter out unwanted files in a hdfs directory.

Parameters:

blackList - black list for HDFS consumer inputs.

Returns:

this system descriptor

withConsumerGroupPattern

public HdfsSystemDescriptor withConsumerGroupPattern(java.lang.String groupPattern)

Group pattern used by directory partitioner for advanced partitioning.

Parameters:

groupPattern - group parttern for HDFS consumer inputs.

Returns:

this system descriptor

withReaderType

public HdfsSystemDescriptor withReaderType(java.lang.String readerType)

The type of the file reader for consumer (avro, plain, etc.)

Parameters:

readerType - reader type for HDFS consumer inputs.

Returns:

this system descriptor

withStagingDirectory

public HdfsSystemDescriptor withStagingDirectory(java.lang.String stagingDirectory)

Staging directory for storing partition description. If not set, will use the staging directory set by yarn job.

Parameters:

stagingDirectory - staging directory for HDFS consumer inputs.

Returns:

this system descriptor

toConfig

public java.util.Map<java.lang.String,java.lang.String> toConfig()

Overrides:

toConfig in class org.apache.samza.system.descriptors.SystemDescriptor<HdfsSystemDescriptor>