org.apache.samza.system.hdfs.reader

Class MultiFileHdfsReader

java.lang.Object

org.apache.samza.system.hdfs.reader.MultiFileHdfsReader

public class MultiFileHdfsReader
extends java.lang.Object

A wrapper on top of SingleFileHdfsReader to manage the situation of multiple files per partition. The offset for MultiFileHdfsReader, which is also the offset that gets committed in and used by Samza, consists of two parts: file index, actual offset within file. For example, 3:127 Format of the offset within file is defined by the implementation of SingleFileHdfsReader itself.

Constructor Summary

Constructors

Constructor and Description
MultiFileHdfsReader(HdfsReaderFactory.ReaderType readerType, org.apache.samza.system.SystemStreamPartition systemStreamPartition, java.util.List<java.lang.String> partitionDescriptors, java.lang.String offset)
MultiFileHdfsReader(HdfsReaderFactory.ReaderType readerType, org.apache.samza.system.SystemStreamPartition systemStreamPartition, java.util.List<java.lang.String> partitionDescriptors, java.lang.String offset, int numMaxRetries)

Method Summary

Modifier and Type	Method and Description
void	close()
static java.lang.String	generateOffset(int fileIndex, java.lang.String singleFileOffset) Generate the offset based on file index and offset within single file
static int	getCurFileIndex(java.lang.String offset) Get the current file index from the offset string
static java.lang.String	getCurSingleFileOffset(java.lang.String offset) Get the offset within file from the offset string
org.apache.samza.system.SystemStreamPartition	getSystemStreamPartition()
boolean	hasNext()
org.apache.samza.system.IncomingMessageEnvelope	readNext()
void	reconnect() Reconnect to the file systems in case of failure.
void	reconnect(java.lang.String offset) Reconnect to the file systems in case of failures.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiFileHdfsReader

public MultiFileHdfsReader(HdfsReaderFactory.ReaderType readerType,
                           org.apache.samza.system.SystemStreamPartition systemStreamPartition,
                           java.util.List<java.lang.String> partitionDescriptors,
                           java.lang.String offset)

MultiFileHdfsReader

public MultiFileHdfsReader(HdfsReaderFactory.ReaderType readerType,
                           org.apache.samza.system.SystemStreamPartition systemStreamPartition,
                           java.util.List<java.lang.String> partitionDescriptors,
                           java.lang.String offset,
                           int numMaxRetries)

Method Detail

getCurFileIndex

public static int getCurFileIndex(java.lang.String offset)

Get the current file index from the offset string

Parameters:

offset - offset string that contains both file index and offset within file

Returns:

the file index part

getCurSingleFileOffset

public static java.lang.String getCurSingleFileOffset(java.lang.String offset)

Get the offset within file from the offset string

Parameters:

offset - offset string that contains both file index and offset within file

Returns:

the single file offset part

generateOffset

public static java.lang.String generateOffset(int fileIndex,
                                              java.lang.String singleFileOffset)

Generate the offset based on file index and offset within single file

Parameters:

fileIndex - index of the file

singleFileOffset - offset within single file

Returns:

the complete offset

hasNext

public boolean hasNext()

readNext

public org.apache.samza.system.IncomingMessageEnvelope readNext()

reconnect

public void reconnect()

Reconnect to the file systems in case of failure. Reset offset to the last checkpoint (last successfully read message). Throw SamzaException if reaches max number of retries.

reconnect

public void reconnect(java.lang.String offset)

Reconnect to the file systems in case of failures.

Parameters:

offset - reset offset to the specified offset Throw SamzaException if reaches max number of retries.

close

public void close()

getSystemStreamPartition

public org.apache.samza.system.SystemStreamPartition getSystemStreamPartition()