org.apache.tez.dag.history.logging.proto

Class DatePartitionedLogger<T extends com.google.protobuf.MessageLite>

java.lang.Object

org.apache.tez.dag.history.logging.proto.DatePartitionedLogger<T>

Type Parameters:

T - The proto message type.

public class DatePartitionedLogger<T extends com.google.protobuf.MessageLite>
extends Object

Class to create proto reader and writer for a date partitioned directory structure.

Constructor Summary

Constructors

Constructor and Description
DatePartitionedLogger(com.google.protobuf.Parser<T> parser, org.apache.hadoop.fs.Path baseDir, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.util.Clock clock)

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.conf.Configuration	getConfig()
LocalDate	getDateFromDir(String dirName) Extract the date from the directory name, this should be a directory created by this class.
String	getDirForDate(LocalDate date) Returns the directory name for a given date.
String	getNextDirectory(String currentDir) Find next available directory, after the given directory.
LocalDateTime	getNow() Returns the current time, using the underlying clock in UTC time.
org.apache.hadoop.fs.Path	getPathForDate(LocalDate date, String fileName) Create a path for the given date and fileName.
org.apache.hadoop.fs.Path	getPathForSubdir(String dirName, String fileName)
ProtoMessageReader<T>	getReader(org.apache.hadoop.fs.Path filePath) Creates a reader for the given filePath, no validation is done.
ProtoMessageWriter<T>	getWriter(String fileName) Creates a writer for the given fileName, with date as today.
List<org.apache.hadoop.fs.FileStatus>	scanForChangedFiles(String subDir, Map<String,Long> currentOffsets) Returns new or changed files in the given directory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DatePartitionedLogger

public DatePartitionedLogger(com.google.protobuf.Parser<T> parser,
                             org.apache.hadoop.fs.Path baseDir,
                             org.apache.hadoop.conf.Configuration conf,
                             org.apache.hadoop.yarn.util.Clock clock)
                      throws IOException

Throws:

IOException

Method Detail

getWriter

public ProtoMessageWriter<T> getWriter(String fileName)
                                throws IOException

Creates a writer for the given fileName, with date as today.

Throws:

IOException

getReader

public ProtoMessageReader<T> getReader(org.apache.hadoop.fs.Path filePath)
                                throws IOException

Creates a reader for the given filePath, no validation is done.

Throws:

IOException

getPathForDate

public org.apache.hadoop.fs.Path getPathForDate(LocalDate date,
                                                String fileName)
                                         throws IOException

Create a path for the given date and fileName. This can be used to create a reader.

Throws:

IOException

getPathForSubdir

public org.apache.hadoop.fs.Path getPathForSubdir(String dirName,
                                                  String fileName)

getDateFromDir

public LocalDate getDateFromDir(String dirName)

Extract the date from the directory name, this should be a directory created by this class.

getDirForDate

public String getDirForDate(LocalDate date)

Returns the directory name for a given date.

getNextDirectory

public String getNextDirectory(String currentDir)
                        throws IOException

Find next available directory, after the given directory.

Throws:

IOException

scanForChangedFiles

public List<org.apache.hadoop.fs.FileStatus> scanForChangedFiles(String subDir,
                                                                 Map<String,Long> currentOffsets)
                                                          throws IOException

Returns new or changed files in the given directory. The offsets are used to find changed files.

Throws:

IOException

getNow

public LocalDateTime getNow()

Returns the current time, using the underlying clock in UTC time.

getConfig

public org.apache.hadoop.conf.Configuration getConfig()