org.apache.hudi.integ.testsuite.reader

Class DFSDeltaInputReader

java.lang.Object

org.apache.hudi.integ.testsuite.reader.DFSDeltaInputReader

All Implemented Interfaces:

Serializable, DeltaInputReader<org.apache.avro.generic.GenericRecord>

Direct Known Subclasses:

DFSAvroDeltaInputReader, DFSHoodieDatasetInputReader

public abstract class DFSDeltaInputReader
extends Object
implements DeltaInputReader<org.apache.avro.generic.GenericRecord>

This class helps to estimate the number of files to read a given number of total records. Use this class for all DFS based implementations of DeltaInputReader

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DFSDeltaInputReader()

Method Summary

Modifier and Type	Method and Description
protected long	analyzeSingleFile(String filePath) Implementation of DeltaInputReaders to provide a way to read a single file on DFS and provide an average number of records across N files.
protected List<String>	getFilePathsToRead(String basePath, org.apache.hadoop.fs.PathFilter filter, long totalRecordsToRead)
protected org.apache.hudi.common.util.collection.Pair<Integer,Integer>	getFileStatusIndexRange(List<org.apache.hadoop.fs.FileStatus> fileStatuses, long averageFileSize, long totalSizeToRead)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.integ.testsuite.reader.DeltaInputReader

read, read, read, read

Constructor Detail

DFSDeltaInputReader

public DFSDeltaInputReader()

Method Detail

getFilePathsToRead

protected List<String> getFilePathsToRead(String basePath,
                                          org.apache.hadoop.fs.PathFilter filter,
                                          long totalRecordsToRead)
                                   throws IOException

Throws:

IOException

getFileStatusIndexRange

protected org.apache.hudi.common.util.collection.Pair<Integer,Integer> getFileStatusIndexRange(List<org.apache.hadoop.fs.FileStatus> fileStatuses,
                                                                                               long averageFileSize,
                                                                                               long totalSizeToRead)

analyzeSingleFile

protected long analyzeSingleFile(String filePath)

Implementation of DeltaInputReaders to provide a way to read a single file on DFS and provide an average number of records across N files.