org.apache.hudi.integ.testsuite.reader

Class DFSHoodieDatasetInputReader

java.lang.Object

org.apache.hudi.integ.testsuite.reader.DFSDeltaInputReader

org.apache.hudi.integ.testsuite.reader.DFSHoodieDatasetInputReader

All Implemented Interfaces:

Serializable, DeltaInputReader<org.apache.avro.generic.GenericRecord>

public class DFSHoodieDatasetInputReader
extends DFSDeltaInputReader

This class helps to generate updates from an already existing hoodie dataset. It supports generating updates in across partitions, files and records.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DFSHoodieDatasetInputReader(org.apache.spark.api.java.JavaSparkContext jsc, String basePath, String schemaStr)

Method Summary

Modifier and Type	Method and Description
protected long	analyzeSingleFile(String filePath) Implementation of DeltaInputReaders to provide a way to read a single file on DFS and provide an average number of records across N files.
protected List<String>	getPartitions(org.apache.hudi.common.util.Option<Integer> partitionsLimit)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	read(int numPartitions, int numFiles, double percentageRecordsPerFile)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	read(int numPartitions, int numFiles, long numRecords)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	read(int numPartitions, long approxNumRecords)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	read(long numRecords) Attempts to reads an approximate number of records close to approxNumRecords.

Methods inherited from class org.apache.hudi.integ.testsuite.reader.DFSDeltaInputReader

getFilePathsToRead, getFileStatusIndexRange

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DFSHoodieDatasetInputReader

public DFSHoodieDatasetInputReader(org.apache.spark.api.java.JavaSparkContext jsc,
                                   String basePath,
                                   String schemaStr)

Method Detail

getPartitions

protected List<String> getPartitions(org.apache.hudi.common.util.Option<Integer> partitionsLimit)
                              throws IOException

Throws:

IOException

analyzeSingleFile

protected long analyzeSingleFile(String filePath)

Description copied from class: DFSDeltaInputReader

Implementation of DeltaInputReaders to provide a way to read a single file on DFS and provide an average number of records across N files.

Overrides:

analyzeSingleFile in class DFSDeltaInputReader

read

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> read(long numRecords)
                                                                              throws IOException

Description copied from interface: DeltaInputReader

Attempts to reads an approximate number of records close to approxNumRecords. This highly depends on the number of records already present in the input.

Throws:

IOException

read

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> read(int numPartitions,
                                                                                     long approxNumRecords)
                                                                              throws IOException

Throws:

IOException - Attempts to read approx number of records (exact if equal or more records available) across requested number of partitions.

read

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> read(int numPartitions,
                                                                                     int numFiles,
                                                                                     long numRecords)
                                                                              throws IOException

Throws:

IOException - Attempts to read approx number of records (exact if equal or more records available) across requested number of partitions and number of files. 1. Find numFiles across numPartitions 2. numRecordsToReadPerFile = approxNumRecords / numFiles

read

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> read(int numPartitions,
                                                                                     int numFiles,
                                                                                     double percentageRecordsPerFile)
                                                                              throws IOException

Throws:

IOException - Attempts to a % of records per file across requested number of partitions and number of files. 1. Find numFiles across numPartitions 2. numRecordsToReadPerFile = approxNumRecordsPerFile * percentageRecordsPerFile