org.apache.hudi.integ.testsuite.reader

Class DFSAvroDeltaInputReader

java.lang.Object

org.apache.hudi.integ.testsuite.reader.DFSDeltaInputReader

org.apache.hudi.integ.testsuite.reader.DFSAvroDeltaInputReader

All Implemented Interfaces:

Serializable, DeltaInputReader<org.apache.avro.generic.GenericRecord>

public class DFSAvroDeltaInputReader
extends DFSDeltaInputReader

A reader of DeltaOutputMode.DFS and DeltaInputType.AVRO.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.fs.PathFilter	filter

Constructor Summary

Constructors

Constructor and Description
DFSAvroDeltaInputReader(org.apache.spark.sql.SparkSession sparkSession, String schemaStr, String basePath, org.apache.hudi.common.util.Option<String> structName, org.apache.hudi.common.util.Option<String> nameSpace)

Method Summary

Modifier and Type	Method and Description
protected long	analyzeSingleFile(String filePath) Implementation of DeltaInputReaders to provide a way to read a single file on DFS and provide an average number of records across N files.
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	read(int numPartitions, int numFiles, double percentageRecordsPerFile)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	read(int numPartitions, int numFiles, long approxNumRecords)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	read(int numPartitions, long approxNumRecords)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	read(long totalRecordsToRead) Attempts to reads an approximate number of records close to approxNumRecords.

Methods inherited from class org.apache.hudi.integ.testsuite.reader.DFSDeltaInputReader

getFilePathsToRead, getFileStatusIndexRange

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

filter

protected org.apache.hadoop.fs.PathFilter filter

Constructor Detail

DFSAvroDeltaInputReader

public DFSAvroDeltaInputReader(org.apache.spark.sql.SparkSession sparkSession,
                               String schemaStr,
                               String basePath,
                               org.apache.hudi.common.util.Option<String> structName,
                               org.apache.hudi.common.util.Option<String> nameSpace)

Method Detail

read

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> read(long totalRecordsToRead)
                                                                              throws IOException

Description copied from interface: DeltaInputReader

Attempts to reads an approximate number of records close to approxNumRecords. This highly depends on the number of records already present in the input.

Throws:

IOException

read

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> read(int numPartitions,
                                                                                     long approxNumRecords)
                                                                              throws IOException

Throws:

IOException - Attempts to read approx number of records (exact if equal or more records available) across requested number of partitions.

read

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> read(int numPartitions,
                                                                                     int numFiles,
                                                                                     long approxNumRecords)
                                                                              throws IOException

Throws:

IOException - Attempts to read approx number of records (exact if equal or more records available) across requested number of partitions and number of files. 1. Find numFiles across numPartitions 2. numRecordsToReadPerFile = approxNumRecords / numFiles

read

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> read(int numPartitions,
                                                                                     int numFiles,
                                                                                     double percentageRecordsPerFile)
                                                                              throws IOException

Throws:

IOException - Attempts to a % of records per file across requested number of partitions and number of files. 1. Find numFiles across numPartitions 2. numRecordsToReadPerFile = approxNumRecordsPerFile * percentageRecordsPerFile

analyzeSingleFile

protected long analyzeSingleFile(String filePath)

Description copied from class: DFSDeltaInputReader

Implementation of DeltaInputReaders to provide a way to read a single file on DFS and provide an average number of records across N files.

Overrides:

analyzeSingleFile in class DFSDeltaInputReader