org.apache.hudi.integ.testsuite.reader

Interface DeltaInputReader<O>

Type Parameters:

O - Read result data type

All Superinterfaces:

Serializable

All Known Implementing Classes:

DFSAvroDeltaInputReader, DFSDeltaInputReader, DFSHoodieDatasetInputReader

public interface DeltaInputReader<O>
extends Serializable

Implementations of DeltaInputReader will read the configured input type and provide an RDD of records to the client.

Method Summary

Modifier and Type	Method and Description
org.apache.spark.api.java.JavaRDD<O>	read(int numPartitions, int numFiles, double percentageRecordsPerFile)
org.apache.spark.api.java.JavaRDD<O>	read(int numPartitions, int numFiles, long approxNumRecords)
org.apache.spark.api.java.JavaRDD<O>	read(int numPartitions, long approxNumRecords)
org.apache.spark.api.java.JavaRDD<O>	read(long approxNumRecords) Attempts to reads an approximate number of records close to approxNumRecords.

Method Detail

read

org.apache.spark.api.java.JavaRDD<O> read(long approxNumRecords)
                                   throws IOException

Attempts to reads an approximate number of records close to approxNumRecords. This highly depends on the number of records already present in the input.

Throws:

IOException

read

org.apache.spark.api.java.JavaRDD<O> read(int numPartitions,
                                          long approxNumRecords)
                                   throws IOException

Throws:

IOException - Attempts to read approx number of records (exact if equal or more records available) across requested number of partitions.

read

org.apache.spark.api.java.JavaRDD<O> read(int numPartitions,
                                          int numFiles,
                                          long approxNumRecords)
                                   throws IOException

Throws:

IOException - Attempts to read approx number of records (exact if equal or more records available) across requested number of partitions and number of files. 1. Find numFiles across numPartitions 2. numRecordsToReadPerFile = approxNumRecords / numFiles

read

org.apache.spark.api.java.JavaRDD<O> read(int numPartitions,
                                          int numFiles,
                                          double percentageRecordsPerFile)
                                   throws IOException

Throws:

IOException - Attempts to a % of records per file across requested number of partitions and number of files. 1. Find numFiles across numPartitions 2. numRecordsToReadPerFile = approxNumRecordsPerFile * percentageRecordsPerFile