org.apache.hudi.integ.testsuite.reader

Class SparkBasedReader

java.lang.Object

org.apache.hudi.integ.testsuite.reader.SparkBasedReader

public class SparkBasedReader
extends Object

Helper class to read avro and/or parquet files and generate a RDD of GenericRecord.

Field Summary

Fields

Modifier and Type	Field and Description
static String	SPARK_AVRO_FORMAT
static String	SPARK_PARQUET_FORMAT

Constructor Summary

Constructors

Constructor and Description
SparkBasedReader()

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	readAvro(org.apache.spark.sql.SparkSession sparkSession, String schemaStr, List<String> listOfPaths, org.apache.hudi.common.util.Option<String> structName, org.apache.hudi.common.util.Option<String> nameSpace)
static org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	readOrc(org.apache.spark.sql.SparkSession sparkSession, List<String> listOfPaths, org.apache.hudi.common.util.Option<String> structName, org.apache.hudi.common.util.Option<String> nameSpace)
static org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	readParquet(org.apache.spark.sql.SparkSession sparkSession, List<String> listOfPaths, org.apache.hudi.common.util.Option<String> structName, org.apache.hudi.common.util.Option<String> nameSpace)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPARK_AVRO_FORMAT

public static final String SPARK_AVRO_FORMAT

See Also:

Constant Field Values

SPARK_PARQUET_FORMAT

public static final String SPARK_PARQUET_FORMAT

See Also:

Constant Field Values

Constructor Detail

SparkBasedReader

public SparkBasedReader()

Method Detail

readAvro

public static org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> readAvro(org.apache.spark.sql.SparkSession sparkSession,
                                                                                                String schemaStr,
                                                                                                List<String> listOfPaths,
                                                                                                org.apache.hudi.common.util.Option<String> structName,
                                                                                                org.apache.hudi.common.util.Option<String> nameSpace)

readParquet

public static org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> readParquet(org.apache.spark.sql.SparkSession sparkSession,
                                                                                                   List<String> listOfPaths,
                                                                                                   org.apache.hudi.common.util.Option<String> structName,
                                                                                                   org.apache.hudi.common.util.Option<String> nameSpace)

readOrc

public static org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> readOrc(org.apache.spark.sql.SparkSession sparkSession,
                                                                                               List<String> listOfPaths,
                                                                                               org.apache.hudi.common.util.Option<String> structName,
                                                                                               org.apache.hudi.common.util.Option<String> nameSpace)