org.apache.hudi.cli

Class HDFSParquetImporterUtils

java.lang.Object

org.apache.hudi.cli.HDFSParquetImporterUtils

All Implemented Interfaces:

Serializable

public class HDFSParquetImporterUtils
extends Object
implements Serializable

Loads data from Parquet Sources.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
HDFSParquetImporterUtils(String command, String srcPath, String targetPath, String tableName, String tableType, String rowKey, String partitionKey, int parallelism, String schemaFile, int retry, String propsFilePath)

Method Summary

Modifier and Type	Method and Description
org.apache.spark.api.java.JavaRDD<HoodieRecord<HoodieRecordPayload>>	buildHoodieRecordsForImport(org.apache.spark.api.java.JavaSparkContext jsc, String schemaStr)
static TypedProperties	buildProperties(List<String> props)
static SparkRDDWriteClient<HoodieRecordPayload>	createHoodieClient(org.apache.spark.api.java.JavaSparkContext jsc, String basePath, String schemaStr, int parallelism, Option<String> compactionStrategyClass, TypedProperties properties) Build Hoodie write client.
int	dataImport(org.apache.spark.api.java.JavaSparkContext jsc)
int	dataImport(org.apache.spark.api.java.JavaSparkContext jsc, org.apache.hadoop.fs.FileSystem fs)
static int	handleErrors(org.apache.spark.api.java.JavaSparkContext jsc, String instantTime, org.apache.spark.api.java.JavaRDD<WriteStatus> writeResponse)
boolean	isUpsert()
<T extends HoodieRecordPayload> org.apache.spark.api.java.JavaRDD<WriteStatus>	load(SparkRDDWriteClient<T> client, String instantTime, org.apache.spark.api.java.JavaRDD<HoodieRecord<T>> hoodieRecords) Imports records to Hoodie table.
static String	parseSchema(org.apache.hadoop.fs.FileSystem fs, String schemaFile) Parse Schema from file.
static DFSPropertiesConfiguration	readConfig(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.hadoop.fs.Path cfgPath, List<String> overriddenProps)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HDFSParquetImporterUtils

public HDFSParquetImporterUtils(String command,
                                String srcPath,
                                String targetPath,
                                String tableName,
                                String tableType,
                                String rowKey,
                                String partitionKey,
                                int parallelism,
                                String schemaFile,
                                int retry,
                                String propsFilePath)

Method Detail

isUpsert

public boolean isUpsert()

dataImport

public int dataImport(org.apache.spark.api.java.JavaSparkContext jsc)

dataImport

public int dataImport(org.apache.spark.api.java.JavaSparkContext jsc,
                      org.apache.hadoop.fs.FileSystem fs)

buildHoodieRecordsForImport

public org.apache.spark.api.java.JavaRDD<HoodieRecord<HoodieRecordPayload>> buildHoodieRecordsForImport(org.apache.spark.api.java.JavaSparkContext jsc,
                                                                                                        String schemaStr)
                                                                                                 throws IOException

Throws:

IOException

load

public <T extends HoodieRecordPayload> org.apache.spark.api.java.JavaRDD<WriteStatus> load(SparkRDDWriteClient<T> client,
                                                                                           String instantTime,
                                                                                           org.apache.spark.api.java.JavaRDD<HoodieRecord<T>> hoodieRecords)

Imports records to Hoodie table.

Type Parameters:

T - Type

Parameters:

client - Hoodie Client

instantTime - Instant Time

hoodieRecords - Hoodie Records

buildProperties

public static TypedProperties buildProperties(List<String> props)

readConfig

public static DFSPropertiesConfiguration readConfig(org.apache.hadoop.conf.Configuration hadoopConfig,
                                                    org.apache.hadoop.fs.Path cfgPath,
                                                    List<String> overriddenProps)

createHoodieClient

public static SparkRDDWriteClient<HoodieRecordPayload> createHoodieClient(org.apache.spark.api.java.JavaSparkContext jsc,
                                                                          String basePath,
                                                                          String schemaStr,
                                                                          int parallelism,
                                                                          Option<String> compactionStrategyClass,
                                                                          TypedProperties properties)

Build Hoodie write client.

Parameters:

jsc - Java Spark Context

basePath - Base Path

schemaStr - Schema

parallelism - Parallelism

parseSchema

public static String parseSchema(org.apache.hadoop.fs.FileSystem fs,
                                 String schemaFile)
                          throws Exception

Parse Schema from file.

Parameters:

fs - File System

schemaFile - Schema File

Throws:

Exception

handleErrors

public static int handleErrors(org.apache.spark.api.java.JavaSparkContext jsc,
                               String instantTime,
                               org.apache.spark.api.java.JavaRDD<WriteStatus> writeResponse)