org.apache.hudi.utilities

Class HDFSParquetImporter

java.lang.Object

org.apache.hudi.utilities.HDFSParquetImporter

All Implemented Interfaces:

Serializable

public class HDFSParquetImporter
extends Object
implements Serializable

Loads data from Parquet Sources

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HDFSParquetImporter.Config
static class	HDFSParquetImporter.FormatValidator

Field Summary

Fields

Modifier and Type	Field and Description
static SimpleDateFormat	PARTITION_FORMATTER

Constructor Summary

Constructors

Constructor and Description
HDFSParquetImporter(HDFSParquetImporter.Config cfg)

Method Summary

Modifier and Type	Method and Description
protected org.apache.spark.api.java.JavaRDD<HoodieRecord<HoodieRecordPayload>>	buildHoodieRecordsForImport(org.apache.spark.api.java.JavaSparkContext jsc, String schemaStr)
protected int	dataImport(org.apache.spark.api.java.JavaSparkContext jsc)
int	dataImport(org.apache.spark.api.java.JavaSparkContext jsc, int retry)
protected <T extends HoodieRecordPayload> org.apache.spark.api.java.JavaRDD<WriteStatus>	load(HoodieWriteClient client, String instantTime, org.apache.spark.api.java.JavaRDD<HoodieRecord<T>> hoodieRecords) Imports records to Hoodie dataset
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARTITION_FORMATTER

public static final SimpleDateFormat PARTITION_FORMATTER

Constructor Detail

HDFSParquetImporter

public HDFSParquetImporter(HDFSParquetImporter.Config cfg)
                    throws IOException

Throws:

IOException

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

dataImport

public int dataImport(org.apache.spark.api.java.JavaSparkContext jsc,
                      int retry)
               throws Exception

Throws:

Exception

dataImport

protected int dataImport(org.apache.spark.api.java.JavaSparkContext jsc)
                  throws IOException

Throws:

IOException

buildHoodieRecordsForImport

protected org.apache.spark.api.java.JavaRDD<HoodieRecord<HoodieRecordPayload>> buildHoodieRecordsForImport(org.apache.spark.api.java.JavaSparkContext jsc,
                                                                                                           String schemaStr)
                                                                                                    throws IOException

Throws:

IOException

load

protected <T extends HoodieRecordPayload> org.apache.spark.api.java.JavaRDD<WriteStatus> load(HoodieWriteClient client,
                                                                                              String instantTime,
                                                                                              org.apache.spark.api.java.JavaRDD<HoodieRecord<T>> hoodieRecords)

Imports records to Hoodie dataset

Type Parameters:

T - Type

Parameters:

client - Hoodie Client

instantTime - Instant Time

hoodieRecords - Hoodie Records