org.apache.hudi.utilities.transform

Interface Transformer

All Known Implementing Classes:

FlatteningTransformer, IdentityTransformer, SqlQueryBasedTransformer

public interface Transformer

Transform source to target dataset before writing

Method Summary

Modifier and Type	Method and Description
org.apache.spark.sql.Dataset	apply(org.apache.spark.api.java.JavaSparkContext jsc, org.apache.spark.sql.SparkSession sparkSession, org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> rowDataset, TypedProperties properties) Transform source RDD to target RDD

Method Detail

apply

org.apache.spark.sql.Dataset apply(org.apache.spark.api.java.JavaSparkContext jsc,
                                   org.apache.spark.sql.SparkSession sparkSession,
                                   org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> rowDataset,
                                   TypedProperties properties)

Transform source RDD to target RDD

Parameters:

jsc - JavaSparkContext

sparkSession - Spark Session

rowDataset - Source DataSet

properties - Config properties

Returns:

Transformed Dataset