org.apache.hudi.integ.testsuite.writer

Class DFSSparkAvroDeltaWriter

java.lang.Object

org.apache.hudi.integ.testsuite.writer.DFSSparkAvroDeltaWriter

All Implemented Interfaces:

DeltaWriterAdapter.SparkBasedDeltaWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

public class DFSSparkAvroDeltaWriter
extends Object
implements DeltaWriterAdapter.SparkBasedDeltaWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

NEED TO IMPLEMENT A CUSTOM SPARK PARTITIONER TO ENSURE WE WRITE LARGE ENOUGH AVRO FILES.

Constructor Summary

Constructors

Constructor and Description
DFSSparkAvroDeltaWriter(DeltaInputWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>> deltaInputWriter)

Method Summary

Modifier and Type	Method and Description
org.apache.spark.api.java.JavaRDD<DeltaWriteStats>	write(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> input)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DFSSparkAvroDeltaWriter

public DFSSparkAvroDeltaWriter(DeltaInputWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>> deltaInputWriter)

Method Detail

write

public org.apache.spark.api.java.JavaRDD<DeltaWriteStats> write(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> input)
                                                         throws IOException

Specified by:

write in interface DeltaWriterAdapter.SparkBasedDeltaWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

Throws:

IOException