org.apache.hudi.integ.testsuite.writer

Class SparkAvroDeltaInputWriter

java.lang.Object

org.apache.hudi.integ.testsuite.writer.SparkAvroDeltaInputWriter

All Implemented Interfaces:

Closeable, AutoCloseable, DeltaInputWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

public class SparkAvroDeltaInputWriter
extends Object
implements DeltaInputWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

Spark based avro delta input writer. We don't use this yet since we cannot control result file size.

Field Summary

Fields

Modifier and Type	Field and Description
org.apache.spark.sql.SparkSession	sparkSession

Constructor Summary

Constructors

Constructor and Description
SparkAvroDeltaInputWriter(org.apache.spark.sql.SparkSession sparkSession, String schemaStr, String basePath)

Method Summary

Modifier and Type	Method and Description
boolean	canWrite() Check whether more data can/should be written.
void	close()
DeltaWriteStats	getDeltaWriteStats() Return the statistics of data written.
DeltaInputWriter	getNewWriter() Return a new instance of this writer.
void	writeData(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> iData) Generate any type of data.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sparkSession

public org.apache.spark.sql.SparkSession sparkSession

Constructor Detail

SparkAvroDeltaInputWriter

public SparkAvroDeltaInputWriter(org.apache.spark.sql.SparkSession sparkSession,
                                 String schemaStr,
                                 String basePath)

Method Detail

writeData

public void writeData(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> iData)
               throws IOException

Description copied from interface: DeltaInputWriter

Generate any type of data.

Specified by:

writeData in interface DeltaInputWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

Throws:

IOException

canWrite

public boolean canWrite()

Description copied from interface: DeltaInputWriter

Check whether more data can/should be written.

Specified by:

canWrite in interface DeltaInputWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getDeltaWriteStats

public DeltaWriteStats getDeltaWriteStats()

Description copied from interface: DeltaInputWriter

Return the statistics of data written.

Specified by:

getDeltaWriteStats in interface DeltaInputWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

getNewWriter

public DeltaInputWriter getNewWriter()
                              throws IOException

Description copied from interface: DeltaInputWriter

Return a new instance of this writer.

Specified by:

getNewWriter in interface DeltaInputWriter<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

Returns:

Throws:

IOException