org.apache.hudi.io

Class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>

java.lang.Object

org.apache.hudi.io.HoodieIOHandle<T,I,K,O>

org.apache.hudi.io.HoodieWriteHandle<T,I,K,O>

Direct Known Subclasses:

HoodieAppendHandle, HoodieCreateHandle, HoodieMergeHandle

public abstract class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>
extends HoodieIOHandle<T,I,K,O>

Base class for all write operations logically performed at the file group level.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	fileId
static org.apache.hudi.io.HoodieWriteHandle.IgnoreRecord	IGNORE_RECORD A special record returned by HoodieRecordPayload, which means HoodieWriteHandle should just skip this record.
protected String	partitionPath
protected boolean	schemaOnReadEnabled
protected org.apache.avro.Schema	tableSchema The specified schema of the table.
protected org.apache.avro.Schema	tableSchemaWithMetaFields
protected TaskContextSupplier	taskContextSupplier
protected HoodieTimer	timer
protected org.apache.avro.Schema	writeSchema The write schema.
protected org.apache.avro.Schema	writeSchemaWithMetaFields
protected WriteStatus	writeStatus
protected String	writeToken

Fields inherited from class org.apache.hudi.io.HoodieIOHandle

config, fs, hoodieTable, instantTime

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath, String fileId, HoodieTable<T,I,K,O> hoodieTable, Option<org.apache.avro.Schema> overriddenSchema, TaskContextSupplier taskContextSupplier)
	HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath, String fileId, HoodieTable<T,I,K,O> hoodieTable, TaskContextSupplier taskContextSupplier)

Method Summary

Modifier and Type	Method and Description
boolean	canWrite(HoodieRecord record) Determines whether we can accept the incoming records, into the current file.
abstract List<WriteStatus>	close()
protected void	createMarkerFile(String partitionPath, String dataFileName) Creates an empty marker file corresponding to storage writer path.
protected HoodieFileWriter	createNewFileWriter(String instantTime, org.apache.hadoop.fs.Path path, HoodieTable<T,I,K,O> hoodieTable, HoodieWriteConfig config, org.apache.avro.Schema schema, TaskContextSupplier taskContextSupplier)
protected long	getAttemptId()
protected org.apache.hadoop.fs.FileSystem	getFileSystem()
abstract IOType	getIOType()
protected int	getPartitionId()
String	getPartitionPath()
protected int	getStageId()
org.apache.avro.Schema	getWriterSchemaWithMetaFields()
protected org.apache.hadoop.fs.Path	makeNewFilePath(String partitionPath, String fileName) Make new file path with given file name.
org.apache.hadoop.fs.Path	makeNewPath(String partitionPath)
protected org.apache.avro.generic.GenericRecord	rewriteRecord(org.apache.avro.generic.GenericRecord record) Rewrite the GenericRecord with the Schema containing the Hoodie Metadata fields.
protected org.apache.avro.generic.GenericRecord	rewriteRecordWithMetadata(org.apache.avro.generic.GenericRecord record, String fileName)
void	write(HoodieRecord record, Option<org.apache.avro.generic.IndexedRecord> insertValue) Perform the actual writing of the given record into the backing file.
void	write(HoodieRecord record, Option<org.apache.avro.generic.IndexedRecord> avroRecord, Option<Exception> exception) Perform the actual writing of the given record into the backing file.
List<WriteStatus>	writeStatuses()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

IGNORE_RECORD

public static org.apache.hudi.io.HoodieWriteHandle.IgnoreRecord IGNORE_RECORD

A special record returned by HoodieRecordPayload, which means HoodieWriteHandle should just skip this record. This record is only used for HoodieRecordPayload currently, so it should not shuffle though network, we can compare the record locally by the equal method. The HoodieRecordPayload#combineAndGetUpdateValue and HoodieRecordPayload#getInsertValue have 3 kind of return: 1、Option.empty This means we should delete this record. 2、IGNORE_RECORD This means we should not process this record,just skip. 3、Other non-empty record This means we should process this record. We can see the usage of IGNORE_RECORD in org.apache.spark.sql.hudi.command.payload.ExpressionPayload

tableSchema

protected final org.apache.avro.Schema tableSchema

The specified schema of the table. ("specified" denotes that this is configured by the client, as opposed to being implicitly fetched out of the commit metadata)

tableSchemaWithMetaFields

protected final org.apache.avro.Schema tableSchemaWithMetaFields

writeSchema

protected final org.apache.avro.Schema writeSchema

The write schema. In most case the write schema is the same to the input schema. But if HoodieWriteConfig#WRITE_SCHEMA is specified, we use the WRITE_SCHEMA as the write schema. This is useful for the case of custom HoodieRecordPayload which do some conversion to the incoming record in it. e.g. the ExpressionPayload do the sql expression conversion to the input.

writeSchemaWithMetaFields

protected final org.apache.avro.Schema writeSchemaWithMetaFields

timer

protected HoodieTimer timer

writeStatus

protected WriteStatus writeStatus

partitionPath

protected final String partitionPath

fileId

protected final String fileId

writeToken

protected final String writeToken

taskContextSupplier

protected final TaskContextSupplier taskContextSupplier

schemaOnReadEnabled

protected final boolean schemaOnReadEnabled

Constructor Detail

HoodieWriteHandle

public HoodieWriteHandle(HoodieWriteConfig config,
                         String instantTime,
                         String partitionPath,
                         String fileId,
                         HoodieTable<T,I,K,O> hoodieTable,
                         TaskContextSupplier taskContextSupplier)

HoodieWriteHandle

protected HoodieWriteHandle(HoodieWriteConfig config,
                            String instantTime,
                            String partitionPath,
                            String fileId,
                            HoodieTable<T,I,K,O> hoodieTable,
                            Option<org.apache.avro.Schema> overriddenSchema,
                            TaskContextSupplier taskContextSupplier)

Method Detail

makeNewPath

public org.apache.hadoop.fs.Path makeNewPath(String partitionPath)

makeNewFilePath

protected org.apache.hadoop.fs.Path makeNewFilePath(String partitionPath,
                                                    String fileName)

Make new file path with given file name.

createMarkerFile

protected void createMarkerFile(String partitionPath,
                                String dataFileName)

Creates an empty marker file corresponding to storage writer path.

Parameters:

partitionPath - Partition path

getWriterSchemaWithMetaFields

public org.apache.avro.Schema getWriterSchemaWithMetaFields()

canWrite

public boolean canWrite(HoodieRecord record)

Determines whether we can accept the incoming records, into the current file. Depending on

- Whether it belongs to the same partitionPath as existing records - Whether the current file written bytes lt max file size

write

public void write(HoodieRecord record,
                  Option<org.apache.avro.generic.IndexedRecord> insertValue)

Perform the actual writing of the given record into the backing file.

write

public void write(HoodieRecord record,
                  Option<org.apache.avro.generic.IndexedRecord> avroRecord,
                  Option<Exception> exception)

Perform the actual writing of the given record into the backing file.

rewriteRecord

protected org.apache.avro.generic.GenericRecord rewriteRecord(org.apache.avro.generic.GenericRecord record)

Rewrite the GenericRecord with the Schema containing the Hoodie Metadata fields.

rewriteRecordWithMetadata

protected org.apache.avro.generic.GenericRecord rewriteRecordWithMetadata(org.apache.avro.generic.GenericRecord record,
                                                                          String fileName)

close

public abstract List<WriteStatus> close()

writeStatuses

public List<WriteStatus> writeStatuses()

getPartitionPath

public String getPartitionPath()

getIOType

public abstract IOType getIOType()

getFileSystem

protected org.apache.hadoop.fs.FileSystem getFileSystem()

Specified by:

getFileSystem in class HoodieIOHandle<T extends HoodieRecordPayload,I,K,O>

getPartitionId

protected int getPartitionId()

getStageId

protected int getStageId()

getAttemptId

protected long getAttemptId()

createNewFileWriter

protected HoodieFileWriter createNewFileWriter(String instantTime,
                                               org.apache.hadoop.fs.Path path,
                                               HoodieTable<T,I,K,O> hoodieTable,
                                               HoodieWriteConfig config,
                                               org.apache.avro.Schema schema,
                                               TaskContextSupplier taskContextSupplier)
                                        throws IOException

Throws:

IOException