org.apache.hudi.io

Class HoodieAppendHandle<T extends HoodieRecordPayload,I,K,O>

java.lang.Object

org.apache.hudi.io.HoodieIOHandle<T,I,K,O>

org.apache.hudi.io.HoodieWriteHandle<T,I,K,O>

org.apache.hudi.io.HoodieAppendHandle<T,I,K,O>

public class HoodieAppendHandle<T extends HoodieRecordPayload,I,K,O>
extends HoodieWriteHandle<T,I,K,O>

IO Operation to append data onto an existing file.

Field Summary

Fields

Modifier and Type	Field and Description
protected long	estimatedNumberOfBytesWritten
protected String	fileId
protected Map<HoodieLogBlock.HeaderMetadataType,String>	header
protected long	insertRecordsWritten
protected Iterator<HoodieRecord<T>>	recordItr
protected long	recordsDeleted
protected long	recordsWritten
protected List<WriteStatus>	statuses
protected long	updatedRecordsWritten
protected HoodieLogFormat.Writer	writer

Fields inherited from class org.apache.hudi.io.HoodieWriteHandle

IGNORE_RECORD, partitionPath, schemaOnReadEnabled, tableSchema, tableSchemaWithMetaFields, taskContextSupplier, timer, writeSchema, writeSchemaWithMetaFields, writeStatus, writeToken

Fields inherited from class org.apache.hudi.io.HoodieIOHandle

config, fs, hoodieTable, instantTime

Constructor Summary

Constructors

Constructor and Description
HoodieAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T,I,K,O> hoodieTable, String partitionPath, String fileId, Iterator<HoodieRecord<T>> recordItr, TaskContextSupplier taskContextSupplier)
HoodieAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T,I,K,O> hoodieTable, String partitionPath, String fileId, TaskContextSupplier sparkTaskContextSupplier)

Method Summary

Modifier and Type	Method and Description
protected void	appendDataAndDeleteBlocks(Map<HoodieLogBlock.HeaderMetadataType,String> header)
boolean	canWrite(HoodieRecord record) Determines whether we can accept the incoming records, into the current file.
List<WriteStatus>	close()
void	doAppend()
IOType	getIOType()
protected boolean	isUpdateRecord(HoodieRecord<T> hoodieRecord) Returns whether the hoodie record is an UPDATE.
protected boolean	needsUpdateLocation() Whether there is need to update the record location.
void	write(HoodieRecord record, Option<org.apache.avro.generic.IndexedRecord> insertValue) Perform the actual writing of the given record into the backing file.
List<WriteStatus>	writeStatuses()

Methods inherited from class org.apache.hudi.io.HoodieWriteHandle

createMarkerFile, createNewFileWriter, getAttemptId, getFileSystem, getPartitionId, getPartitionPath, getStageId, getWriterSchemaWithMetaFields, makeNewFilePath, makeNewPath, rewriteRecord, rewriteRecordWithMetadata, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fileId

protected final String fileId

recordItr

protected Iterator<HoodieRecord<T extends HoodieRecordPayload>> recordItr

writer

protected HoodieLogFormat.Writer writer

statuses

protected final List<WriteStatus> statuses

recordsWritten

protected long recordsWritten

recordsDeleted

protected long recordsDeleted

updatedRecordsWritten

protected long updatedRecordsWritten

insertRecordsWritten

protected long insertRecordsWritten

estimatedNumberOfBytesWritten

protected long estimatedNumberOfBytesWritten

header

protected final Map<HoodieLogBlock.HeaderMetadataType,String> header

Constructor Detail

HoodieAppendHandle

public HoodieAppendHandle(HoodieWriteConfig config,
                          String instantTime,
                          HoodieTable<T,I,K,O> hoodieTable,
                          String partitionPath,
                          String fileId,
                          Iterator<HoodieRecord<T>> recordItr,
                          TaskContextSupplier taskContextSupplier)

HoodieAppendHandle

public HoodieAppendHandle(HoodieWriteConfig config,
                          String instantTime,
                          HoodieTable<T,I,K,O> hoodieTable,
                          String partitionPath,
                          String fileId,
                          TaskContextSupplier sparkTaskContextSupplier)

Method Detail

isUpdateRecord

protected boolean isUpdateRecord(HoodieRecord<T> hoodieRecord)

Returns whether the hoodie record is an UPDATE.

doAppend

public void doAppend()

appendDataAndDeleteBlocks

protected void appendDataAndDeleteBlocks(Map<HoodieLogBlock.HeaderMetadataType,String> header)

canWrite

public boolean canWrite(HoodieRecord record)

Description copied from class: HoodieWriteHandle

Determines whether we can accept the incoming records, into the current file. Depending on

- Whether it belongs to the same partitionPath as existing records - Whether the current file written bytes lt max file size

Overrides:

canWrite in class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>

write

public void write(HoodieRecord record,
                  Option<org.apache.avro.generic.IndexedRecord> insertValue)

Description copied from class: HoodieWriteHandle

Perform the actual writing of the given record into the backing file.

Overrides:

write in class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>

close

public List<WriteStatus> close()

Specified by:

close in class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>

getIOType

public IOType getIOType()

Specified by:

getIOType in class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>

writeStatuses

public List<WriteStatus> writeStatuses()

Overrides:

writeStatuses in class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>

needsUpdateLocation

protected boolean needsUpdateLocation()

Whether there is need to update the record location.