org.apache.hudi.io

Class HoodieAppendHandle<T,I,K,O>

java.lang.Object

org.apache.hudi.io.HoodieIOHandle<T,I,K,O>

org.apache.hudi.io.HoodieWriteHandle<T,I,K,O>

org.apache.hudi.io.HoodieAppendHandle<T,I,K,O>

public class HoodieAppendHandle<T,I,K,O>
extends HoodieWriteHandle<T,I,K,O>

IO Operation to append data onto an existing file.

Field Summary

Fields

Modifier and Type	Field and Description
protected long	estimatedNumberOfBytesWritten
protected String	fileId
protected Map<HoodieLogBlock.HeaderMetadataType,String>	header
protected long	insertRecordsWritten
protected Iterator<HoodieRecord<T>>	recordItr
protected long	recordsDeleted
protected long	recordsWritten
protected List<WriteStatus>	statuses
protected long	updatedRecordsWritten
protected HoodieLogFormat.Writer	writer

Fields inherited from class org.apache.hudi.io.HoodieWriteHandle

newRecordLocation, partitionPath, recordMerger, schemaOnReadEnabled, taskContextSupplier, timer, writeSchema, writeSchemaWithMetaFields, writeStatus, writeToken

Fields inherited from class org.apache.hudi.io.HoodieIOHandle

config, fs, hoodieTable, instantTime

Constructor Summary

Constructors

Constructor and Description
HoodieAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T,I,K,O> hoodieTable, String partitionPath, String fileId, Iterator<HoodieRecord<T>> recordItr, TaskContextSupplier taskContextSupplier)
HoodieAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T,I,K,O> hoodieTable, String partitionPath, String fileId, Iterator<HoodieRecord<T>> recordItr, TaskContextSupplier taskContextSupplier, Map<HoodieLogBlock.HeaderMetadataType,String> header) This is used by log compaction only.
HoodieAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T,I,K,O> hoodieTable, String partitionPath, String fileId, TaskContextSupplier sparkTaskContextSupplier)

Method Summary

Modifier and Type	Method and Description
protected boolean	addBlockIdentifier()
protected void	appendDataAndDeleteBlocks(Map<HoodieLogBlock.HeaderMetadataType,String> header, boolean appendDeleteBlocks) Appends data and delete blocks.
boolean	canWrite(HoodieRecord record) Determines whether we can accept the incoming records, into the current file.
List<WriteStatus>	close()
void	doAppend()
protected void	doWrite(HoodieRecord record, org.apache.avro.Schema schema, TypedProperties props) Perform the actual writing of the given record into the backing file.
IOType	getIOType()
protected boolean	isUpdateRecord(HoodieRecord<T> hoodieRecord) Returns whether the hoodie record is an UPDATE.
protected boolean	needsUpdateLocation() Whether there is need to update the record location.
void	write(Map<String,HoodieRecord<T>> recordMap)
List<WriteStatus>	writeStatuses()

Methods inherited from class org.apache.hudi.io.HoodieWriteHandle

createLogWriter, createLogWriter, createLogWriter, createMarkerFile, getAttemptId, getConfig, getFileId, getFileSystem, getHoodieTableMetaClient, getPartitionId, getPartitionPath, getStageId, getWriterSchema, getWriterSchemaWithMetaFields, isClosed, makeNewFilePath, makeNewPath, markClosed, toAvroRecord, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fileId

protected final String fileId

recordItr

protected Iterator<HoodieRecord<T>> recordItr

writer

protected HoodieLogFormat.Writer writer

statuses

protected final List<WriteStatus> statuses

recordsWritten

protected long recordsWritten

recordsDeleted

protected long recordsDeleted

updatedRecordsWritten

protected long updatedRecordsWritten

insertRecordsWritten

protected long insertRecordsWritten

estimatedNumberOfBytesWritten

protected long estimatedNumberOfBytesWritten

header

protected final Map<HoodieLogBlock.HeaderMetadataType,String> header

Constructor Detail

HoodieAppendHandle

public HoodieAppendHandle(HoodieWriteConfig config,
                          String instantTime,
                          HoodieTable<T,I,K,O> hoodieTable,
                          String partitionPath,
                          String fileId,
                          Iterator<HoodieRecord<T>> recordItr,
                          TaskContextSupplier taskContextSupplier,
                          Map<HoodieLogBlock.HeaderMetadataType,String> header)

This is used by log compaction only.

HoodieAppendHandle

public HoodieAppendHandle(HoodieWriteConfig config,
                          String instantTime,
                          HoodieTable<T,I,K,O> hoodieTable,
                          String partitionPath,
                          String fileId,
                          Iterator<HoodieRecord<T>> recordItr,
                          TaskContextSupplier taskContextSupplier)

HoodieAppendHandle

public HoodieAppendHandle(HoodieWriteConfig config,
                          String instantTime,
                          HoodieTable<T,I,K,O> hoodieTable,
                          String partitionPath,
                          String fileId,
                          TaskContextSupplier sparkTaskContextSupplier)

Method Detail

isUpdateRecord

protected boolean isUpdateRecord(HoodieRecord<T> hoodieRecord)

Returns whether the hoodie record is an UPDATE.

doAppend

public void doAppend()

appendDataAndDeleteBlocks

protected void appendDataAndDeleteBlocks(Map<HoodieLogBlock.HeaderMetadataType,String> header,
                                         boolean appendDeleteBlocks)

Appends data and delete blocks. When appendDeleteBlocks value is false, only data blocks are appended. This is done so that all the data blocks are created first and then a single delete block is added. Otherwise what can end up happening is creation of multiple small delete blocks get added after each data block.

canWrite

public boolean canWrite(HoodieRecord record)

Description copied from class: HoodieWriteHandle

Determines whether we can accept the incoming records, into the current file. Depending on

- Whether it belongs to the same partitionPath as existing records - Whether the current file written bytes lt max file size

Overrides:

canWrite in class HoodieWriteHandle<T,I,K,O>

doWrite

protected void doWrite(HoodieRecord record,
                       org.apache.avro.Schema schema,
                       TypedProperties props)

Description copied from class: HoodieWriteHandle

Perform the actual writing of the given record into the backing file.

Overrides:

doWrite in class HoodieWriteHandle<T,I,K,O>

close

public List<WriteStatus> close()

Specified by:

close in class HoodieWriteHandle<T,I,K,O>

write

public void write(Map<String,HoodieRecord<T>> recordMap)

getIOType

public IOType getIOType()

Specified by:

getIOType in class HoodieWriteHandle<T,I,K,O>

writeStatuses

public List<WriteStatus> writeStatuses()

Overrides:

writeStatuses in class HoodieWriteHandle<T,I,K,O>

needsUpdateLocation

protected boolean needsUpdateLocation()

Whether there is need to update the record location.

addBlockIdentifier

protected boolean addBlockIdentifier()