org.apache.hudi.table

Class HoodieTable<T,I,K,O>

java.lang.Object

org.apache.hudi.table.HoodieTable<T,I,K,O>

Type Parameters:

T - Sub type of HoodieRecordPayload

I - Type of inputs

K - Type of keys

O - Type of outputs

All Implemented Interfaces:

Serializable

public abstract class HoodieTable<T,I,K,O>
extends Object
implements Serializable

Abstract implementation of a HoodieTable.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieWriteConfig	config
protected HoodieEngineContext	context
protected HoodieIndex<?,?>	index
protected HoodieTableMetaClient	metaClient
protected TaskContextSupplier	taskContextSupplier

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HoodieTable(HoodieWriteConfig config, HoodieEngineContext context, HoodieTableMetaClient metaClient)

Method Summary

Modifier and Type	Method and Description
abstract HoodieBootstrapWriteMetadata<O>	bootstrap(HoodieEngineContext context, Option<Map<String,String>> extraMetadata) Perform metadata/full bootstrap of a Hudi table.
abstract HoodieWriteMetadata<O>	bulkInsert(HoodieEngineContext context, String instantTime, I records, Option<BulkInsertPartitioner> bulkInsertPartitioner) Bulk Insert a batch of new records into Hoodie table at the supplied instantTime.
abstract HoodieWriteMetadata<O>	bulkInsertPrepped(HoodieEngineContext context, String instantTime, I preppedRecords, Option<BulkInsertPartitioner> bulkInsertPartitioner) Bulk Insert the given prepared records into the Hoodie table, at the supplied instantTime.
abstract HoodieCleanMetadata	clean(HoodieEngineContext context, String cleanInstantTime) Executes a new clean action.
HoodieCleanMetadata	clean(HoodieEngineContext context, String cleanInstantTime, boolean skipLocking) Deprecated.
abstract HoodieWriteMetadata<O>	cluster(HoodieEngineContext context, String clusteringInstantTime) Execute Clustering on the table.
abstract HoodieWriteMetadata<O>	compact(HoodieEngineContext context, String compactionInstantTime) Run Compaction on the table.
abstract HoodieWriteMetadata<O>	delete(HoodieEngineContext context, String instantTime, K keys) Deletes a list of HoodieKeys from the Hoodie table, at the supplied instantTime HoodieKeys will be de-duped and non existent keys will be removed before deleting.
void	deleteMetadataIndexIfNecessary() Deletes the metadata partition if the writer disables any metadata index.
abstract HoodieWriteMetadata<O>	deletePartitions(HoodieEngineContext context, String instantTime, List<String> partitions) Deletes all data of partitions.
abstract HoodieWriteMetadata<O>	deletePrepped(HoodieEngineContext context, String instantTime, I preppedRecords) Delete records from Hoodie table based on HoodieKey and HoodieRecordLocation specified in preppedRecords.
void	finalizeWrite(HoodieEngineContext context, String instantTs, List<HoodieWriteStat> stats) Finalize the written data onto storage.
HoodieActiveTimeline	getActiveTimeline()
String	getBaseFileExtension()
HoodieFileFormat	getBaseFileFormat()
TableFileSystemView.BaseFileOnlyView	getBaseFileOnlyView() Get the base file only view of the file system for this table.
HoodieTimeline	getCleanTimeline() Get clean timeline.
HoodieTimeline	getCompletedCleanTimeline() Get only the completed (no-inflights) clean timeline.
HoodieTimeline	getCompletedCommitsTimeline() Get only the completed (no-inflights) commit + deltacommit timeline.
HoodieTimeline	getCompletedCommitTimeline() Get only the completed (no-inflights) commit timeline.
HoodieTimeline	getCompletedSavepointTimeline() Get only the completed (no-inflights) savepoint timeline.
HoodieWriteConfig	getConfig()
static ConsistencyGuard	getConsistencyGuard(org.apache.hadoop.fs.FileSystem fs, ConsistencyGuardConfig consistencyGuardConfig) Instantiate ConsistencyGuard based on configs.
HoodieEngineContext	getContext()
TableFileSystemView	getFileSystemView() Get the view of the file system for this table.
org.apache.hadoop.conf.Configuration	getHadoopConf()
SyncableFileSystemView	getHoodieView() Get complete view of the file system for this table with ability to force sync.
HoodieIndex<?,?>	getIndex() Return the index.
protected abstract HoodieIndex<?,?>	getIndex(HoodieWriteConfig config, HoodieEngineContext context)
Option<HoodieTableMetadataWriter>	getIndexingMetadataWriter(String triggeringInstantTimestamp) Gets the metadata writer for async indexer.
protected Set<String>	getInvalidDataPaths(WriteMarkers markers) Returns the possible invalid data file name with given marker files.
HoodieFileFormat	getLogFileFormat()
HoodieTableMetaClient	getMetaClient()
HoodieTableMetadata	getMetadata()
HoodieTableMetadata	getMetadataTable()
Option<HoodieTableMetadataWriter>	getMetadataWriter(String triggeringInstantTimestamp) Get Table metadata writer.
protected Option<HoodieTableMetadataWriter>	getMetadataWriter(String triggeringInstantTimestamp, HoodieFailedWritesCleaningPolicy failedWritesCleaningPolicy) Get Table metadata writer.
Option<HoodieFileFormat>	getPartitionMetafileFormat()
HoodieTimeline	getPendingCommitTimeline() Get only the inflights (no-completed) commit timeline.
Runnable	getPreExecuteRunnable()
HoodieTimeline	getRestoreTimeline() Get restore timeline.
HoodieTimeline	getRollbackTimeline() Get rollback timeline.
Set<String>	getSavepointTimestamps() Get the list of savepoint timestamps in this table.
TableFileSystemView.SliceView	getSliceView() Get the full view of the file system for this table.
HoodieStorageLayout	getStorageLayout()
protected HoodieStorageLayout	getStorageLayout(HoodieWriteConfig config)
TaskContextSupplier	getTaskContextSupplier()
abstract Option<HoodieIndexCommitMetadata>	index(HoodieEngineContext context, String indexInstantTime) Execute requested index action.
abstract HoodieWriteMetadata<O>	insert(HoodieEngineContext context, String instantTime, I records) Insert a batch of new records into Hoodie table at the supplied instantTime.
abstract HoodieWriteMetadata<O>	insertOverwrite(HoodieEngineContext context, String instantTime, I records) Replaces all the existing records and inserts the specified new records into Hoodie table at the supplied instantTime, for the partition paths contained in input records.
abstract HoodieWriteMetadata<O>	insertOverwriteTable(HoodieEngineContext context, String instantTime, I records) Delete all the existing records of the Hoodie table and inserts the specified new records into Hoodie table at the supplied instantTime, for the partition paths contained in input records.
abstract HoodieWriteMetadata<O>	insertPrepped(HoodieEngineContext context, String instantTime, I preppedRecords) Inserts the given prepared records into the Hoodie table, at the supplied instantTime.
boolean	isMetadataTable()
boolean	isPartitioned()
HoodieWriteMetadata<O>	logCompact(HoodieEngineContext context, String logCompactionInstantTime) Run Log Compaction on the table.
void	maybeDeleteMetadataTable() Deletes the metadata table if the writer disables metadata table with hoodie.metadata.enable=false
protected void	reconcileAgainstMarkers(HoodieEngineContext context, String instantTs, List<HoodieWriteStat> stats, boolean consistencyCheckEnabled) Reconciles WriteStats and marker files to detect and safely delete duplicate data files created because of Spark retries.
boolean	requireSortedRecords()
abstract HoodieRestoreMetadata	restore(HoodieEngineContext context, String restoreInstantTimestamp, String savepointToRestoreTimestamp) Restore the table to the given instant.
abstract HoodieRollbackMetadata	rollback(HoodieEngineContext context, String rollbackInstantTime, HoodieInstant commitInstant, boolean deleteInstants, boolean skipLocking) Rollback the (inflight/committed) record changes with the given commit time.
abstract void	rollbackBootstrap(HoodieEngineContext context, String instantTime) Perform rollback of bootstrap of a Hudi table.
void	rollbackInflightClustering(HoodieInstant inflightInstant, Function<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInstantFunc) Rollback inflight clustering instant to requested clustering instant
void	rollbackInflightCompaction(HoodieInstant inflightInstant)
void	rollbackInflightCompaction(HoodieInstant inflightInstant, Function<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInstantFunc) Rollback failed compactions.
void	rollbackInflightLogCompaction(HoodieInstant inflightInstant)
void	rollbackInflightLogCompaction(HoodieInstant inflightInstant, Function<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInstantFunc) Rollback failed compactions.
abstract HoodieSavepointMetadata	savepoint(HoodieEngineContext context, String instantToSavepoint, String user, String comment) Create a savepoint at the specified instant, so that the table can be restored to this point-in-timeline later if needed.
abstract Option<HoodieCleanerPlan>	scheduleCleaning(HoodieEngineContext context, String instantTime, Option<Map<String,String>> extraMetadata) Schedule cleaning for the instant time.
abstract Option<HoodieClusteringPlan>	scheduleClustering(HoodieEngineContext context, String instantTime, Option<Map<String,String>> extraMetadata) Schedule clustering for the instant time.
abstract Option<HoodieCompactionPlan>	scheduleCompaction(HoodieEngineContext context, String instantTime, Option<Map<String,String>> extraMetadata) Schedule compaction for the instant time.
abstract Option<HoodieIndexPlan>	scheduleIndexing(HoodieEngineContext context, String indexInstantTime, List<MetadataPartitionType> partitionsToIndex) Schedules Indexing for the table to the given instant.
Option<HoodieCompactionPlan>	scheduleLogCompaction(HoodieEngineContext context, String instantTime, Option<Map<String,String>> extraMetadata) Schedule log compaction for the instant time.
abstract Option<HoodieRestorePlan>	scheduleRestore(HoodieEngineContext context, String restoreInstantTimestamp, String savepointToRestoreTimestamp) Schedules Restore for the table to the given instant.
abstract Option<HoodieRollbackPlan>	scheduleRollback(HoodieEngineContext context, String instantTime, HoodieInstant instantToRollback, boolean skipTimelinePublish, boolean shouldRollbackUsingMarkers, boolean isRestore) Schedule rollback for the instant time.
boolean	shouldTrackSuccessRecords() When HoodieTableConfig.POPULATE_META_FIELDS is enabled, we need to track written records within WriteStatus in two cases: When the HoodieIndex being used is not implicit with storage If any of the metadata table partitions (record index, etc) which require written record tracking are enabled
abstract HoodieWriteMetadata<O>	upsert(HoodieEngineContext context, String instantTime, I records) Upsert a batch of new records into Hoodie table at the supplied instantTime.
abstract HoodieWriteMetadata<O>	upsertPrepped(HoodieEngineContext context, String instantTime, I preppedRecords) Upserts the given prepared records into the Hoodie table, at the supplied instantTime.
void	validateInsertSchema()
void	validateUpsertSchema()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected final HoodieWriteConfig config

metaClient

protected final HoodieTableMetaClient metaClient

index

protected final HoodieIndex<?,?> index

taskContextSupplier

protected final TaskContextSupplier taskContextSupplier

context

protected final transient HoodieEngineContext context

Constructor Detail

HoodieTable

protected HoodieTable(HoodieWriteConfig config,
                      HoodieEngineContext context,
                      HoodieTableMetaClient metaClient)

Method Detail

isMetadataTable

public boolean isMetadataTable()

getIndex

protected abstract HoodieIndex<?,?> getIndex(HoodieWriteConfig config,
                                             HoodieEngineContext context)

getStorageLayout

protected HoodieStorageLayout getStorageLayout(HoodieWriteConfig config)

getMetadata

public HoodieTableMetadata getMetadata()

upsert

public abstract HoodieWriteMetadata<O> upsert(HoodieEngineContext context,
                                              String instantTime,
                                              I records)

Upsert a batch of new records into Hoodie table at the supplied instantTime.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

records - hoodieRecords to upsert

Returns:

HoodieWriteMetadata

insert

public abstract HoodieWriteMetadata<O> insert(HoodieEngineContext context,
                                              String instantTime,
                                              I records)

Insert a batch of new records into Hoodie table at the supplied instantTime.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

records - hoodieRecords to upsert

Returns:

HoodieWriteMetadata

bulkInsert

public abstract HoodieWriteMetadata<O> bulkInsert(HoodieEngineContext context,
                                                  String instantTime,
                                                  I records,
                                                  Option<BulkInsertPartitioner> bulkInsertPartitioner)

Bulk Insert a batch of new records into Hoodie table at the supplied instantTime.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

records - hoodieRecords to upsert

bulkInsertPartitioner - User Defined Partitioner

Returns:

HoodieWriteMetadata

delete

public abstract HoodieWriteMetadata<O> delete(HoodieEngineContext context,
                                              String instantTime,
                                              K keys)

Deletes a list of HoodieKeys from the Hoodie table, at the supplied instantTime HoodieKeys will be de-duped and non existent keys will be removed before deleting.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

keys - List of HoodieKeys to be deleted

Returns:

HoodieWriteMetadata

deletePrepped

public abstract HoodieWriteMetadata<O> deletePrepped(HoodieEngineContext context,
                                                     String instantTime,
                                                     I preppedRecords)

Delete records from Hoodie table based on HoodieKey and HoodieRecordLocation specified in preppedRecords.

Parameters:

context - HoodieEngineContext.

instantTime - Instant Time for the action.

preppedRecords - Empty records with key and locator set.

Returns:

HoodieWriteMetadata

deletePartitions

public abstract HoodieWriteMetadata<O> deletePartitions(HoodieEngineContext context,
                                                        String instantTime,
                                                        List<String> partitions)

Deletes all data of partitions.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

partitions - List of partition to be deleted

Returns:

HoodieWriteMetadata

upsertPrepped

public abstract HoodieWriteMetadata<O> upsertPrepped(HoodieEngineContext context,
                                                     String instantTime,
                                                     I preppedRecords)

Upserts the given prepared records into the Hoodie table, at the supplied instantTime.

This implementation requires that the input records are already tagged, and de-duped if needed.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

preppedRecords - hoodieRecords to upsert

Returns:

HoodieWriteMetadata

insertPrepped

public abstract HoodieWriteMetadata<O> insertPrepped(HoodieEngineContext context,
                                                     String instantTime,
                                                     I preppedRecords)

Inserts the given prepared records into the Hoodie table, at the supplied instantTime.

This implementation requires that the input records are already tagged, and de-duped if needed.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

preppedRecords - hoodieRecords to upsert

Returns:

HoodieWriteMetadata

bulkInsertPrepped

public abstract HoodieWriteMetadata<O> bulkInsertPrepped(HoodieEngineContext context,
                                                         String instantTime,
                                                         I preppedRecords,
                                                         Option<BulkInsertPartitioner> bulkInsertPartitioner)

Bulk Insert the given prepared records into the Hoodie table, at the supplied instantTime.

This implementation requires that the input records are already tagged, and de-duped if needed.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

preppedRecords - hoodieRecords to upsert

bulkInsertPartitioner - User Defined Partitioner

Returns:

HoodieWriteMetadata

insertOverwrite

public abstract HoodieWriteMetadata<O> insertOverwrite(HoodieEngineContext context,
                                                       String instantTime,
                                                       I records)

Replaces all the existing records and inserts the specified new records into Hoodie table at the supplied instantTime, for the partition paths contained in input records.

Parameters:

context - HoodieEngineContext

instantTime - Instant time for the replace action

records - input records

Returns:

HoodieWriteMetadata

insertOverwriteTable

public abstract HoodieWriteMetadata<O> insertOverwriteTable(HoodieEngineContext context,
                                                            String instantTime,
                                                            I records)

Delete all the existing records of the Hoodie table and inserts the specified new records into Hoodie table at the supplied instantTime, for the partition paths contained in input records.

Parameters:

context - HoodieEngineContext

instantTime - Instant time for the replace action

records - input records

Returns:

HoodieWriteMetadata

getConfig

public HoodieWriteConfig getConfig()

getMetaClient

public HoodieTableMetaClient getMetaClient()

isPartitioned

public boolean isPartitioned()

Returns:

if the table is physically partitioned, based on the partition fields stored in the table config.

getHadoopConf

public org.apache.hadoop.conf.Configuration getHadoopConf()

getFileSystemView

public TableFileSystemView getFileSystemView()

Get the view of the file system for this table.

getBaseFileOnlyView

public TableFileSystemView.BaseFileOnlyView getBaseFileOnlyView()

Get the base file only view of the file system for this table.

getSliceView

public TableFileSystemView.SliceView getSliceView()

Get the full view of the file system for this table.

getHoodieView

public SyncableFileSystemView getHoodieView()

Get complete view of the file system for this table with ability to force sync.

getCompletedCommitsTimeline

public HoodieTimeline getCompletedCommitsTimeline()

Get only the completed (no-inflights) commit + deltacommit timeline.

getCompletedCommitTimeline

public HoodieTimeline getCompletedCommitTimeline()

Get only the completed (no-inflights) commit timeline.

getPendingCommitTimeline

public HoodieTimeline getPendingCommitTimeline()

Get only the inflights (no-completed) commit timeline.

getCompletedCleanTimeline

public HoodieTimeline getCompletedCleanTimeline()

Get only the completed (no-inflights) clean timeline.

getCleanTimeline

public HoodieTimeline getCleanTimeline()

Get clean timeline.

getRollbackTimeline

public HoodieTimeline getRollbackTimeline()

Get rollback timeline.

getRestoreTimeline

public HoodieTimeline getRestoreTimeline()

Get restore timeline.

getCompletedSavepointTimeline

public HoodieTimeline getCompletedSavepointTimeline()

Get only the completed (no-inflights) savepoint timeline.

getSavepointTimestamps

public Set<String> getSavepointTimestamps()

Get the list of savepoint timestamps in this table.

getActiveTimeline

public HoodieActiveTimeline getActiveTimeline()

getIndex

public HoodieIndex<?,?> getIndex()

Return the index.

getStorageLayout

public HoodieStorageLayout getStorageLayout()

scheduleCompaction

public abstract Option<HoodieCompactionPlan> scheduleCompaction(HoodieEngineContext context,
                                                                String instantTime,
                                                                Option<Map<String,String>> extraMetadata)

Schedule compaction for the instant time.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for scheduling compaction

extraMetadata - additional metadata to write into plan

Returns:

compact

public abstract HoodieWriteMetadata<O> compact(HoodieEngineContext context,
                                               String compactionInstantTime)

Run Compaction on the table. Compaction arranges the data so that it is optimized for data access.

Parameters:

context - HoodieEngineContext

compactionInstantTime - Instant Time

scheduleLogCompaction

public Option<HoodieCompactionPlan> scheduleLogCompaction(HoodieEngineContext context,
                                                          String instantTime,
                                                          Option<Map<String,String>> extraMetadata)

Schedule log compaction for the instant time.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for scheduling log compaction

extraMetadata - additional metadata to write into plan

Returns:

logCompact

public HoodieWriteMetadata<O> logCompact(HoodieEngineContext context,
                                         String logCompactionInstantTime)

Run Log Compaction on the table. Log Compaction arranges the data so that it is optimized for data access.

Parameters:

context - HoodieEngineContext

logCompactionInstantTime - Instant Time

scheduleClustering

public abstract Option<HoodieClusteringPlan> scheduleClustering(HoodieEngineContext context,
                                                                String instantTime,
                                                                Option<Map<String,String>> extraMetadata)

Schedule clustering for the instant time.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for scheduling clustering

extraMetadata - additional metadata to write into plan

Returns:

HoodieClusteringPlan, if there is enough data for clustering.

cluster

public abstract HoodieWriteMetadata<O> cluster(HoodieEngineContext context,
                                               String clusteringInstantTime)

Execute Clustering on the table. Clustering re-arranges the data so that it is optimized for data access.

Parameters:

context - HoodieEngineContext

clusteringInstantTime - Instant Time

bootstrap

public abstract HoodieBootstrapWriteMetadata<O> bootstrap(HoodieEngineContext context,
                                                          Option<Map<String,String>> extraMetadata)

Perform metadata/full bootstrap of a Hudi table.

Parameters:

context - HoodieEngineContext

extraMetadata - Additional Metadata for storing in commit file.

Returns:

HoodieBootstrapWriteMetadata

rollbackBootstrap

public abstract void rollbackBootstrap(HoodieEngineContext context,
                                       String instantTime)

Perform rollback of bootstrap of a Hudi table.

Parameters:

context - HoodieEngineContext

scheduleCleaning

public abstract Option<HoodieCleanerPlan> scheduleCleaning(HoodieEngineContext context,
                                                           String instantTime,
                                                           Option<Map<String,String>> extraMetadata)

Schedule cleaning for the instant time.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for scheduling cleaning

extraMetadata - additional metadata to write into plan

Returns:

HoodieCleanerPlan, if there is anything to clean.

clean

@Deprecated
public HoodieCleanMetadata clean(HoodieEngineContext context,
                                             String cleanInstantTime,
                                             boolean skipLocking)

Deprecated.

Executes a new clean action.

Returns:

information on cleaned file slices

clean

public abstract HoodieCleanMetadata clean(HoodieEngineContext context,
                                          String cleanInstantTime)

Executes a new clean action.

Returns:

information on cleaned file slices

scheduleRollback

public abstract Option<HoodieRollbackPlan> scheduleRollback(HoodieEngineContext context,
                                                            String instantTime,
                                                            HoodieInstant instantToRollback,
                                                            boolean skipTimelinePublish,
                                                            boolean shouldRollbackUsingMarkers,
                                                            boolean isRestore)

Schedule rollback for the instant time.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for scheduling rollback

instantToRollback - instant to be rolled back

shouldRollbackUsingMarkers - uses marker based rollback strategy when set to true. uses list based rollback when false.

Returns:

HoodieRollbackPlan containing info on rollback.

rollback

public abstract HoodieRollbackMetadata rollback(HoodieEngineContext context,
                                                String rollbackInstantTime,
                                                HoodieInstant commitInstant,
                                                boolean deleteInstants,
                                                boolean skipLocking)

Rollback the (inflight/committed) record changes with the given commit time.

   Three steps:
   (1) Atomically unpublish this commit
   (2) clean indexing data
   (3) clean new generated parquet files.
   (4) Finally delete .commit or .inflight file, if deleteInstants = true
 

scheduleIndexing

public abstract Option<HoodieIndexPlan> scheduleIndexing(HoodieEngineContext context,
                                                         String indexInstantTime,
                                                         List<MetadataPartitionType> partitionsToIndex)

Schedules Indexing for the table to the given instant.

Parameters:

context - HoodieEngineContext

indexInstantTime - Instant time for scheduling index action.

partitionsToIndex - List of MetadataPartitionType that should be indexed.

Returns:

HoodieIndexPlan containing metadata partitions and instant upto which they should be indexed.

index

public abstract Option<HoodieIndexCommitMetadata> index(HoodieEngineContext context,
                                                        String indexInstantTime)

Execute requested index action.

Parameters:

context - HoodieEngineContext

indexInstantTime - Instant time for which index action was scheduled.

Returns:

HoodieIndexCommitMetadata containing write stats for each metadata partition.

savepoint

public abstract HoodieSavepointMetadata savepoint(HoodieEngineContext context,
                                                  String instantToSavepoint,
                                                  String user,
                                                  String comment)

Create a savepoint at the specified instant, so that the table can be restored to this point-in-timeline later if needed.

restore

public abstract HoodieRestoreMetadata restore(HoodieEngineContext context,
                                              String restoreInstantTimestamp,
                                              String savepointToRestoreTimestamp)

Restore the table to the given instant. Note that this is a admin table recovery operation that would cause any running queries that are accessing file slices written after the instant to fail.

scheduleRestore

public abstract Option<HoodieRestorePlan> scheduleRestore(HoodieEngineContext context,
                                                          String restoreInstantTimestamp,
                                                          String savepointToRestoreTimestamp)

Schedules Restore for the table to the given instant.

rollbackInflightCompaction

public void rollbackInflightCompaction(HoodieInstant inflightInstant)

rollbackInflightLogCompaction

public void rollbackInflightLogCompaction(HoodieInstant inflightInstant)

rollbackInflightCompaction

public void rollbackInflightCompaction(HoodieInstant inflightInstant,
                                       Function<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInstantFunc)

Rollback failed compactions. Inflight rollbacks for compactions revert the .inflight file to the .requested file.

Parameters:

inflightInstant - Inflight Compaction Instant

rollbackInflightClustering

public void rollbackInflightClustering(HoodieInstant inflightInstant,
                                       Function<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInstantFunc)

Rollback inflight clustering instant to requested clustering instant

Parameters:

inflightInstant - Inflight clustering instant

getPendingRollbackInstantFunc - Function to get rollback instant

rollbackInflightLogCompaction

public void rollbackInflightLogCompaction(HoodieInstant inflightInstant,
                                          Function<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInstantFunc)

Rollback failed compactions. Inflight rollbacks for compactions revert the .inflight file to the .requested file.

Parameters:

inflightInstant - Inflight Compaction Instant

finalizeWrite

public void finalizeWrite(HoodieEngineContext context,
                          String instantTs,
                          List<HoodieWriteStat> stats)
                   throws HoodieIOException

Finalize the written data onto storage. Perform any final cleanups.

Parameters:

context - HoodieEngineContext

stats - List of HoodieWriteStats

Throws:

HoodieIOException - if some paths can't be finalized on storage

getInvalidDataPaths

protected Set<String> getInvalidDataPaths(WriteMarkers markers)
                                   throws IOException

Returns the possible invalid data file name with given marker files.

Throws:

IOException

reconcileAgainstMarkers

protected void reconcileAgainstMarkers(HoodieEngineContext context,
                                       String instantTs,
                                       List<HoodieWriteStat> stats,
                                       boolean consistencyCheckEnabled)
                                throws HoodieIOException

Reconciles WriteStats and marker files to detect and safely delete duplicate data files created because of Spark retries.

Parameters:

context - HoodieEngineContext

instantTs - Instant Timestamp

stats - Hoodie Write Stat

consistencyCheckEnabled - Consistency Check Enabled

Throws:

HoodieIOException

getConsistencyGuard

public static ConsistencyGuard getConsistencyGuard(org.apache.hadoop.fs.FileSystem fs,
                                                   ConsistencyGuardConfig consistencyGuardConfig)
                                            throws IOException

Instantiate ConsistencyGuard based on configs.

Default consistencyGuard class is OptimisticConsistencyGuard.

Throws:

IOException

getTaskContextSupplier

public TaskContextSupplier getTaskContextSupplier()

validateUpsertSchema

public void validateUpsertSchema()
                          throws HoodieUpsertException

Throws:

HoodieUpsertException

validateInsertSchema

public void validateInsertSchema()
                          throws HoodieInsertException

Throws:

HoodieInsertException

getBaseFileFormat

public HoodieFileFormat getBaseFileFormat()

getLogFileFormat

public HoodieFileFormat getLogFileFormat()

getPartitionMetafileFormat

public Option<HoodieFileFormat> getPartitionMetafileFormat()

getBaseFileExtension

public String getBaseFileExtension()

requireSortedRecords

public boolean requireSortedRecords()

getContext

public HoodieEngineContext getContext()

getMetadataWriter

public final Option<HoodieTableMetadataWriter> getMetadataWriter(String triggeringInstantTimestamp)

Get Table metadata writer.

Parameters:

triggeringInstantTimestamp - - The instant that is triggering this metadata write

Returns:

instance of HoodieTableMetadataWriter

getIndexingMetadataWriter

public Option<HoodieTableMetadataWriter> getIndexingMetadataWriter(String triggeringInstantTimestamp)

Gets the metadata writer for async indexer.

Parameters:

triggeringInstantTimestamp - The instant that is triggering this metadata write.

Returns:

An instance of HoodieTableMetadataWriter.

getMetadataWriter

protected Option<HoodieTableMetadataWriter> getMetadataWriter(String triggeringInstantTimestamp,
                                                              HoodieFailedWritesCleaningPolicy failedWritesCleaningPolicy)

Get Table metadata writer.

Note: Get the metadata writer for the conf. If the metadata table doesn't exist, this wil trigger the creation of the table and the initial bootstrapping. Since this call is under the transaction lock, other concurrent writers are blocked from doing the similar initial metadata table creation and the bootstrapping.

Parameters:

triggeringInstantTimestamp - The instant that is triggering this metadata write

failedWritesCleaningPolicy - Cleaning policy on failed writes

Returns:

instance of HoodieTableMetadataWriter

maybeDeleteMetadataTable

public void maybeDeleteMetadataTable()

Deletes the metadata table if the writer disables metadata table with hoodie.metadata.enable=false

deleteMetadataIndexIfNecessary

public void deleteMetadataIndexIfNecessary()

Deletes the metadata partition if the writer disables any metadata index.

getMetadataTable

public HoodieTableMetadata getMetadataTable()

shouldTrackSuccessRecords

public boolean shouldTrackSuccessRecords()

When HoodieTableConfig.POPULATE_META_FIELDS is enabled, we need to track written records within WriteStatus in two cases:

1. When the HoodieIndex being used is not implicit with storage
2. If any of the metadata table partitions (record index, etc) which require written record tracking are enabled

getPreExecuteRunnable

public Runnable getPreExecuteRunnable()