| Modifier and Type | Method and Description |
|---|---|
protected abstract HoodieTable<?,I,?,T> |
BaseHoodieTableServiceClient.createTable(HoodieWriteConfig config,
org.apache.hadoop.conf.Configuration hadoopConf) |
protected abstract HoodieTable<T,I,K,O> |
BaseHoodieWriteClient.createTable(HoodieWriteConfig config,
org.apache.hadoop.conf.Configuration hadoopConf) |
protected abstract HoodieTable<T,I,K,O> |
BaseHoodieWriteClient.createTable(HoodieWriteConfig config,
org.apache.hadoop.conf.Configuration hadoopConf,
HoodieTableMetaClient metaClient) |
HoodieTable |
BaseHoodieWriteClient.initTable(WriteOperationType operationType,
Option<String> instantTime)
Instantiates and initializes instance of
HoodieTable, performing crucial bootstrapping
operations such as:
NOTE: This method is engine-agnostic and SHOULD NOT be overloaded, please check on
BaseHoodieWriteClient.doInitTable(WriteOperationType, HoodieTableMetaClient, Option) instead
Checking whether upgrade/downgrade is required
Bootstrapping Metadata Table (if required)
Initializing metrics contexts
|
protected HoodieTable |
BaseHoodieWriteClient.initTable(WriteOperationType operationType,
Option<String> instantTime,
boolean initMetadataTable) |
| Modifier and Type | Method and Description |
|---|---|
protected void |
BaseHoodieTableServiceClient.archive(HoodieTable table)
Trigger archival for the table.
|
protected void |
BaseHoodieWriteClient.archive(HoodieTable table)
Trigger archival for the table.
|
protected void |
BaseHoodieWriteClient.autoArchiveOnCommit(HoodieTable table) |
protected void |
BaseHoodieWriteClient.commit(HoodieTable table,
String commitActionType,
String instantTime,
HoodieCommitMetadata metadata,
List<HoodieWriteStat> stats,
HoodieData<WriteStatus> writeStatuses) |
protected void |
BaseHoodieTableServiceClient.completeCompaction(HoodieCommitMetadata metadata,
HoodieTable table,
String compactionCommitTime)
Commit Compaction and track metrics.
|
protected void |
BaseHoodieWriteClient.completeCompaction(HoodieCommitMetadata metadata,
HoodieTable table,
String compactionCommitTime)
Commit Compaction and track metrics.
|
protected void |
BaseHoodieTableServiceClient.completeLogCompaction(HoodieCommitMetadata metadata,
HoodieTable table,
String logCompactionCommitTime)
Commit Log Compaction and track metrics.
|
protected void |
BaseHoodieWriteClient.completeLogCompaction(HoodieCommitMetadata metadata,
HoodieTable table,
String logCompactionCommitTime)
Commit Log Compaction and track metrics.
|
protected void |
BaseHoodieClient.finalizeWrite(HoodieTable table,
String instantTime,
List<HoodieWriteStat> stats)
Finalize Write operation.
|
protected void |
BaseHoodieWriteClient.mayBeCleanAndArchive(HoodieTable table)
Triggers cleaning and archival for the table of interest.
|
protected void |
BaseHoodieWriteClient.postCommit(HoodieTable table,
HoodieCommitMetadata metadata,
String instantTime,
Option<Map<String,String>> extraMetadata)
Post Commit Hook.
|
O |
BaseHoodieWriteClient.postWrite(HoodieWriteMetadata<O> result,
String instantTime,
HoodieTable hoodieTable)
Common method containing steps to be performed after write (upsert/insert/..) operations including auto-commit.
|
protected void |
BaseHoodieClient.resolveWriteConflict(HoodieTable table,
HoodieCommitMetadata metadata,
Set<String> pendingInflightAndRequestedInstants)
Resolve write conflicts before commit.
|
protected void |
BaseHoodieTableServiceClient.runAnyPendingClustering(HoodieTable table) |
protected void |
BaseHoodieTableServiceClient.runAnyPendingCompactions(HoodieTable table) |
protected void |
BaseHoodieTableServiceClient.runAnyPendingLogCompactions(HoodieTable table) |
protected void |
BaseHoodieTableServiceClient.runTableServicesInline(HoodieTable table,
HoodieCommitMetadata metadata,
Option<Map<String,String>> extraMetadata) |
protected void |
BaseHoodieWriteClient.runTableServicesInline(HoodieTable table,
HoodieCommitMetadata metadata,
Option<Map<String,String>> extraMetadata) |
protected abstract void |
BaseHoodieTableServiceClient.validateClusteringCommit(HoodieWriteMetadata<O> clusteringMetadata,
String clusteringCommitTime,
HoodieTable table) |
protected void |
BaseHoodieTableServiceClient.writeTableMetadata(HoodieTable table,
String instantTime,
HoodieCommitMetadata metadata,
HoodieData<WriteStatus> writeStatuses)
Write the HoodieCommitMetadata to metadata table if available.
|
protected void |
BaseHoodieWriteClient.writeTableMetadata(HoodieTable table,
String instantTime,
HoodieCommitMetadata metadata,
HoodieData<WriteStatus> writeStatuses)
Write the HoodieCommitMetadata to metadata table if available.
|
| Constructor and Description |
|---|
HoodieTimelineArchiver(HoodieWriteConfig config,
HoodieTable<T,I,K,O> table) |
| Modifier and Type | Method and Description |
|---|---|
static void |
HeartbeatUtils.abortIfHeartbeatExpired(String instantTime,
HoodieTable table,
HoodieHeartbeatClient heartbeatClient,
HoodieWriteConfig config)
Check if the heartbeat corresponding to instantTime has expired.
|
| Modifier and Type | Method and Description |
|---|---|
Option<HoodieCommitMetadata> |
SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(HoodieTable table,
ConcurrentOperation thisOperation,
ConcurrentOperation otherOperation) |
Option<HoodieCommitMetadata> |
ConflictResolutionStrategy.resolveConflict(HoodieTable table,
ConcurrentOperation thisOperation,
ConcurrentOperation otherOperation)
Implementations of this method will determine how to resolve a conflict between 2 commits.
|
| Modifier and Type | Method and Description |
|---|---|
static void |
DeletePartitionUtils.checkForPendingTableServiceActions(HoodieTable table,
List<String> partitionsToDrop)
Check if there are any pending table service actions (requested + inflight) on a table affecting the partitions to
be dropped.
|
static Pair<Integer,Integer> |
ArchivalUtils.getMinAndMaxInstantsToKeep(HoodieTable<?,?,?,?> table,
HoodieTableMetaClient metaClient)
getMinAndMaxInstantsToKeep is used by archival service to find the
min instants and max instants to keep in the active timeline
|
static Option<HoodieCommitMetadata> |
TransactionUtils.resolveWriteConflictIfAny(HoodieTable table,
Option<HoodieInstant> currentTxnOwnerInstant,
Option<HoodieCommitMetadata> thisCommitMetadata,
HoodieWriteConfig config,
Option<HoodieInstant> lastCompletedTxnOwnerInstant,
boolean reloadActiveTimeline,
Set<String> pendingInstants)
Resolve any write conflicts when committing data.
|
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTable |
HoodieLazyInsertIterable.hoodieTable |
| Constructor and Description |
|---|
CopyOnWriteInsertHandler(HoodieWriteConfig config,
String instantTime,
boolean areRecordsSorted,
HoodieTable hoodieTable,
String idPrefix,
TaskContextSupplier taskContextSupplier,
WriteHandleFactory writeHandleFactory) |
HoodieLazyInsertIterable(Iterator<HoodieRecord<T>> recordItr,
boolean areRecordsSorted,
HoodieWriteConfig config,
String instantTime,
HoodieTable hoodieTable,
String idPrefix,
TaskContextSupplier taskContextSupplier) |
HoodieLazyInsertIterable(Iterator<HoodieRecord<T>> recordItr,
boolean areRecordsSorted,
HoodieWriteConfig config,
String instantTime,
HoodieTable hoodieTable,
String idPrefix,
TaskContextSupplier taskContextSupplier,
WriteHandleFactory writeHandleFactory) |
| Modifier and Type | Method and Description |
|---|---|
static List<Pair<String,HoodieBaseFile>> |
HoodieIndexUtils.getLatestBaseFilesForAllPartitions(List<String> partitions,
HoodieEngineContext context,
HoodieTable hoodieTable)
Fetches Pair of partition path and
HoodieBaseFiles for interested partitions. |
static List<HoodieBaseFile> |
HoodieIndexUtils.getLatestBaseFilesForPartition(String partition,
HoodieTable hoodieTable)
Fetches Pair of partition path and
HoodieBaseFiles for interested partitions. |
static List<FileSlice> |
HoodieIndexUtils.getLatestFileSlicesForPartition(String partition,
HoodieTable hoodieTable)
Fetches Pair of partition path and
FileSlices for interested partitions. |
static <R> HoodieData<HoodieRecord<R>> |
HoodieIndexUtils.mergeForPartitionUpdatesIfNeeded(HoodieData<Pair<HoodieRecord<R>,Option<HoodieRecordGlobalLocation>>> incomingRecordsAndLocations,
HoodieWriteConfig config,
HoodieTable hoodieTable)
Merge tagged incoming records with existing records in case of partition path updated.
|
static <R> HoodieData<HoodieRecord<R>> |
HoodieIndexUtils.tagGlobalLocationBackToRecords(HoodieData<HoodieRecord<R>> incomingRecords,
HoodiePairData<String,HoodieRecordGlobalLocation> keyAndExistingLocations,
boolean mayContainDuplicateLookup,
boolean shouldUpdatePartitionPath,
HoodieWriteConfig config,
HoodieTable table) |
<R> HoodieData<HoodieRecord<R>> |
HoodieInternalProxyIndex.tagLocation(HoodieData<HoodieRecord<R>> records,
HoodieEngineContext context,
HoodieTable hoodieTable) |
abstract <R> HoodieData<HoodieRecord<R>> |
HoodieIndex.tagLocation(HoodieData<HoodieRecord<R>> records,
HoodieEngineContext context,
HoodieTable hoodieTable)
Looks up the index and tags each incoming record with a location of a file that contains
the row (if it is actually present).
|
I |
HoodieIndex.tagLocation(I records,
HoodieEngineContext context,
HoodieTable hoodieTable)
Deprecated.
|
HoodieData<WriteStatus> |
HoodieInternalProxyIndex.updateLocation(HoodieData<WriteStatus> writeStatuses,
HoodieEngineContext context,
HoodieTable hoodieTable) |
abstract HoodieData<WriteStatus> |
HoodieIndex.updateLocation(HoodieData<WriteStatus> writeStatuses,
HoodieEngineContext context,
HoodieTable hoodieTable)
Extracts the location of written records, and updates the index.
|
HoodieData<WriteStatus> |
HoodieIndex.updateLocation(HoodieData<WriteStatus> writeStatuses,
HoodieEngineContext context,
HoodieTable hoodieTable,
String instant)
Extracts the location of written records, and updates the index.
|
O |
HoodieIndex.updateLocation(O writeStatuses,
HoodieEngineContext context,
HoodieTable hoodieTable)
Deprecated.
|
| Modifier and Type | Method and Description |
|---|---|
HoodiePairData<HoodieKey,HoodieRecordLocation> |
ListBasedHoodieBloomIndexHelper.findMatchingFilesForRecordKeys(HoodieWriteConfig config,
HoodieEngineContext context,
HoodieTable hoodieTable,
HoodiePairData<String,String> partitionRecordKeyPairs,
HoodiePairData<HoodieFileGroupId,String> fileComparisonPairs,
Map<String,List<BloomIndexFileInfo>> partitionToFileInfo,
Map<String,Long> recordsPerPartition) |
abstract HoodiePairData<HoodieKey,HoodieRecordLocation> |
BaseHoodieBloomIndexHelper.findMatchingFilesForRecordKeys(HoodieWriteConfig config,
HoodieEngineContext context,
HoodieTable hoodieTable,
HoodiePairData<String,String> partitionRecordKeyPairs,
HoodiePairData<HoodieFileGroupId,String> fileComparisonPairs,
Map<String,List<BloomIndexFileInfo>> partitionToFileInfo,
Map<String,Long> recordsPerPartition)
Find out
|
protected List<Pair<String,BloomIndexFileInfo>> |
HoodieBloomIndex.loadColumnRangesFromMetaIndex(List<String> partitions,
HoodieEngineContext context,
HoodieTable<?,?,?,?> hoodieTable)
Load the column stats index as BloomIndexFileInfo for all the involved files in the partition.
|
<R> HoodieData<HoodieRecord<R>> |
HoodieBloomIndex.tagLocation(HoodieData<HoodieRecord<R>> records,
HoodieEngineContext context,
HoodieTable hoodieTable) |
protected <R> HoodieData<HoodieRecord<R>> |
HoodieGlobalBloomIndex.tagLocationBacktoRecords(HoodiePairData<HoodieKey,HoodieRecordLocation> keyLocationPairs,
HoodieData<HoodieRecord<R>> records,
HoodieTable hoodieTable)
Tagging for global index should only consider the record key.
|
protected <R> HoodieData<HoodieRecord<R>> |
HoodieBloomIndex.tagLocationBacktoRecords(HoodiePairData<HoodieKey,HoodieRecordLocation> keyFilenamePair,
HoodieData<HoodieRecord<R>> records,
HoodieTable hoodieTable)
Tag the
|
HoodieData<WriteStatus> |
HoodieBloomIndex.updateLocation(HoodieData<WriteStatus> writeStatusData,
HoodieEngineContext context,
HoodieTable hoodieTable) |
| Constructor and Description |
|---|
HoodieBloomIndexCheckFunction(HoodieTable hoodieTable,
HoodieWriteConfig config,
SerializableFunction<I,HoodieFileGroupId> fileGroupIdExtractor,
SerializableFunction<I,String> recordKeyExtractor) |
| Modifier and Type | Method and Description |
|---|---|
protected abstract BucketIndexLocationMapper |
HoodieBucketIndex.getLocationMapper(HoodieTable table,
List<String> partitionPath)
Get a location mapper for the given table & partitionPath
|
protected BucketIndexLocationMapper |
HoodieConsistentBucketIndex.getLocationMapper(HoodieTable table,
List<String> partitionPath) |
protected BucketIndexLocationMapper |
HoodieSimpleBucketIndex.getLocationMapper(HoodieTable table,
List<String> partitionPath) |
Map<Integer,HoodieRecordLocation> |
HoodieSimpleBucketIndex.loadBucketIdToFileIdMappingForPartition(HoodieTable hoodieTable,
String partition) |
static Option<HoodieConsistentHashingMetadata> |
ConsistentBucketIndexUtils.loadMetadata(HoodieTable table,
String partition)
Loads hashing metadata of the given partition, if it does not exist, returns empty.
|
static HoodieConsistentHashingMetadata |
ConsistentBucketIndexUtils.loadOrCreateMetadata(HoodieTable table,
String partition,
int numBuckets)
Loads hashing metadata of the given partition, if it does not exist, creates a new one (also persist it into storage).
|
static boolean |
ConsistentBucketIndexUtils.saveMetadata(HoodieTable table,
HoodieConsistentHashingMetadata metadata,
boolean overwrite)
Saves the metadata into storage
|
<R> HoodieData<HoodieRecord<R>> |
HoodieBucketIndex.tagLocation(HoodieData<HoodieRecord<R>> records,
HoodieEngineContext context,
HoodieTable hoodieTable) |
HoodieData<WriteStatus> |
HoodieBucketIndex.updateLocation(HoodieData<WriteStatus> writeStatuses,
HoodieEngineContext context,
HoodieTable hoodieTable) |
HoodieData<WriteStatus> |
HoodieConsistentBucketIndex.updateLocation(HoodieData<WriteStatus> writeStatuses,
HoodieEngineContext context,
HoodieTable hoodieTable) |
| Constructor and Description |
|---|
ConsistentBucketIndexLocationMapper(HoodieTable table,
List<String> partitions) |
SimpleBucketIndexLocationMapper(HoodieTable table,
List<String> partitions) |
| Modifier and Type | Method and Description |
|---|---|
<R> HoodieData<HoodieRecord<R>> |
HoodieInMemoryHashIndex.tagLocation(HoodieData<HoodieRecord<R>> records,
HoodieEngineContext context,
HoodieTable hoodieTable) |
HoodieData<WriteStatus> |
HoodieInMemoryHashIndex.updateLocation(HoodieData<WriteStatus> writeStatuses,
HoodieEngineContext context,
HoodieTable hoodieTable) |
| Modifier and Type | Method and Description |
|---|---|
protected HoodiePairData<HoodieKey,HoodieRecordLocation> |
HoodieSimpleIndex.fetchRecordLocations(HoodieEngineContext context,
HoodieTable hoodieTable,
int parallelism,
List<Pair<String,HoodieBaseFile>> baseFiles) |
protected HoodiePairData<HoodieKey,HoodieRecordLocation> |
HoodieSimpleIndex.fetchRecordLocationsForAffectedPartitions(HoodieData<HoodieKey> hoodieKeys,
HoodieEngineContext context,
HoodieTable hoodieTable,
int parallelism)
Fetch record locations for passed in
HoodieKeys. |
<R> HoodieData<HoodieRecord<R>> |
HoodieGlobalSimpleIndex.tagLocation(HoodieData<HoodieRecord<R>> records,
HoodieEngineContext context,
HoodieTable hoodieTable) |
<R> HoodieData<HoodieRecord<R>> |
HoodieSimpleIndex.tagLocation(HoodieData<HoodieRecord<R>> records,
HoodieEngineContext context,
HoodieTable hoodieTable) |
protected <R> HoodieData<HoodieRecord<R>> |
HoodieGlobalSimpleIndex.tagLocationInternal(HoodieData<HoodieRecord<R>> inputRecords,
HoodieEngineContext context,
HoodieTable hoodieTable)
Tags records location for incoming records.
|
protected <R> HoodieData<HoodieRecord<R>> |
HoodieSimpleIndex.tagLocationInternal(HoodieData<HoodieRecord<R>> inputRecords,
HoodieEngineContext context,
HoodieTable hoodieTable)
Tags records location for incoming records.
|
HoodieData<WriteStatus> |
HoodieSimpleIndex.updateLocation(HoodieData<WriteStatus> writeStatuses,
HoodieEngineContext context,
HoodieTable hoodieTable) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTable<T,I,K,O> |
HoodieIOHandle.hoodieTable |
| Modifier and Type | Method and Description |
|---|---|
static <T,I,K,O> HoodieMergeHandle<T,I,K,O> |
HoodieMergeHandleFactory.create(HoodieWriteConfig writeConfig,
String instantTime,
HoodieTable<T,I,K,O> table,
Map<String,HoodieRecord<T>> keyToNewRecords,
String partitionPath,
String fileId,
HoodieBaseFile dataFileToBeMerged,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt)
Creates a merge handle for compaction path.
|
abstract HoodieWriteHandle<T,I,K,O> |
WriteHandleFactory.create(HoodieWriteConfig config,
String commitTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileIdPrefix,
TaskContextSupplier taskContextSupplier) |
HoodieAppendHandle<T,I,K,O> |
AppendHandleFactory.create(HoodieWriteConfig hoodieConfig,
String commitTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileIdPrefix,
TaskContextSupplier sparkTaskContextSupplier) |
HoodieWriteHandle<T,I,K,O> |
CreateHandleFactory.create(HoodieWriteConfig hoodieConfig,
String commitTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileIdPrefix,
TaskContextSupplier taskContextSupplier) |
HoodieWriteHandle<T,I,K,O> |
SingleFileHandleCreateFactory.create(HoodieWriteConfig hoodieConfig,
String commitTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileIdPrefix,
TaskContextSupplier taskContextSupplier) |
static <T,I,K,O> HoodieMergeHandle<T,I,K,O> |
HoodieMergeHandleFactory.create(WriteOperationType operationType,
HoodieWriteConfig writeConfig,
String instantTime,
HoodieTable<T,I,K,O> table,
Iterator<HoodieRecord<T>> recordItr,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt)
Creates a merge handle for normal write path.
|
static HoodieBaseFile |
HoodieMergeHandle.getLatestBaseFile(HoodieTable<?,?,?,?> hoodieTable,
String partitionPath,
String fileId) |
| Constructor and Description |
|---|
HoodieAppendHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
Iterator<HoodieRecord<T>> recordItr,
TaskContextSupplier taskContextSupplier) |
HoodieAppendHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
Iterator<HoodieRecord<T>> recordItr,
TaskContextSupplier taskContextSupplier,
Map<HoodieLogBlock.HeaderMetadataType,String> header)
This is used by log compaction only.
|
HoodieAppendHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
TaskContextSupplier sparkTaskContextSupplier) |
HoodieBootstrapHandle(HoodieWriteConfig config,
String commitTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier) |
HoodieConcatHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Iterator<HoodieRecord<T>> recordItr,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt) |
HoodieConcatHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable hoodieTable,
Map<String,HoodieRecord<T>> keyToNewRecords,
String partitionPath,
String fileId,
HoodieBaseFile dataFileToBeMerged,
TaskContextSupplier taskContextSupplier) |
HoodieCreateHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
Map<String,HoodieRecord<T>> recordMap,
TaskContextSupplier taskContextSupplier)
Called by the compactor code path.
|
HoodieCreateHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
Option<org.apache.avro.Schema> overriddenSchema,
TaskContextSupplier taskContextSupplier) |
HoodieCreateHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
Option<org.apache.avro.Schema> overriddenSchema,
TaskContextSupplier taskContextSupplier,
boolean preserveMetadata) |
HoodieCreateHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier) |
HoodieCreateHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
boolean preserveMetadata) |
HoodieKeyLocationFetchHandle(HoodieWriteConfig config,
HoodieTable<T,I,K,O> hoodieTable,
Pair<String,HoodieBaseFile> partitionPathBaseFilePair,
Option<BaseKeyGenerator> keyGeneratorOpt) |
HoodieKeyLookupHandle(HoodieWriteConfig config,
HoodieTable<T,I,K,O> hoodieTable,
Pair<String,String> partitionPathFileIDPair) |
HoodieMergedReadHandle(HoodieWriteConfig config,
Option<String> instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Pair<String,String> partitionPathFileIDPair) |
HoodieMergeHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Iterator<HoodieRecord<T>> recordItr,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
HoodieBaseFile baseFile,
Option<BaseKeyGenerator> keyGeneratorOpt) |
HoodieMergeHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Iterator<HoodieRecord<T>> recordItr,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt) |
HoodieMergeHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Map<String,HoodieRecord<T>> keyToNewRecords,
String partitionPath,
String fileId,
HoodieBaseFile dataFileToBeMerged,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt)
Called by compactor code path.
|
HoodieMergeHandleWithChangeLog(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Iterator<HoodieRecord<T>> recordItr,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt) |
HoodieMergeHandleWithChangeLog(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Map<String,HoodieRecord<T>> keyToNewRecords,
String partitionPath,
String fileId,
HoodieBaseFile dataFileToBeMerged,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt)
Called by compactor code path.
|
HoodieRangeInfoHandle(HoodieWriteConfig config,
HoodieTable<T,I,K,O> hoodieTable,
Pair<String,String> partitionPathFilePair) |
HoodieReadHandle(HoodieWriteConfig config,
HoodieTable<T,I,K,O> hoodieTable,
Pair<String,String> partitionPathFileIDPair) |
HoodieReadHandle(HoodieWriteConfig config,
Option<String> instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Pair<String,String> partitionPathFileIDPair) |
HoodieSortedMergeHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Iterator<HoodieRecord<T>> recordItr,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt) |
HoodieSortedMergeHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Map<String,HoodieRecord<T>> keyToNewRecordsOrig,
String partitionPath,
String fileId,
HoodieBaseFile dataFileToBeMerged,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt)
Called by compactor code path.
|
HoodieSortedMergeHandleWithChangeLog(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Iterator<HoodieRecord<T>> recordItr,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt) |
HoodieSortedMergeHandleWithChangeLog(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
Map<String,HoodieRecord<T>> keyToNewRecords,
String partitionPath,
String fileId,
HoodieBaseFile dataFileToBeMerged,
TaskContextSupplier taskContextSupplier,
Option<BaseKeyGenerator> keyGeneratorOpt)
Called by compactor code path.
|
HoodieUnboundedCreateHandle(HoodieWriteConfig config,
String instantTime,
HoodieTable<T,I,K,O> hoodieTable,
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
boolean preserveHoodieMetadata) |
HoodieWriteHandle(HoodieWriteConfig config,
String instantTime,
String partitionPath,
String fileId,
HoodieTable<T,I,K,O> hoodieTable,
Option<org.apache.avro.Schema> overriddenSchema,
TaskContextSupplier taskContextSupplier) |
HoodieWriteHandle(HoodieWriteConfig config,
String instantTime,
String partitionPath,
String fileId,
HoodieTable<T,I,K,O> hoodieTable,
TaskContextSupplier taskContextSupplier) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTable |
BucketIndexBulkInsertPartitioner.table |
| Constructor and Description |
|---|
BucketIndexBulkInsertPartitioner(HoodieTable table,
String sortString,
boolean preserveHoodieMetadata) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTable<T,I,K,O> |
BaseActionExecutor.table |
| Constructor and Description |
|---|
BaseActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime) |
| Constructor and Description |
|---|
CleanActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime) |
CleanActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
boolean skipLocking) |
CleanPlanActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
Option<Map<String,String>> extraMetadata) |
CleanPlanner(HoodieEngineContext context,
HoodieTable<T,I,K,O> hoodieTable,
HoodieWriteConfig config) |
| Constructor and Description |
|---|
ClusteringPlanActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
Option<Map<String,String>> extraMetadata) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTable |
UpdateStrategy.table |
| Modifier and Type | Method and Description |
|---|---|
protected HoodieTable<T,I,K,O> |
ClusteringPlanStrategy.getHoodieTable() |
protected HoodieTable<T,I,K,O> |
ClusteringExecutionStrategy.getHoodieTable() |
| Constructor and Description |
|---|
BaseConsistentHashingBucketClusteringPlanStrategy(HoodieTable table,
HoodieEngineContext engineContext,
HoodieWriteConfig writeConfig) |
ClusteringExecutionStrategy(HoodieTable table,
HoodieEngineContext engineContext,
HoodieWriteConfig writeConfig) |
ClusteringPlanStrategy(HoodieTable table,
HoodieEngineContext engineContext,
HoodieWriteConfig writeConfig) |
PartitionAwareClusteringPlanStrategy(HoodieTable table,
HoodieEngineContext engineContext,
HoodieWriteConfig writeConfig) |
UpdateStrategy(HoodieEngineContext engineContext,
HoodieTable table,
Set<HoodieFileGroupId> fileGroupsInPendingClustering) |
| Modifier and Type | Method and Description |
|---|---|
static Map<String,Pair<String,ConsistentBucketIdentifier>> |
ConsistentHashingUpdateStrategyUtils.constructPartitionToIdentifier(Set<String> partitions,
HoodieTable table)
Construct identifier for the given partitions that are under concurrent resizing (i.e., clustering).
|
| Modifier and Type | Method and Description |
|---|---|
abstract HoodieWriteMetadata<O> |
BaseBulkInsertHelper.bulkInsert(I inputRecords,
String instantTime,
HoodieTable<T,I,K,O> table,
HoodieWriteConfig config,
BaseCommitActionExecutor<T,I,K,O,R> executor,
boolean performDedupe,
Option<BulkInsertPartitioner> userDefinedBulkInsertPartitioner)
Mark instant as inflight, write input records, update index and return result.
|
abstract O |
BaseBulkInsertHelper.bulkInsert(I inputRecords,
String instantTime,
HoodieTable<T,I,K,O> table,
HoodieWriteConfig config,
boolean performDedupe,
BulkInsertPartitioner partitioner,
boolean addMetadataFields,
int parallelism,
WriteHandleFactory writeHandleFactory)
Only write input records.
|
I |
BaseWriteHelper.combineOnCondition(boolean condition,
I records,
int parallelism,
HoodieTable<T,I,K,O> table) |
HoodieData<HoodieKey> |
HoodieDeleteHelper.deduplicateKeys(HoodieData<HoodieKey> keys,
HoodieTable<T,HoodieData<HoodieRecord<T>>,HoodieData<HoodieKey>,HoodieData<WriteStatus>> table,
int parallelism) |
abstract K |
BaseDeleteHelper.deduplicateKeys(K keys,
HoodieTable<T,I,K,O> table,
int parallelism)
Deduplicate Hoodie records, using the given deduplication function.
|
I |
BaseWriteHelper.deduplicateRecords(I records,
HoodieTable<T,I,K,O> table,
int parallelism)
Deduplicate Hoodie records, using the given deduplication function.
|
HoodieWriteMetadata<HoodieData<WriteStatus>> |
HoodieDeleteHelper.execute(String instantTime,
HoodieData<HoodieKey> keys,
HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,HoodieData<HoodieRecord<T>>,HoodieData<HoodieKey>,HoodieData<WriteStatus>> table,
BaseCommitActionExecutor<T,HoodieData<HoodieRecord<T>>,HoodieData<HoodieKey>,HoodieData<WriteStatus>,R> deleteExecutor) |
abstract HoodieWriteMetadata<O> |
BaseDeleteHelper.execute(String instantTime,
K keys,
HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
BaseCommitActionExecutor<T,I,K,O,R> deleteExecutor) |
void |
HoodieMergeHelper.runMerge(HoodieTable<?,?,?,?> table,
HoodieMergeHandle<?,?,?,?> mergeHandle) |
abstract void |
BaseMergeHelper.runMerge(HoodieTable<?,?,?,?> table,
HoodieMergeHandle<?,?,?,?> upsertHandle)
Read records from previous version of base file and merge.
|
protected HoodieData<HoodieRecord<T>> |
HoodieWriteHelper.tag(HoodieData<HoodieRecord<T>> dedupedRecords,
HoodieEngineContext context,
HoodieTable<T,HoodieData<HoodieRecord<T>>,HoodieData<HoodieKey>,HoodieData<WriteStatus>> table) |
protected abstract I |
BaseWriteHelper.tag(I dedupedRecords,
HoodieEngineContext context,
HoodieTable<T,I,K,O> table) |
HoodieWriteMetadata<O> |
BaseWriteHelper.write(String instantTime,
I inputRecords,
HoodieEngineContext context,
HoodieTable<T,I,K,O> table,
boolean shouldCombine,
int configuredShuffleParallelism,
BaseCommitActionExecutor<T,I,K,O,R> executor,
WriteOperationType operationType) |
| Constructor and Description |
|---|
BaseCommitActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
WriteOperationType operationType,
Option<Map<String,String>> extraMetadata) |
| Modifier and Type | Method and Description |
|---|---|
HoodieData<WriteStatus> |
HoodieCompactor.compact(HoodieEngineContext context,
HoodieCompactionPlan compactionPlan,
HoodieTable table,
HoodieWriteConfig config,
String compactionInstantTime,
HoodieCompactionHandler compactionHandler)
Execute compaction operations and report back status.
|
void |
CompactHelpers.completeInflightCompaction(HoodieTable table,
String compactionCommitTime,
HoodieCommitMetadata commitMetadata) |
void |
CompactHelpers.completeInflightLogCompaction(HoodieTable table,
String logCompactionCommitTime,
HoodieCommitMetadata commitMetadata) |
HoodieCommitMetadata |
CompactHelpers.createCompactionMetadata(HoodieTable table,
String compactionInstantTime,
HoodieData<WriteStatus> writeStatuses,
String schema) |
abstract void |
HoodieCompactor.preCompact(HoodieTable table,
HoodieTimeline pendingCompactionTimeline,
WriteOperationType operationType,
String instantTime)
Handles the compaction timeline based on the compaction instant before actual compaction.
|
protected void |
CompactionExecutionHelper.transitionRequestedToInflight(HoodieTable table,
String compactionInstantTime) |
protected void |
LogCompactionExecutionHelper.transitionRequestedToInflight(HoodieTable table,
String logCompactionInstantTime) |
| Constructor and Description |
|---|
RunCompactionActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable table,
String instantTime,
HoodieCompactor compactor,
HoodieCompactionHandler compactionHandler,
WriteOperationType operationType) |
ScheduleCompactionActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
Option<Map<String,String>> extraMetadata,
WriteOperationType operationType) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTable<T,I,K,O> |
BaseHoodieCompactionPlanGenerator.hoodieTable |
| Constructor and Description |
|---|
BaseHoodieCompactionPlanGenerator(HoodieTable table,
HoodieEngineContext engineContext,
HoodieWriteConfig writeConfig) |
HoodieCompactionPlanGenerator(HoodieTable table,
HoodieEngineContext engineContext,
HoodieWriteConfig writeConfig) |
HoodieLogCompactionPlanGenerator(HoodieTable table,
HoodieEngineContext engineContext,
HoodieWriteConfig writeConfig) |
| Constructor and Description |
|---|
RunIndexActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime) |
ScheduleIndexActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
List<MetadataPartitionType> partitionIndexTypes) |
| Modifier and Type | Method and Description |
|---|---|
static String |
RestoreUtils.getSavepointToRestoreTimestamp(HoodieTable table,
HoodieInstant restoreInstant)
Get the savepoint timestamp that this restore instant is restoring
|
static String |
RestoreUtils.getSavepointToRestoreTimestampV1Schema(HoodieTable table,
HoodieRestorePlan plan) |
| Constructor and Description |
|---|
BaseRestoreActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
String savepointToRestoreTimestamp) |
CopyOnWriteRestoreActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable table,
String instantTime,
String savepointToRestoreTimestamp) |
MergeOnReadRestoreActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
String savepointToRestoreTimestamp) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTable<?,?,?,?> |
MarkerBasedRollbackStrategy.table |
protected HoodieTable<?,?,?,?> |
ListingBasedRollbackStrategy.table |
| Modifier and Type | Method and Description |
|---|---|
static List<HoodieRollbackRequest> |
ListingBasedRollbackStrategy.getRollbackRequestToAppend(String partitionPath,
HoodieInstant rollbackInstant,
HoodieCommitMetadata commitMetadata,
HoodieTable<?,?,?,?> table) |
| Constructor and Description |
|---|
BaseRollbackActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
HoodieInstant instantToRollback,
boolean deleteInstants,
boolean skipLocking,
boolean isRestore) |
BaseRollbackActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
HoodieInstant instantToRollback,
boolean deleteInstants,
boolean skipTimelinePublish,
boolean skipLocking,
boolean isRestore) |
BaseRollbackPlanActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
HoodieInstant instantToRollback,
boolean skipTimelinePublish,
boolean shouldRollbackUsingMarkers,
boolean isRestore) |
CopyOnWriteRollbackActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
HoodieInstant commitInstant,
boolean deleteInstants,
boolean skipLocking) |
CopyOnWriteRollbackActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
HoodieInstant commitInstant,
boolean deleteInstants,
boolean skipTimelinePublish,
boolean skipLocking) |
ListingBasedRollbackStrategy(HoodieTable<?,?,?,?> table,
HoodieEngineContext context,
HoodieWriteConfig config,
String instantTime,
boolean isRestore) |
MarkerBasedRollbackStrategy(HoodieTable<?,?,?,?> table,
HoodieEngineContext context,
HoodieWriteConfig config,
String instantTime) |
MergeOnReadRollbackActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
HoodieInstant commitInstant,
boolean deleteInstants,
boolean skipLocking) |
MergeOnReadRollbackActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
HoodieInstant commitInstant,
boolean deleteInstants,
boolean skipTimelinePublish,
boolean skipLocking) |
RestorePlanActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
String savepointToRestoreTimestamp) |
| Modifier and Type | Method and Description |
|---|---|
static void |
SavepointHelpers.deleteSavepoint(HoodieTable table,
String savepointTime) |
static void |
SavepointHelpers.validateSavepointPresence(HoodieTable table,
String savepointTime) |
static void |
SavepointHelpers.validateSavepointRestore(HoodieTable table,
String savepointTime) |
| Constructor and Description |
|---|
SavepointActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T,I,K,O> table,
String instantTime,
String user,
String comment) |
| Modifier and Type | Method and Description |
|---|---|
static WriteMarkers |
WriteMarkersFactory.get(MarkerType markerType,
HoodieTable table,
String instantTime) |
static List<String> |
MarkerBasedRollbackUtils.getAllMarkerPaths(HoodieTable table,
HoodieEngineContext context,
String instant,
int parallelism)
Gets all marker paths.
|
| Constructor and Description |
|---|
DirectWriteMarkers(HoodieTable table,
String instantTime) |
TimelineServerBasedWriteMarkers(HoodieTable table,
String instantTime) |
| Modifier and Type | Method and Description |
|---|---|
HoodieTable |
SupportsUpgradeDowngrade.getTable(HoodieWriteConfig config,
HoodieEngineContext context) |
| Modifier and Type | Method and Description |
|---|---|
protected void |
ZeroToOneUpgradeHandler.recreateMarkers(String commitInstantTime,
HoodieTable table,
HoodieEngineContext context,
int parallelism)
Recreate markers in new format.
|
Copyright © 2023 The Apache Software Foundation. All rights reserved.