org.apache.hudi.common.table.view

Class IncrementalTimelineSyncFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.AbstractTableFileSystemView

org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView

All Implemented Interfaces:

Serializable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Direct Known Subclasses:

HoodieTableFileSystemView, RocksDbBasedFileSystemView

public abstract class IncrementalTimelineSyncFileSystemView
extends AbstractTableFileSystemView

Adds the capability to incrementally sync the changes to file-system view as and when new instants gets completed.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Field Summary

Fields inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

metaClient, readLock, writeLock

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	IncrementalTimelineSyncFileSystemView(boolean enableIncrementalTimelineSync)

Method Summary

Modifier and Type	Method and Description
protected void	applyDeltaFileSlicesToPartitionView(String partition, List<HoodieFileGroup> deltaFileGroups, org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView.DeltaApplyMode mode) Apply changes to partition file-system view.
HoodieTimeline	getTimeline() Timeline corresponding to the view.
protected void	maySyncIncrementally()
protected void	refreshTimeline(HoodieTimeline visibleActiveTimeline) Refresh commits timeline.
void	sync() Syncs the file system view from storage to memory.

Methods inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addFilesToView, addFilesToView, addReplacedFileGroups, buildFileGroups, buildFileGroups, clear, close, fetchFileGroupsInPendingClustering, fetchLatestBaseFile, fetchLatestBaseFiles, fetchLatestFileSlice, filterBaseFileAfterPendingCompaction, getAllBaseFiles, getAllFileGroups, getAllFileGroupsStateless, getAllFileSlices, getAllLatestBaseFilesBeforeOrOn, getAllLatestFileSlicesBeforeOrOn, getAllLogsMergedFileSliceBeforeOrOn, getAllPartitionPaths, getAllReplacedFileGroups, getBaseFileOn, getBootstrapBaseFile, getFileGroupsInPendingClustering, getLastInstant, getLatestBaseFile, getLatestBaseFile, getLatestBaseFiles, getLatestBaseFiles, getLatestBaseFilesBeforeOrOn, getLatestBaseFilesInRange, getLatestFileSlice, getLatestFileSliceInRange, getLatestFileSlices, getLatestFileSlicesBeforeOrOn, getLatestFileSlicesStateless, getLatestMergedFileSlicesBeforeOrOn, getLatestUnCompactedFileSlices, getPartitionPaths, getPendingClusteringInstant, getPendingCompactionOperations, getPendingCompactionOperationWithInstant, getPendingLogCompactionOperations, getPendingLogCompactionOperationWithInstant, getReplacedFileGroupsAfterOrOn, getReplacedFileGroupsBefore, getReplacedFileGroupsBeforeOrOn, getReplaceInstant, getVisibleCommitsAndCompactionTimeline, hasReplacedFilesInPartition, init, isBaseFileDueToPendingClustering, isBaseFileDueToPendingCompaction, isBootstrapBaseFilePresentForFileId, isFileSliceAfterPendingCompaction, isPendingClusteringScheduledForFileId, isPendingCompactionScheduledForFileId, isPendingLogCompactionScheduledForFileId, listPartition, listPartitions, loadAllPartitions, loadPartitions, removeReplacedFileIdsAtInstants, reset, resetReplacedFileGroups, resetViewState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IncrementalTimelineSyncFileSystemView

protected IncrementalTimelineSyncFileSystemView(boolean enableIncrementalTimelineSync)

Method Detail

refreshTimeline

protected void refreshTimeline(HoodieTimeline visibleActiveTimeline)

Description copied from class: AbstractTableFileSystemView

Refresh commits timeline.

Overrides:

refreshTimeline in class AbstractTableFileSystemView

Parameters:

visibleActiveTimeline - Visible Active Timeline

sync

public void sync()

Description copied from class: AbstractTableFileSystemView

Syncs the file system view from storage to memory. Performs complete reset of file-system view. Subsequent partition view calls will load file slices against the latest timeline.

NOTE: The logic MUST BE guarded by the write lock.

Specified by:

sync in interface SyncableFileSystemView

Overrides:

sync in class AbstractTableFileSystemView

maySyncIncrementally

protected void maySyncIncrementally()

applyDeltaFileSlicesToPartitionView

protected void applyDeltaFileSlicesToPartitionView(String partition,
                                                   List<HoodieFileGroup> deltaFileGroups,
                                                   org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView.DeltaApplyMode mode)

Apply changes to partition file-system view. Base Implementation overwrites the entire partitions view assuming some sort of map (in-mem/disk-based) is used. For View implementation which supports fine-granular updates (e:g RocksDB), override this method.

Parameters:

partition - PartitionPath

deltaFileGroups - Changed file-slices aggregated as file-groups

mode - Delta Apply mode

getTimeline

public HoodieTimeline getTimeline()

Description copied from interface: TableFileSystemView

Timeline corresponding to the view.

Specified by:

getTimeline in interface TableFileSystemView

Overrides:

getTimeline in class AbstractTableFileSystemView