org.apache.hudi.common.table.view

Class HoodieTableFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.AbstractTableFileSystemView

org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView

org.apache.hudi.common.table.view.HoodieTableFileSystemView

All Implemented Interfaces:

Serializable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Direct Known Subclasses:

HoodieMetadataFileSystemView, SpillableMapBasedFileSystemView

public class HoodieTableFileSystemView
extends IncrementalTimelineSyncFileSystemView

TableFileSystemView Implementations based on in-memory storage.

Since:

0.3.0

See Also:

TableFileSystemView, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<HoodieFileGroupId,BootstrapBaseFileMapping>	fgIdToBootstrapBaseFile PartitionPath + File-Id to bootstrap base File (Index Only bootstrapped).
protected Map<HoodieFileGroupId,HoodieInstant>	fgIdToPendingClustering Track file groups in pending clustering.
protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>>	fgIdToPendingCompaction PartitionPath + File-Id to pending compaction instant time.
protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>>	fgIdToPendingLogCompaction PartitionPath + File-Id to pending logcompaction instant time.
protected Map<HoodieFileGroupId,HoodieInstant>	fgIdToReplaceInstants Track replace time for replaced file groups.
protected Map<String,List<HoodieFileGroup>>	partitionToFileGroupsMap

Fields inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

metaClient, readLock, writeLock

Constructor Summary

Constructors

Constructor and Description
HoodieTableFileSystemView(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline) Create a file system view, as of the given timeline.
HoodieTableFileSystemView(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline, boolean enableIncrementalTimelineSync) Create a file system view, as of the given timeline.
HoodieTableFileSystemView(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline, List<StoragePathInfo> pathInfoList) Create a file system view, as of the given timeline, with the provided file statuses.

Method Summary

Modifier and Type	Method and Description
protected void	addPendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) Add pending compaction operations to store.
protected void	addPendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) Add pending Log compaction operations to store.
protected void	addReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups) Track instant time for new file groups replaced.
void	close() Allow View to release resources and close.
protected Map<HoodieFileGroupId,BootstrapBaseFileMapping>	createFileIdToBootstrapBaseFileMap(Map<HoodieFileGroupId,BootstrapBaseFileMapping> fileGroupIdBootstrapBaseFileMap)
protected Map<HoodieFileGroupId,HoodieInstant>	createFileIdToPendingClusteringMap(Map<HoodieFileGroupId,HoodieInstant> fileGroupsInClustering)
protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>>	createFileIdToPendingCompactionMap(Map<HoodieFileGroupId,Pair<String,CompactionOperation>> fileIdToPendingCompaction)
protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>>	createFileIdToPendingLogCompactionMap(Map<HoodieFileGroupId,Pair<String,CompactionOperation>> fileIdToPendingLogCompaction)
protected Map<HoodieFileGroupId,HoodieInstant>	createFileIdToReplaceInstantMap(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)
protected Map<String,List<HoodieFileGroup>>	createPartitionToFileGroups()
Stream<HoodieFileGroup>	fetchAllStoredFileGroups() Fetch all Stored file-groups across all partitions loaded.
protected Stream<Pair<HoodieFileGroupId,HoodieInstant>>	fetchFileGroupsInPendingClustering() Fetch all file groups in pending clustering.
Stream<FileSlice>	fetchLatestFileSlicesIncludingInflight(String partitionPath) Get the latest file slices for a given partition including the inflight ones.
Stream<HoodieFileGroup>	getAllFileGroups()
protected Option<BootstrapBaseFileMapping>	getBootstrapBaseFile(HoodieFileGroupId fileGroupId) Return pending compaction operation for a file-group.
protected Option<HoodieInstant>	getPendingClusteringInstant(HoodieFileGroupId fgId) Get pending clustering instant time for specified file group.
protected Option<Pair<String,CompactionOperation>>	getPendingCompactionOperationWithInstant(HoodieFileGroupId fgId) Return pending compaction operation for a file-group.
protected Option<Pair<String,CompactionOperation>>	getPendingLogCompactionOperationWithInstant(HoodieFileGroupId fgId) Return pending Log compaction operation for a file-group.
protected Option<HoodieInstant>	getReplaceInstant(HoodieFileGroupId fileGroupId) Track instant time for file groups replaced.
protected boolean	hasReplacedFilesInPartition(String partitionPath) Returns whether there are replaced files within the given partition.
void	init(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline) Initialize the view.
void	init(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline, List<StoragePathInfo> pathInfoList) Visible for testing
protected boolean	isBootstrapBaseFilePresentForFileId(HoodieFileGroupId fgId) Check if there is an bootstrap base file present for this file.
boolean	isClosed() Check if the view is already closed.
protected boolean	isPartitionAvailableInStore(String partitionPath) Checks if partition is pre-loaded and available in store.
protected boolean	isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId) Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.
protected boolean	isPendingCompactionScheduledForFileId(HoodieFileGroupId fgId) Check if there is an outstanding compaction scheduled for this file.
protected boolean	isPendingLogCompactionScheduledForFileId(HoodieFileGroupId fgId) Check if there is an outstanding log compaction scheduled for this file.
protected void	removePendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) Remove pending compaction operations from store.
protected void	removePendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) Remove pending Log compaction operations from store.
protected void	removeReplacedFileIdsAtInstants(Set<String> instants) Remove file groups that are replaced in any of the specified instants.
protected void	resetPendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) resets the pending compaction operation and overwrite with the new list.
protected void	resetPendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) resets the pending Log compaction operation and overwrite with the new list.
protected void	resetReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups) Track instant time for file groups replaced.
protected void	resetViewState() Allows all view metadata in file system view storage to be reset by subclasses.
protected void	storePartitionView(String partitionPath, List<HoodieFileGroup> fileGroups) Add a complete partition view to store.

Methods inherited from class org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView

applyDeltaFileSlicesToPartitionView, getTimeline, maySyncIncrementally, refreshTimeline, sync

Methods inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addFilesToView, addFilesToView, buildFileGroups, buildFileGroups, clear, fetchLatestBaseFile, fetchLatestBaseFiles, fetchLatestFileSlice, filterBaseFileAfterPendingCompaction, getAllBaseFiles, getAllFileGroups, getAllFileGroupsStateless, getAllFileSlices, getAllLatestBaseFilesBeforeOrOn, getAllLatestFileSlicesBeforeOrOn, getAllLogsMergedFileSliceBeforeOrOn, getAllPartitionPaths, getAllReplacedFileGroups, getBaseFileOn, getFileGroupsInPendingClustering, getLastInstant, getLatestBaseFile, getLatestBaseFile, getLatestBaseFiles, getLatestBaseFiles, getLatestBaseFilesBeforeOrOn, getLatestBaseFilesInRange, getLatestFileSlice, getLatestFileSliceInRange, getLatestFileSlices, getLatestFileSlicesBeforeOrOn, getLatestFileSlicesStateless, getLatestMergedFileSlicesBeforeOrOn, getLatestUnCompactedFileSlices, getPartitionPaths, getPendingCompactionOperations, getPendingLogCompactionOperations, getReplacedFileGroupsAfterOrOn, getReplacedFileGroupsBefore, getReplacedFileGroupsBeforeOrOn, getVisibleCommitsAndCompactionTimeline, isBaseFileDueToPendingClustering, isBaseFileDueToPendingCompaction, isFileSliceAfterPendingCompaction, listPartition, listPartitions, loadAllPartitions, loadPartitions, reset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

partitionToFileGroupsMap

protected Map<String,List<HoodieFileGroup>> partitionToFileGroupsMap

fgIdToPendingCompaction

protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>> fgIdToPendingCompaction

PartitionPath + File-Id to pending compaction instant time.

fgIdToPendingLogCompaction

protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>> fgIdToPendingLogCompaction

PartitionPath + File-Id to pending logcompaction instant time.

fgIdToBootstrapBaseFile

protected Map<HoodieFileGroupId,BootstrapBaseFileMapping> fgIdToBootstrapBaseFile

PartitionPath + File-Id to bootstrap base File (Index Only bootstrapped).

fgIdToReplaceInstants

protected Map<HoodieFileGroupId,HoodieInstant> fgIdToReplaceInstants

Track replace time for replaced file groups.

fgIdToPendingClustering

protected Map<HoodieFileGroupId,HoodieInstant> fgIdToPendingClustering

Track file groups in pending clustering.

Constructor Detail

HoodieTableFileSystemView

public HoodieTableFileSystemView(HoodieTableMetaClient metaClient,
                                 HoodieTimeline visibleActiveTimeline)

Create a file system view, as of the given timeline.

HoodieTableFileSystemView

public HoodieTableFileSystemView(HoodieTableMetaClient metaClient,
                                 HoodieTimeline visibleActiveTimeline,
                                 boolean enableIncrementalTimelineSync)

Create a file system view, as of the given timeline.

HoodieTableFileSystemView

public HoodieTableFileSystemView(HoodieTableMetaClient metaClient,
                                 HoodieTimeline visibleActiveTimeline,
                                 List<StoragePathInfo> pathInfoList)

Create a file system view, as of the given timeline, with the provided file statuses.

Method Detail

init

public void init(HoodieTableMetaClient metaClient,
                 HoodieTimeline visibleActiveTimeline)

Description copied from class: AbstractTableFileSystemView

Initialize the view.

Overrides:

init in class AbstractTableFileSystemView

init

public void init(HoodieTableMetaClient metaClient,
                 HoodieTimeline visibleActiveTimeline,
                 List<StoragePathInfo> pathInfoList)

Visible for testing

resetViewState

protected void resetViewState()

Description copied from class: AbstractTableFileSystemView

Allows all view metadata in file system view storage to be reset by subclasses.

Specified by:

resetViewState in class AbstractTableFileSystemView

createPartitionToFileGroups

protected Map<String,List<HoodieFileGroup>> createPartitionToFileGroups()

createFileIdToPendingCompactionMap

protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>> createFileIdToPendingCompactionMap(Map<HoodieFileGroupId,Pair<String,CompactionOperation>> fileIdToPendingCompaction)

createFileIdToPendingLogCompactionMap

protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>> createFileIdToPendingLogCompactionMap(Map<HoodieFileGroupId,Pair<String,CompactionOperation>> fileIdToPendingLogCompaction)

createFileIdToBootstrapBaseFileMap

protected Map<HoodieFileGroupId,BootstrapBaseFileMapping> createFileIdToBootstrapBaseFileMap(Map<HoodieFileGroupId,BootstrapBaseFileMapping> fileGroupIdBootstrapBaseFileMap)

createFileIdToReplaceInstantMap

protected Map<HoodieFileGroupId,HoodieInstant> createFileIdToReplaceInstantMap(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)

createFileIdToPendingClusteringMap

protected Map<HoodieFileGroupId,HoodieInstant> createFileIdToPendingClusteringMap(Map<HoodieFileGroupId,HoodieInstant> fileGroupsInClustering)

isPendingCompactionScheduledForFileId

protected boolean isPendingCompactionScheduledForFileId(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Check if there is an outstanding compaction scheduled for this file.

Specified by:

isPendingCompactionScheduledForFileId in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

Returns:

true if there is a pending compaction, false otherwise

resetPendingCompactionOperations

protected void resetPendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

resets the pending compaction operation and overwrite with the new list.

Parameters:

operations - Pending Compaction Operations

addPendingCompactionOperations

protected void addPendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

Add pending compaction operations to store.

Parameters:

operations - Pending compaction operations to be added

removePendingCompactionOperations

protected void removePendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

Remove pending compaction operations from store.

Parameters:

operations - Pending compaction operations to be removed

isPendingLogCompactionScheduledForFileId

protected boolean isPendingLogCompactionScheduledForFileId(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Check if there is an outstanding log compaction scheduled for this file.

Specified by:

isPendingLogCompactionScheduledForFileId in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

Returns:

true if there is a pending log compaction, false otherwise

resetPendingLogCompactionOperations

protected void resetPendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

resets the pending Log compaction operation and overwrite with the new list.

Parameters:

operations - Pending Log Compaction Operations

addPendingLogCompactionOperations

protected void addPendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

Add pending Log compaction operations to store.

Parameters:

operations - Pending Log compaction operations to be added

removePendingLogCompactionOperations

protected void removePendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

Remove pending Log compaction operations from store.

Parameters:

operations - Pending Log compaction operations to be removed

isPendingClusteringScheduledForFileId

protected boolean isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.

Specified by:

isPendingClusteringScheduledForFileId in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

Returns:

true if there is a pending clustering, false otherwise

getPendingClusteringInstant

protected Option<HoodieInstant> getPendingClusteringInstant(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Get pending clustering instant time for specified file group. Return None if file group is not in pending clustering operation.

Specified by:

getPendingClusteringInstant in class AbstractTableFileSystemView

fetchFileGroupsInPendingClustering

protected Stream<Pair<HoodieFileGroupId,HoodieInstant>> fetchFileGroupsInPendingClustering()

Description copied from class: AbstractTableFileSystemView

Fetch all file groups in pending clustering.

Specified by:

fetchFileGroupsInPendingClustering in class AbstractTableFileSystemView

getAllFileGroups

public Stream<HoodieFileGroup> getAllFileGroups()

isBootstrapBaseFilePresentForFileId

protected boolean isBootstrapBaseFilePresentForFileId(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Check if there is an bootstrap base file present for this file.

Specified by:

isBootstrapBaseFilePresentForFileId in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

Returns:

true if there is associated bootstrap base-file, false otherwise

getBootstrapBaseFile

protected Option<BootstrapBaseFileMapping> getBootstrapBaseFile(HoodieFileGroupId fileGroupId)

Description copied from class: AbstractTableFileSystemView

Return pending compaction operation for a file-group.

Specified by:

getBootstrapBaseFile in class AbstractTableFileSystemView

Parameters:

fileGroupId - File-Group Id

getPendingCompactionOperationWithInstant

protected Option<Pair<String,CompactionOperation>> getPendingCompactionOperationWithInstant(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Return pending compaction operation for a file-group.

Specified by:

getPendingCompactionOperationWithInstant in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

getPendingLogCompactionOperationWithInstant

protected Option<Pair<String,CompactionOperation>> getPendingLogCompactionOperationWithInstant(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Return pending Log compaction operation for a file-group.

Specified by:

getPendingLogCompactionOperationWithInstant in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

isPartitionAvailableInStore

protected boolean isPartitionAvailableInStore(String partitionPath)

Description copied from class: AbstractTableFileSystemView

Checks if partition is pre-loaded and available in store.

Parameters:

partitionPath - Partition Path

storePartitionView

protected void storePartitionView(String partitionPath,
                                  List<HoodieFileGroup> fileGroups)

Description copied from class: AbstractTableFileSystemView

Add a complete partition view to store.

Parameters:

partitionPath - Partition Path

fileGroups - File Groups for the partition path

fetchAllStoredFileGroups

public Stream<HoodieFileGroup> fetchAllStoredFileGroups()

Description copied from class: AbstractTableFileSystemView

Fetch all Stored file-groups across all partitions loaded.

Returns:

file-group stream

resetReplacedFileGroups

protected void resetReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)

Description copied from class: AbstractTableFileSystemView

Track instant time for file groups replaced.

Specified by:

resetReplacedFileGroups in class AbstractTableFileSystemView

addReplacedFileGroups

protected void addReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)

Description copied from class: AbstractTableFileSystemView

Track instant time for new file groups replaced.

Specified by:

addReplacedFileGroups in class AbstractTableFileSystemView

removeReplacedFileIdsAtInstants

protected void removeReplacedFileIdsAtInstants(Set<String> instants)

Description copied from class: AbstractTableFileSystemView

Remove file groups that are replaced in any of the specified instants.

Specified by:

removeReplacedFileIdsAtInstants in class AbstractTableFileSystemView

hasReplacedFilesInPartition

protected boolean hasReplacedFilesInPartition(String partitionPath)

Description copied from class: AbstractTableFileSystemView

Returns whether there are replaced files within the given partition.

Specified by:

hasReplacedFilesInPartition in class AbstractTableFileSystemView

getReplaceInstant

protected Option<HoodieInstant> getReplaceInstant(HoodieFileGroupId fileGroupId)

Description copied from class: AbstractTableFileSystemView

Track instant time for file groups replaced.

Specified by:

getReplaceInstant in class AbstractTableFileSystemView

fetchLatestFileSlicesIncludingInflight

public Stream<FileSlice> fetchLatestFileSlicesIncludingInflight(String partitionPath)

Get the latest file slices for a given partition including the inflight ones.

Parameters:

partitionPath - The partition path of interest

Returns:

Stream of latest FileSlice in the partition path.

close

public void close()

Description copied from interface: SyncableFileSystemView

Allow View to release resources and close.

Specified by:

close in interface SyncableFileSystemView

Overrides:

close in class AbstractTableFileSystemView

isClosed

public boolean isClosed()

Description copied from class: AbstractTableFileSystemView

Check if the view is already closed.