org.apache.hudi.common.table.view

Class SpillableMapBasedFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.AbstractTableFileSystemView

org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView

org.apache.hudi.common.table.view.HoodieTableFileSystemView

org.apache.hudi.common.table.view.SpillableMapBasedFileSystemView

All Implemented Interfaces:

Serializable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

public class SpillableMapBasedFileSystemView
extends HoodieTableFileSystemView

Table FileSystemView implementation where view is stored in spillable disk using fixed memory.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Field Summary

Fields inherited from class org.apache.hudi.common.table.view.HoodieTableFileSystemView

fgIdToBootstrapBaseFile, fgIdToPendingClustering, fgIdToPendingCompaction, fgIdToReplaceInstants, partitionToFileGroupsMap

Fields inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

metaClient

Constructor Summary

Constructors

Constructor and Description
SpillableMapBasedFileSystemView(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline, org.apache.hadoop.fs.FileStatus[] fileStatuses, FileSystemViewStorageConfig config, HoodieCommonConfig commonConfig)
SpillableMapBasedFileSystemView(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline, FileSystemViewStorageConfig config, HoodieCommonConfig commonConfig)

Method Summary

Modifier and Type	Method and Description
protected Map<HoodieFileGroupId,BootstrapBaseFileMapping>	createFileIdToBootstrapBaseFileMap(Map<HoodieFileGroupId,BootstrapBaseFileMapping> fileGroupIdBootstrapBaseFileMap)
protected Map<HoodieFileGroupId,HoodieInstant>	createFileIdToPendingClusteringMap(Map<HoodieFileGroupId,HoodieInstant> fileGroupsInClustering)
protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>>	createFileIdToPendingCompactionMap(Map<HoodieFileGroupId,Pair<String,CompactionOperation>> fgIdToPendingCompaction)
protected Map<HoodieFileGroupId,HoodieInstant>	createFileIdToReplaceInstantMap(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)
protected Map<String,List<HoodieFileGroup>>	createPartitionToFileGroups()
Stream<HoodieFileGroup>	fetchAllStoredFileGroups() Fetch all Stored file-groups across all partitions loaded.
Stream<HoodieFileGroup>	getAllFileGroups()
protected void	removeReplacedFileIdsAtInstants(Set<String> instants) Remove file groups that are replaced in any of the specified instants.

Methods inherited from class org.apache.hudi.common.table.view.HoodieTableFileSystemView

addPendingCompactionOperations, addReplacedFileGroups, close, fetchFileGroupsInPendingClustering, fetchLatestFileSlicesIncludingInflight, getBootstrapBaseFile, getPendingClusteringInstant, getPendingCompactionOperationWithInstant, getReplaceInstant, init, init, isBootstrapBaseFilePresentForFileId, isClosed, isPartitionAvailableInStore, isPendingClusteringScheduledForFileId, isPendingCompactionScheduledForFileId, removePendingCompactionOperations, resetPendingCompactionOperations, resetReplacedFileGroups, resetViewState, storePartitionView

Methods inherited from class org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView

applyDeltaFileSlicesToPartitionView, getTimeline, refreshTimeline, runSync

Methods inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addFilesToView, buildFileGroups, buildFileGroups, fetchLatestBaseFile, fetchLatestBaseFiles, fetchLatestFileSlice, filterBaseFileAfterPendingCompaction, getAllBaseFiles, getAllFileGroups, getAllFileSlices, getAllReplacedFileGroups, getBaseFileOn, getFileGroupsInPendingClustering, getLastInstant, getLatestBaseFile, getLatestBaseFile, getLatestBaseFiles, getLatestBaseFiles, getLatestBaseFilesBeforeOrOn, getLatestBaseFilesInRange, getLatestFileSlice, getLatestFileSliceInRange, getLatestFileSlices, getLatestFileSlicesBeforeOrOn, getLatestMergedFileSlicesBeforeOrOn, getLatestUnCompactedFileSlices, getPartitionPaths, getPendingCompactionOperations, getReplacedFileGroupsBefore, getReplacedFileGroupsBeforeOrOn, getVisibleCommitsAndCompactionTimeline, isBaseFileDueToPendingClustering, isBaseFileDueToPendingCompaction, isFileSliceAfterPendingCompaction, listPartition, reset, sync

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SpillableMapBasedFileSystemView

public SpillableMapBasedFileSystemView(HoodieTableMetaClient metaClient,
                                       HoodieTimeline visibleActiveTimeline,
                                       FileSystemViewStorageConfig config,
                                       HoodieCommonConfig commonConfig)

SpillableMapBasedFileSystemView

public SpillableMapBasedFileSystemView(HoodieTableMetaClient metaClient,
                                       HoodieTimeline visibleActiveTimeline,
                                       org.apache.hadoop.fs.FileStatus[] fileStatuses,
                                       FileSystemViewStorageConfig config,
                                       HoodieCommonConfig commonConfig)

Method Detail

createPartitionToFileGroups

protected Map<String,List<HoodieFileGroup>> createPartitionToFileGroups()

Overrides:

createPartitionToFileGroups in class HoodieTableFileSystemView

createFileIdToPendingCompactionMap

protected Map<HoodieFileGroupId,Pair<String,CompactionOperation>> createFileIdToPendingCompactionMap(Map<HoodieFileGroupId,Pair<String,CompactionOperation>> fgIdToPendingCompaction)

Overrides:

createFileIdToPendingCompactionMap in class HoodieTableFileSystemView

createFileIdToBootstrapBaseFileMap

protected Map<HoodieFileGroupId,BootstrapBaseFileMapping> createFileIdToBootstrapBaseFileMap(Map<HoodieFileGroupId,BootstrapBaseFileMapping> fileGroupIdBootstrapBaseFileMap)

Overrides:

createFileIdToBootstrapBaseFileMap in class HoodieTableFileSystemView

createFileIdToReplaceInstantMap

protected Map<HoodieFileGroupId,HoodieInstant> createFileIdToReplaceInstantMap(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)

Overrides:

createFileIdToReplaceInstantMap in class HoodieTableFileSystemView

createFileIdToPendingClusteringMap

protected Map<HoodieFileGroupId,HoodieInstant> createFileIdToPendingClusteringMap(Map<HoodieFileGroupId,HoodieInstant> fileGroupsInClustering)

Overrides:

createFileIdToPendingClusteringMap in class HoodieTableFileSystemView

getAllFileGroups

public Stream<HoodieFileGroup> getAllFileGroups()

Overrides:

getAllFileGroups in class HoodieTableFileSystemView

fetchAllStoredFileGroups

public Stream<HoodieFileGroup> fetchAllStoredFileGroups()

Description copied from class: AbstractTableFileSystemView

Fetch all Stored file-groups across all partitions loaded.

Overrides:

fetchAllStoredFileGroups in class HoodieTableFileSystemView

Returns:

file-group stream

removeReplacedFileIdsAtInstants

protected void removeReplacedFileIdsAtInstants(Set<String> instants)

Description copied from class: AbstractTableFileSystemView

Remove file groups that are replaced in any of the specified instants.

Overrides:

removeReplacedFileIdsAtInstants in class HoodieTableFileSystemView