org.apache.hudi.common.table.view

Class PriorityBasedFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.PriorityBasedFileSystemView

All Implemented Interfaces:

Serializable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

public class PriorityBasedFileSystemView
extends Object
implements SyncableFileSystemView, Serializable

A file system view which proxies request to a preferred File System View implementation. In case of error, flip all subsequent calls to a backup file-system view implementation.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Constructor Summary

Constructors

Constructor and Description
PriorityBasedFileSystemView(SyncableFileSystemView preferredView, SyncableFileSystemView secondaryView)

Method Summary

Modifier and Type	Method and Description
void	close() Allow View to release resources and close.
Stream<HoodieBaseFile>	getAllBaseFiles(String partitionPath) Stream all the data file versions grouped by FileId for a given partition.
Stream<HoodieFileGroup>	getAllFileGroups(String partitionPath) Stream all the file groups for a given partition.
Stream<FileSlice>	getAllFileSlices(String partitionPath) Stream all the file slices for a given partition, latest or not.
Stream<HoodieFileGroup>	getAllReplacedFileGroups(String partitionPath) Stream all the replaced file groups for given partition.
Option<HoodieBaseFile>	getBaseFileOn(String partitionPath, String instantTime, String fileId) Get the version of data file matching the instant time in the given partition.
Stream<Pair<HoodieFileGroupId,HoodieInstant>>	getFileGroupsInPendingClustering() Filegroups that are in pending clustering.
Option<HoodieInstant>	getLastInstant() Last Known Instant on which the view is built.
Option<HoodieBaseFile>	getLatestBaseFile(String partitionPath, String fileId) Get Latest data file for a partition and file-Id.
Stream<HoodieBaseFile>	getLatestBaseFiles() Stream all the latest data files, in the file system view.
Stream<HoodieBaseFile>	getLatestBaseFiles(String partitionPath) Stream all the latest data files in the given partition.
Stream<HoodieBaseFile>	getLatestBaseFilesBeforeOrOn(String partitionPath, String maxCommitTime) Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.
Stream<HoodieBaseFile>	getLatestBaseFilesInRange(List<String> commitsToReturn) Stream all the latest data files pass.
Option<FileSlice>	getLatestFileSlice(String partitionPath, String fileId) Get Latest File Slice for a given fileId in a given partition.
Stream<FileSlice>	getLatestFileSliceInRange(List<String> commitsToReturn) Stream all the latest file slices, in the given range.
Stream<FileSlice>	getLatestFileSlices(String partitionPath) Stream all the latest file slices in the given partition.
Stream<FileSlice>	getLatestFileSlicesBeforeOrOn(String partitionPath, String maxCommitTime, boolean includeFileSlicesInPendingCompaction) Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.
Stream<FileSlice>	getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime) Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.
Stream<FileSlice>	getLatestUnCompactedFileSlices(String partitionPath) Stream all the latest uncompacted file slices in the given partition.
Stream<Pair<String,CompactionOperation>>	getPendingCompactionOperations() Return Pending Compaction Operations.
SyncableFileSystemView	getPreferredView()
Stream<HoodieFileGroup>	getReplacedFileGroupsBefore(String maxCommitTime, String partitionPath) Stream all the replaced file groups before maxCommitTime for given partition.
Stream<HoodieFileGroup>	getReplacedFileGroupsBeforeOrOn(String maxCommitTime, String partitionPath) Stream all the replaced file groups before or on maxCommitTime for given partition.
SyncableFileSystemView	getSecondaryView()
HoodieTimeline	getTimeline() Timeline corresponding to the view.
void	reset() Reset View so that they can be refreshed.
void	sync() Read the latest timeline and refresh the file-system view to match the current state of the file-system.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PriorityBasedFileSystemView

public PriorityBasedFileSystemView(SyncableFileSystemView preferredView,
                                   SyncableFileSystemView secondaryView)

Method Detail

getLatestBaseFiles

public Stream<HoodieBaseFile> getLatestBaseFiles(String partitionPath)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files in the given partition.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFiles

public Stream<HoodieBaseFile> getLatestBaseFiles()

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files, in the file system view.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFilesBeforeOrOn

public Stream<HoodieBaseFile> getLatestBaseFilesBeforeOrOn(String partitionPath,
                                                           String maxCommitTime)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestBaseFilesBeforeOrOn in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFile

public Option<HoodieBaseFile> getLatestBaseFile(String partitionPath,
                                                String fileId)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Get Latest data file for a partition and file-Id.

Specified by:

getLatestBaseFile in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getBaseFileOn

public Option<HoodieBaseFile> getBaseFileOn(String partitionPath,
                                            String instantTime,
                                            String fileId)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Get the version of data file matching the instant time in the given partition.

Specified by:

getBaseFileOn in interface TableFileSystemView.BaseFileOnlyView

getLatestBaseFilesInRange

public Stream<HoodieBaseFile> getLatestBaseFilesInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files pass.

Specified by:

getLatestBaseFilesInRange in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getAllBaseFiles

public Stream<HoodieBaseFile> getAllBaseFiles(String partitionPath)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Stream all the data file versions grouped by FileId for a given partition.

Specified by:

getAllBaseFiles in interface TableFileSystemView.BaseFileOnlyView

getLatestFileSlices

public Stream<FileSlice> getLatestFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices in the given partition.

Specified by:

getLatestFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestUnCompactedFileSlices

public Stream<FileSlice> getLatestUnCompactedFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest uncompacted file slices in the given partition.

Specified by:

getLatestUnCompactedFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlicesBeforeOrOn

public Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath,
                                                       String maxCommitTime,
                                                       boolean includeFileSlicesInPendingCompaction)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - Partition path

maxCommitTime - Max Instant Time

includeFileSlicesInPendingCompaction - include file-slices that are in pending compaction

getLatestMergedFileSlicesBeforeOrOn

public Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath,
                                                             String maxInstantTime)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.

Specified by:

getLatestMergedFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - Partition Path

maxInstantTime - Max Instant Time

Returns:

getLatestFileSliceInRange

public Stream<FileSlice> getLatestFileSliceInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices, in the given range.

Specified by:

getLatestFileSliceInRange in interface TableFileSystemView.SliceViewWithLatestSlice

getAllFileSlices

public Stream<FileSlice> getAllFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceView

Stream all the file slices for a given partition, latest or not.

Specified by:

getAllFileSlices in interface TableFileSystemView.SliceView

getAllFileGroups

public Stream<HoodieFileGroup> getAllFileGroups(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the file groups for a given partition.

Specified by:

getAllFileGroups in interface TableFileSystemView

getReplacedFileGroupsBeforeOrOn

public Stream<HoodieFileGroup> getReplacedFileGroupsBeforeOrOn(String maxCommitTime,
                                                               String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before or on maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBeforeOrOn in interface TableFileSystemView

getReplacedFileGroupsBefore

public Stream<HoodieFileGroup> getReplacedFileGroupsBefore(String maxCommitTime,
                                                           String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBefore in interface TableFileSystemView

getAllReplacedFileGroups

public Stream<HoodieFileGroup> getAllReplacedFileGroups(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups for given partition.

Specified by:

getAllReplacedFileGroups in interface TableFileSystemView

getPendingCompactionOperations

public Stream<Pair<String,CompactionOperation>> getPendingCompactionOperations()

Description copied from interface: TableFileSystemView

Return Pending Compaction Operations.

Specified by:

getPendingCompactionOperations in interface TableFileSystemView

Returns:

Pair>

getFileGroupsInPendingClustering

public Stream<Pair<HoodieFileGroupId,HoodieInstant>> getFileGroupsInPendingClustering()

Description copied from interface: TableFileSystemView

Filegroups that are in pending clustering.

Specified by:

getFileGroupsInPendingClustering in interface TableFileSystemView

close

public void close()

Description copied from interface: SyncableFileSystemView

Allow View to release resources and close.

Specified by:

close in interface SyncableFileSystemView

reset

public void reset()

Description copied from interface: SyncableFileSystemView

Reset View so that they can be refreshed.

Specified by:

reset in interface SyncableFileSystemView

getLastInstant

public Option<HoodieInstant> getLastInstant()

Description copied from interface: TableFileSystemView

Last Known Instant on which the view is built.

Specified by:

getLastInstant in interface TableFileSystemView

getTimeline

public HoodieTimeline getTimeline()

Description copied from interface: TableFileSystemView

Timeline corresponding to the view.

Specified by:

getTimeline in interface TableFileSystemView

sync

public void sync()

Description copied from interface: SyncableFileSystemView

Read the latest timeline and refresh the file-system view to match the current state of the file-system. The refresh can either be done incrementally (from reading file-slices in metadata files) or from scratch by reseting view storage.

Specified by:

sync in interface SyncableFileSystemView

getLatestFileSlice

public Option<FileSlice> getLatestFileSlice(String partitionPath,
                                            String fileId)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Get Latest File Slice for a given fileId in a given partition.

Specified by:

getLatestFileSlice in interface TableFileSystemView.SliceViewWithLatestSlice

getPreferredView

public SyncableFileSystemView getPreferredView()

getSecondaryView

public SyncableFileSystemView getSecondaryView()