org.apache.hudi.common.table.view

Class RemoteHoodieTableFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView

All Implemented Interfaces:

Serializable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

public class RemoteHoodieTableFileSystemView
extends Object
implements SyncableFileSystemView, Serializable

A proxy for table file-system view which translates local View API calls to REST calls to remote timeline service.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Field Summary

Fields

Modifier and Type	Field and Description
static String	ALL_DATA_FILES
static String	ALL_FILEGROUPS_FOR_PARTITION_URL
static String	ALL_REPLACED_FILEGROUPS_BEFORE
static String	ALL_REPLACED_FILEGROUPS_BEFORE_OR_ON
static String	ALL_REPLACED_FILEGROUPS_PARTITION
static String	ALL_SLICES_URL
static String	BASEPATH_PARAM
static String	FILEID_PARAM
static String	INCLUDE_FILES_IN_PENDING_COMPACTION_PARAM
static String	INSTANT_PARAM
static String	INSTANTS_PARAM
static String	LAST_INSTANT
static String	LAST_INSTANT_TS
static String	LAST_INSTANTS
static String	LATEST_ALL_DATA_FILES
static String	LATEST_DATA_FILE_ON_INSTANT_URL
static String	LATEST_DATA_FILES_BEFORE_ON_INSTANT_URL
static String	LATEST_DATA_FILES_RANGE_INSTANT_URL
static String	LATEST_PARTITION_DATA_FILE_URL
static String	LATEST_PARTITION_DATA_FILES_URL
static String	LATEST_PARTITION_SLICE_URL
static String	LATEST_PARTITION_SLICES_URL
static String	LATEST_PARTITION_UNCOMPACTED_SLICES_URL
static String	LATEST_SLICES_BEFORE_ON_INSTANT_URL
static String	LATEST_SLICES_MERGED_BEFORE_ON_INSTANT_URL
static String	LATEST_SLICES_RANGE_INSTANT_URL
static String	MAX_INSTANT_PARAM
static String	PARTITION_PARAM
static String	PENDING_CLUSTERING_FILEGROUPS
static String	PENDING_COMPACTION_OPS
static String	REFRESH_OFF
static String	REFRESH_TABLE
static String	TIMELINE
static String	TIMELINE_HASH

Constructor Summary

Constructors

Constructor and Description
RemoteHoodieTableFileSystemView(HoodieTableMetaClient metaClient, FileSystemViewStorageConfig viewConf)
RemoteHoodieTableFileSystemView(String server, int port, HoodieTableMetaClient metaClient)

Method Summary

Modifier and Type	Method and Description
void	close() Allow View to release resources and close.
Stream<HoodieBaseFile>	getAllBaseFiles(String partitionPath) Stream all the data file versions grouped by FileId for a given partition.
Stream<HoodieFileGroup>	getAllFileGroups(String partitionPath) Stream all the file groups for a given partition.
Stream<FileSlice>	getAllFileSlices(String partitionPath) Stream all the file slices for a given partition, latest or not.
Stream<HoodieFileGroup>	getAllReplacedFileGroups(String partitionPath) Stream all the replaced file groups for given partition.
Option<HoodieBaseFile>	getBaseFileOn(String partitionPath, String instantTime, String fileId) Get the version of data file matching the instant time in the given partition.
Stream<Pair<HoodieFileGroupId,HoodieInstant>>	getFileGroupsInPendingClustering() Filegroups that are in pending clustering.
Option<HoodieInstant>	getLastInstant() Last Known Instant on which the view is built.
Option<HoodieBaseFile>	getLatestBaseFile(String partitionPath, String fileId) Get Latest data file for a partition and file-Id.
Stream<HoodieBaseFile>	getLatestBaseFiles() Stream all the latest data files, in the file system view.
Stream<HoodieBaseFile>	getLatestBaseFiles(String partitionPath) Stream all the latest data files in the given partition.
Stream<HoodieBaseFile>	getLatestBaseFilesBeforeOrOn(String partitionPath, String maxCommitTime) Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.
Stream<HoodieBaseFile>	getLatestBaseFilesInRange(List<String> commitsToReturn) Stream all the latest data files pass.
Option<FileSlice>	getLatestFileSlice(String partitionPath, String fileId) Get Latest File Slice for a given fileId in a given partition.
Stream<FileSlice>	getLatestFileSliceInRange(List<String> commitsToReturn) Stream all the latest file slices, in the given range.
Stream<FileSlice>	getLatestFileSlices(String partitionPath) Stream all the latest file slices in the given partition.
Stream<FileSlice>	getLatestFileSlicesBeforeOrOn(String partitionPath, String maxCommitTime, boolean includeFileSlicesInPendingCompaction) Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.
Stream<FileSlice>	getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime) Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.
Stream<FileSlice>	getLatestUnCompactedFileSlices(String partitionPath) Stream all the latest uncompacted file slices in the given partition.
Stream<Pair<String,CompactionOperation>>	getPendingCompactionOperations() Return Pending Compaction Operations.
Stream<HoodieFileGroup>	getReplacedFileGroupsBefore(String maxCommitTime, String partitionPath) Stream all the replaced file groups before maxCommitTime for given partition.
Stream<HoodieFileGroup>	getReplacedFileGroupsBeforeOrOn(String maxCommitTime, String partitionPath) Stream all the replaced file groups before or on maxCommitTime for given partition.
HoodieTimeline	getTimeline() Timeline corresponding to the view.
boolean	refresh()
void	reset() Reset View so that they can be refreshed.
void	sync() Read the latest timeline and refresh the file-system view to match the current state of the file-system.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LATEST_PARTITION_SLICES_URL

public static final String LATEST_PARTITION_SLICES_URL

LATEST_PARTITION_SLICE_URL

public static final String LATEST_PARTITION_SLICE_URL

LATEST_PARTITION_UNCOMPACTED_SLICES_URL

public static final String LATEST_PARTITION_UNCOMPACTED_SLICES_URL

ALL_SLICES_URL

public static final String ALL_SLICES_URL

LATEST_SLICES_MERGED_BEFORE_ON_INSTANT_URL

public static final String LATEST_SLICES_MERGED_BEFORE_ON_INSTANT_URL

LATEST_SLICES_RANGE_INSTANT_URL

public static final String LATEST_SLICES_RANGE_INSTANT_URL

LATEST_SLICES_BEFORE_ON_INSTANT_URL

public static final String LATEST_SLICES_BEFORE_ON_INSTANT_URL

PENDING_COMPACTION_OPS

public static final String PENDING_COMPACTION_OPS

LATEST_PARTITION_DATA_FILES_URL

public static final String LATEST_PARTITION_DATA_FILES_URL

LATEST_PARTITION_DATA_FILE_URL

public static final String LATEST_PARTITION_DATA_FILE_URL

ALL_DATA_FILES

public static final String ALL_DATA_FILES

LATEST_ALL_DATA_FILES

public static final String LATEST_ALL_DATA_FILES

LATEST_DATA_FILE_ON_INSTANT_URL

public static final String LATEST_DATA_FILE_ON_INSTANT_URL

LATEST_DATA_FILES_RANGE_INSTANT_URL

public static final String LATEST_DATA_FILES_RANGE_INSTANT_URL

LATEST_DATA_FILES_BEFORE_ON_INSTANT_URL

public static final String LATEST_DATA_FILES_BEFORE_ON_INSTANT_URL

ALL_FILEGROUPS_FOR_PARTITION_URL

public static final String ALL_FILEGROUPS_FOR_PARTITION_URL

ALL_REPLACED_FILEGROUPS_BEFORE_OR_ON

public static final String ALL_REPLACED_FILEGROUPS_BEFORE_OR_ON

ALL_REPLACED_FILEGROUPS_BEFORE

public static final String ALL_REPLACED_FILEGROUPS_BEFORE

ALL_REPLACED_FILEGROUPS_PARTITION

public static final String ALL_REPLACED_FILEGROUPS_PARTITION

PENDING_CLUSTERING_FILEGROUPS

public static final String PENDING_CLUSTERING_FILEGROUPS

LAST_INSTANT

public static final String LAST_INSTANT

LAST_INSTANTS

public static final String LAST_INSTANTS

TIMELINE

public static final String TIMELINE

REFRESH_TABLE

public static final String REFRESH_TABLE

PARTITION_PARAM

public static final String PARTITION_PARAM

See Also:

Constant Field Values

BASEPATH_PARAM

public static final String BASEPATH_PARAM

See Also:

Constant Field Values

INSTANT_PARAM

public static final String INSTANT_PARAM

See Also:

Constant Field Values

MAX_INSTANT_PARAM

public static final String MAX_INSTANT_PARAM

See Also:

Constant Field Values

INSTANTS_PARAM

public static final String INSTANTS_PARAM

See Also:

Constant Field Values

FILEID_PARAM

public static final String FILEID_PARAM

See Also:

Constant Field Values

LAST_INSTANT_TS

public static final String LAST_INSTANT_TS

See Also:

Constant Field Values

TIMELINE_HASH

public static final String TIMELINE_HASH

See Also:

Constant Field Values

REFRESH_OFF

public static final String REFRESH_OFF

See Also:

Constant Field Values

INCLUDE_FILES_IN_PENDING_COMPACTION_PARAM

public static final String INCLUDE_FILES_IN_PENDING_COMPACTION_PARAM

See Also:

Constant Field Values

Constructor Detail

RemoteHoodieTableFileSystemView

public RemoteHoodieTableFileSystemView(String server,
                                       int port,
                                       HoodieTableMetaClient metaClient)

RemoteHoodieTableFileSystemView

public RemoteHoodieTableFileSystemView(HoodieTableMetaClient metaClient,
                                       FileSystemViewStorageConfig viewConf)

Method Detail

getLatestBaseFiles

public Stream<HoodieBaseFile> getLatestBaseFiles(String partitionPath)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files in the given partition.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFiles

public Stream<HoodieBaseFile> getLatestBaseFiles()

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files, in the file system view.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFilesBeforeOrOn

public Stream<HoodieBaseFile> getLatestBaseFilesBeforeOrOn(String partitionPath,
                                                           String maxCommitTime)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestBaseFilesBeforeOrOn in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getBaseFileOn

public Option<HoodieBaseFile> getBaseFileOn(String partitionPath,
                                            String instantTime,
                                            String fileId)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Get the version of data file matching the instant time in the given partition.

Specified by:

getBaseFileOn in interface TableFileSystemView.BaseFileOnlyView

getLatestBaseFilesInRange

public Stream<HoodieBaseFile> getLatestBaseFilesInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files pass.

Specified by:

getLatestBaseFilesInRange in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getAllBaseFiles

public Stream<HoodieBaseFile> getAllBaseFiles(String partitionPath)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Stream all the data file versions grouped by FileId for a given partition.

Specified by:

getAllBaseFiles in interface TableFileSystemView.BaseFileOnlyView

getLatestFileSlices

public Stream<FileSlice> getLatestFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices in the given partition.

Specified by:

getLatestFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlice

public Option<FileSlice> getLatestFileSlice(String partitionPath,
                                            String fileId)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Get Latest File Slice for a given fileId in a given partition.

Specified by:

getLatestFileSlice in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestUnCompactedFileSlices

public Stream<FileSlice> getLatestUnCompactedFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest uncompacted file slices in the given partition.

Specified by:

getLatestUnCompactedFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlicesBeforeOrOn

public Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath,
                                                       String maxCommitTime,
                                                       boolean includeFileSlicesInPendingCompaction)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - Partition path

maxCommitTime - Max Instant Time

includeFileSlicesInPendingCompaction - include file-slices that are in pending compaction

getLatestMergedFileSlicesBeforeOrOn

public Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath,
                                                             String maxInstantTime)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.

Specified by:

getLatestMergedFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - Partition Path

maxInstantTime - Max Instant Time

Returns:

getLatestFileSliceInRange

public Stream<FileSlice> getLatestFileSliceInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices, in the given range.

Specified by:

getLatestFileSliceInRange in interface TableFileSystemView.SliceViewWithLatestSlice

getAllFileSlices

public Stream<FileSlice> getAllFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceView

Stream all the file slices for a given partition, latest or not.

Specified by:

getAllFileSlices in interface TableFileSystemView.SliceView

getAllFileGroups

public Stream<HoodieFileGroup> getAllFileGroups(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the file groups for a given partition.

Specified by:

getAllFileGroups in interface TableFileSystemView

getReplacedFileGroupsBeforeOrOn

public Stream<HoodieFileGroup> getReplacedFileGroupsBeforeOrOn(String maxCommitTime,
                                                               String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before or on maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBeforeOrOn in interface TableFileSystemView

getReplacedFileGroupsBefore

public Stream<HoodieFileGroup> getReplacedFileGroupsBefore(String maxCommitTime,
                                                           String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBefore in interface TableFileSystemView

getAllReplacedFileGroups

public Stream<HoodieFileGroup> getAllReplacedFileGroups(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups for given partition.

Specified by:

getAllReplacedFileGroups in interface TableFileSystemView

refresh

public boolean refresh()

getPendingCompactionOperations

public Stream<Pair<String,CompactionOperation>> getPendingCompactionOperations()

Description copied from interface: TableFileSystemView

Return Pending Compaction Operations.

Specified by:

getPendingCompactionOperations in interface TableFileSystemView

Returns:

Pair>

getFileGroupsInPendingClustering

public Stream<Pair<HoodieFileGroupId,HoodieInstant>> getFileGroupsInPendingClustering()

Description copied from interface: TableFileSystemView

Filegroups that are in pending clustering.

Specified by:

getFileGroupsInPendingClustering in interface TableFileSystemView

close

public void close()

Description copied from interface: SyncableFileSystemView

Allow View to release resources and close.

Specified by:

close in interface SyncableFileSystemView

reset

public void reset()

Description copied from interface: SyncableFileSystemView

Reset View so that they can be refreshed.

Specified by:

reset in interface SyncableFileSystemView

getLastInstant

public Option<HoodieInstant> getLastInstant()

Description copied from interface: TableFileSystemView

Last Known Instant on which the view is built.

Specified by:

getLastInstant in interface TableFileSystemView

getTimeline

public HoodieTimeline getTimeline()

Description copied from interface: TableFileSystemView

Timeline corresponding to the view.

Specified by:

getTimeline in interface TableFileSystemView

sync

public void sync()

Description copied from interface: SyncableFileSystemView

Read the latest timeline and refresh the file-system view to match the current state of the file-system. The refresh can either be done incrementally (from reading file-slices in metadata files) or from scratch by reseting view storage.

Specified by:

sync in interface SyncableFileSystemView

getLatestBaseFile

public Option<HoodieBaseFile> getLatestBaseFile(String partitionPath,
                                                String fileId)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Get Latest data file for a partition and file-Id.

Specified by:

getLatestBaseFile in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice