org.apache.hudi.common.table.view

Interface TableFileSystemView.SliceViewWithLatestSlice

All Known Subinterfaces:

SyncableFileSystemView, TableFileSystemView.SliceView

All Known Implementing Classes:

AbstractTableFileSystemView, HoodieMetadataFileSystemView, HoodieTableFileSystemView, IncrementalTimelineSyncFileSystemView, PriorityBasedFileSystemView, RemoteHoodieTableFileSystemView, RocksDbBasedFileSystemView, SpillableMapBasedFileSystemView

Enclosing interface:

TableFileSystemView

public static interface TableFileSystemView.SliceViewWithLatestSlice

Methods to only access latest version of file-slice for the instant(s) passed.

Method Summary

Modifier and Type	Method and Description
Map<String,Stream<FileSlice>>	getAllLatestFileSlicesBeforeOrOn(String maxCommitTime) Stream all latest file slices with precondition that commitTime(file) before maxCommitTime.
Option<FileSlice>	getLatestFileSlice(String partitionPath, String fileId) Get Latest File Slice for a given fileId in a given partition.
Stream<FileSlice>	getLatestFileSliceInRange(List<String> commitsToReturn) Stream all the latest file slices, in the given range.
Stream<FileSlice>	getLatestFileSlices(String partitionPath) Stream all the latest file slices in the given partition.
Stream<FileSlice>	getLatestFileSlicesBeforeOrOn(String partitionPath, String maxCommitTime, boolean includeFileSlicesInPendingCompaction) Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.
Stream<FileSlice>	getLatestFileSlicesStateless(String partitionPath) Stream all the latest file slices in the given partition without caching the file group mappings.
Stream<FileSlice>	getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime) Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.
Stream<FileSlice>	getLatestUnCompactedFileSlices(String partitionPath) Stream all the latest uncompacted file slices in the given partition.

Method Detail

getLatestFileSlices

Stream<FileSlice> getLatestFileSlices(String partitionPath)

Stream all the latest file slices in the given partition.

getLatestFileSlicesStateless

Stream<FileSlice> getLatestFileSlicesStateless(String partitionPath)

Stream all the latest file slices in the given partition without caching the file group mappings.

This is useful for some table services such as compaction and clustering, these services may search around the files to clean within some ancient data partitions, if there triggers a full table service for enormous number of partitions, the cache could cause a huge memory pressure to the timeline server which induces an OOM exception.

The caching of these file groups does not benefit to writers most often because the writers write to recent data partitions usually.

getLatestFileSlice

Option<FileSlice> getLatestFileSlice(String partitionPath,
                                     String fileId)

Get Latest File Slice for a given fileId in a given partition.

getLatestUnCompactedFileSlices

Stream<FileSlice> getLatestUnCompactedFileSlices(String partitionPath)

Stream all the latest uncompacted file slices in the given partition.

getLatestFileSlicesBeforeOrOn

Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath,
                                                String maxCommitTime,
                                                boolean includeFileSlicesInPendingCompaction)

Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.

Parameters:

partitionPath - Partition path

maxCommitTime - Max Instant Time

includeFileSlicesInPendingCompaction - include file-slices that are in pending compaction

getAllLatestFileSlicesBeforeOrOn

Map<String,Stream<FileSlice>> getAllLatestFileSlicesBeforeOrOn(String maxCommitTime)

Stream all latest file slices with precondition that commitTime(file) before maxCommitTime.

Parameters:

maxCommitTime - Max Instant Time

Returns:

A Map of partition path to the latest file slices before maxCommitTime.

getLatestMergedFileSlicesBeforeOrOn

Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath,
                                                      String maxInstantTime)

Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.

Parameters:

partitionPath - Partition Path

maxInstantTime - Max Instant Time

Returns:

getLatestFileSliceInRange

Stream<FileSlice> getLatestFileSliceInRange(List<String> commitsToReturn)

Stream all the latest file slices, in the given range.