org.apache.hudi.common.table.view

Class FileSystemViewManager

java.lang.Object

org.apache.hudi.common.table.view.FileSystemViewManager

public class FileSystemViewManager
extends Object

A container that can potentially hold one or more table's file-system views. There is one view for each table. This is a view built against a timeline containing completed actions. In an embedded timeline-server mode, this typically holds only one table's view. In a stand-alone server mode, this can hold more than one table's views.

FileSystemView can be stored "locally" using the following storage mechanisms: a. In Memory b. Spillable Map c. RocksDB.

But there can be cases where the file-system view is managed remoted. For example : Embedded Timeline Server). In this case, the clients will configure a remote filesystem view client (RemoteHoodieTableFileSystemView) for the table which can connect to the remote file system view and fetch views. There are 2 modes here : REMOTE_FIRST and REMOTE_ONLY. REMOTE_FIRST : The file-system view implementation on client side will act as a remote proxy. In case, if there is problem (or exceptions) querying remote file-system view, a backup local file-system view(using either one of in-memory, spillable, rocksDB) is used to server file-system view queries. REMOTE_ONLY : In this case, there is no backup local file-system view. If there is problem (or exceptions) querying remote file-system view, then the exceptions are percolated back to client.

FileSystemViewManager is designed to encapsulate the file-system view storage from clients using the file-system view. FileSystemViewManager uses a factory to construct specific implementation of file-system view and passes it to clients for querying.

Method Summary

Modifier and Type	Method and Description
void	clearFileSystemView(String basePath) Drops reference to File-System Views.
void	close() Closes all views opened.
static HoodieTableFileSystemView	createInMemoryFileSystemView(HoodieEngineContext engineContext, HoodieTableMetaClient metaClient, HoodieMetadataConfig metadataConfig)
static HoodieTableFileSystemView	createInMemoryFileSystemViewWithTimeline(HoodieEngineContext engineContext, HoodieTableMetaClient metaClient, HoodieMetadataConfig metadataConfig, HoodieTimeline timeline)
static FileSystemViewManager	createViewManager(HoodieEngineContext context, FileSystemViewStorageConfig config, HoodieCommonConfig commonConfig)
static FileSystemViewManager	createViewManager(HoodieEngineContext context, FileSystemViewStorageConfig config, HoodieCommonConfig commonConfig, SerializableFunctionUnchecked<HoodieTableMetaClient,HoodieTableMetadata> metadataCreator) Main Factory method for building file-system views.
static FileSystemViewManager	createViewManagerWithTableMetadata(HoodieEngineContext context, HoodieMetadataConfig metadataConfig, FileSystemViewStorageConfig config, HoodieCommonConfig commonConfig)
SyncableFileSystemView	getFileSystemView(HoodieTableMetaClient metaClient) Main API to get the file-system view for the base-path.
SyncableFileSystemView	getFileSystemView(String basePath) Main API to get the file-system view for the base-path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

clearFileSystemView

public void clearFileSystemView(String basePath)

Drops reference to File-System Views. Future calls to view results in creating a new view

Parameters:

basePath - Hoodie table base path

getFileSystemView

public SyncableFileSystemView getFileSystemView(String basePath)

Main API to get the file-system view for the base-path.

Parameters:

basePath - Hoodie table base path

Returns:

SyncableFileSystemView

getFileSystemView

public SyncableFileSystemView getFileSystemView(HoodieTableMetaClient metaClient)

Main API to get the file-system view for the base-path.

Parameters:

metaClient - HoodieTableMetaClient

Returns:

SyncableFileSystemView

close

public void close()

Closes all views opened.

createInMemoryFileSystemView

public static HoodieTableFileSystemView createInMemoryFileSystemView(HoodieEngineContext engineContext,
                                                                     HoodieTableMetaClient metaClient,
                                                                     HoodieMetadataConfig metadataConfig)

createInMemoryFileSystemViewWithTimeline

public static HoodieTableFileSystemView createInMemoryFileSystemViewWithTimeline(HoodieEngineContext engineContext,
                                                                                 HoodieTableMetaClient metaClient,
                                                                                 HoodieMetadataConfig metadataConfig,
                                                                                 HoodieTimeline timeline)

createViewManagerWithTableMetadata

public static FileSystemViewManager createViewManagerWithTableMetadata(HoodieEngineContext context,
                                                                       HoodieMetadataConfig metadataConfig,
                                                                       FileSystemViewStorageConfig config,
                                                                       HoodieCommonConfig commonConfig)

createViewManager

public static FileSystemViewManager createViewManager(HoodieEngineContext context,
                                                      FileSystemViewStorageConfig config,
                                                      HoodieCommonConfig commonConfig)

createViewManager

public static FileSystemViewManager createViewManager(HoodieEngineContext context,
                                                      FileSystemViewStorageConfig config,
                                                      HoodieCommonConfig commonConfig,
                                                      SerializableFunctionUnchecked<HoodieTableMetaClient,HoodieTableMetadata> metadataCreator)

Main Factory method for building file-system views.