org.apache.hudi.timeline.service.handlers

Class MarkerHandler

java.lang.Object

org.apache.hudi.timeline.service.handlers.Handler

org.apache.hudi.timeline.service.handlers.MarkerHandler

public class MarkerHandler
extends Handler

REST Handler servicing marker requests. The marker creation requests are handled asynchronous, while other types of requests are handled synchronous. Marker creation requests are batch processed periodically by a thread. Each batch processing thread adds new markers to a marker file. Given that marker file operation can take time, multiple concurrent threads can run at the same, while they operate on different marker files storing mutually exclusive marker entries. At any given time, a marker file is touched by at most one thread to guarantee consistency. Below is an example of running batch processing threads. |-----| batch interval Worker Thread 1 |-------------------------->| writing to MARKERS0 Worker Thread 2 |-------------------------->| writing to MARKERS1 Worker Thread 3 |-------------------------->| writing to MARKERS2

Field Summary

Fields inherited from class org.apache.hudi.timeline.service.handlers.Handler

conf, storage, timelineServiceConfig, viewManager

Constructor Summary

Constructors

Constructor and Description
MarkerHandler(StorageConfiguration<?> conf, TimelineService.Config timelineServiceConfig, HoodieEngineContext hoodieEngineContext, HoodieStorage storage, FileSystemViewManager viewManager, org.apache.hudi.common.metrics.Registry metricsRegistry)

Method Summary

Modifier and Type	Method and Description
CompletableFuture<String>	createMarker(io.javalin.http.Context context, String markerDir, String markerName, String basePath) Generates a future for an async marker creation request The future is added to the marker creation future list and waits for the next batch processing of marker creation requests.
Boolean	deleteMarkers(String markerDir) Deletes markers in the directory.
boolean	doesMarkerDirExist(String markerDir)
Set<String>	getAllMarkers(String markerDir)
Set<String>	getAppendMarkers(String markerDir)
Set<String>	getCreateAndMergeMarkers(String markerDir)
Set<String>	getPendingMarkersToProcess(String markerDir)
void	stop() Stops the dispatching of marker creation requests.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MarkerHandler

public MarkerHandler(StorageConfiguration<?> conf,
                     TimelineService.Config timelineServiceConfig,
                     HoodieEngineContext hoodieEngineContext,
                     HoodieStorage storage,
                     FileSystemViewManager viewManager,
                     org.apache.hudi.common.metrics.Registry metricsRegistry)
              throws IOException

Throws:

IOException

Method Detail

stop

public void stop()

Stops the dispatching of marker creation requests.

getAllMarkers

public Set<String> getAllMarkers(String markerDir)

Parameters:

markerDir - marker directory path

Returns:

all marker paths in the marker directory

getPendingMarkersToProcess

public Set<String> getPendingMarkersToProcess(String markerDir)

Parameters:

markerDir - marker directory path.

Returns:

Pending markers from the requests to process.

getCreateAndMergeMarkers

public Set<String> getCreateAndMergeMarkers(String markerDir)

Parameters:

markerDir - marker directory path

Returns:

all marker paths of write IO type "CREATE" and "MERGE"

getAppendMarkers

public Set<String> getAppendMarkers(String markerDir)

Parameters:

markerDir - marker directory path

Returns:

all marker paths of write IO type "APPEND"

doesMarkerDirExist

public boolean doesMarkerDirExist(String markerDir)

Parameters:

markerDir - marker directory path

Returns:

true if the marker directory exists; false otherwise.

createMarker

public CompletableFuture<String> createMarker(io.javalin.http.Context context,
                                              String markerDir,
                                              String markerName,
                                              String basePath)

Generates a future for an async marker creation request The future is added to the marker creation future list and waits for the next batch processing of marker creation requests.

Parameters:

context - Javalin app context

markerDir - marker directory path

markerName - marker name

Returns:

the CompletableFuture instance for the request

deleteMarkers

public Boolean deleteMarkers(String markerDir)

Deletes markers in the directory.

Parameters:

markerDir - marker directory path

Returns:

true if successful; false otherwise.