org.apache.hudi.timeline.service.handlers.marker

Class MarkerDirState

java.lang.Object

org.apache.hudi.timeline.service.handlers.marker.MarkerDirState

All Implemented Interfaces:

Serializable

public class MarkerDirState
extends Object
implements Serializable

Stores the state of a marker directory. The operations inside this class is designed to be thread-safe.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
MarkerDirState(String markerDirPath, int markerBatchNumThreads, org.apache.hadoop.fs.FileSystem fileSystem, Registry metricsRegistry, HoodieEngineContext hoodieEngineContext, int parallelism)

Method Summary

Modifier and Type	Method and Description
void	addMarkerCreationFuture(MarkerCreationFuture future) Adds a MarkerCreationCompletableFuture instance from a marker creation request to the queue.
boolean	deleteAllMarkers() Deletes markers in the directory.
boolean	exists()
List<MarkerCreationFuture>	fetchPendingMarkerCreationRequests()
Set<String>	getAllMarkers()
Option<Integer>	getNextFileIndexToUse()
void	markFileAsAvailable(int fileIndex) Marks the file as available to use again.
void	processMarkerCreationRequests(List<MarkerCreationFuture> pendingMarkerCreationFutures, int fileIndex) Processes pending marker creation requests.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MarkerDirState

public MarkerDirState(String markerDirPath,
                      int markerBatchNumThreads,
                      org.apache.hadoop.fs.FileSystem fileSystem,
                      Registry metricsRegistry,
                      HoodieEngineContext hoodieEngineContext,
                      int parallelism)

Method Detail

exists

public boolean exists()

Returns:

true if the marker directory exists in the system.

getAllMarkers

public Set<String> getAllMarkers()

Returns:

all markers in the marker directory.

addMarkerCreationFuture

public void addMarkerCreationFuture(MarkerCreationFuture future)

Adds a MarkerCreationCompletableFuture instance from a marker creation request to the queue.

Parameters:

future - MarkerCreationCompletableFuture instance.

getNextFileIndexToUse

public Option<Integer> getNextFileIndexToUse()

Returns:

the next file index to use in a round-robin fashion, or empty if no file is available.

markFileAsAvailable

public void markFileAsAvailable(int fileIndex)

Marks the file as available to use again.

Parameters:

fileIndex - file index

fetchPendingMarkerCreationRequests

public List<MarkerCreationFuture> fetchPendingMarkerCreationRequests()

Returns:

futures of pending marker creation requests and removes them from the list.

processMarkerCreationRequests

public void processMarkerCreationRequests(List<MarkerCreationFuture> pendingMarkerCreationFutures,
                                          int fileIndex)

Processes pending marker creation requests.

Parameters:

pendingMarkerCreationFutures - futures of pending marker creation requests

fileIndex - file index to use to write markers

deleteAllMarkers

public boolean deleteAllMarkers()

Deletes markers in the directory.

Returns:

true if successful; false otherwise.