org.apache.hudi.table.marker

Class TimelineServerBasedWriteMarkers

java.lang.Object

org.apache.hudi.table.marker.WriteMarkers

org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers

All Implemented Interfaces:

Serializable

public class TimelineServerBasedWriteMarkers
extends WriteMarkers

Marker operations of using timeline server as a proxy to create and delete markers. Each data file has a corresponding marker entry, which is stored in a limited number of underlying files maintained by the timeline server (each file contains multiple marker entries).

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.table.marker.WriteMarkers

basePath, instantTime, markerDirPath

Constructor Summary

Constructors

Constructor and Description
TimelineServerBasedWriteMarkers(HoodieTable table, String instantTime)

Method Summary

Modifier and Type	Method and Description
Set<String>	allMarkerFilePaths()
protected Option<org.apache.hadoop.fs.Path>	create(String partitionPath, String dataFileName, IOType type, boolean checkIfExists) Creates a marker.
Set<String>	createdAndMergedDataPaths(HoodieEngineContext context, int parallelism)
Option<org.apache.hadoop.fs.Path>	createWithEarlyConflictDetection(String partitionPath, String dataFileName, IOType type, boolean checkIfExists, HoodieWriteConfig config, String fileId, HoodieActiveTimeline activeTimeline) Creates a marker with early conflict detection for multi-writers.
boolean	deleteMarkerDir(HoodieEngineContext context, int parallelism) Deletes the marker directory.
boolean	doesMarkerDirExist()

Methods inherited from class org.apache.hudi.table.marker.WriteMarkers

create, create, createIfNotExists, createIfNotExists, getMarkerFileName, getMarkerPath, quietDeleteMarkerDir, stripMarkerSuffix

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TimelineServerBasedWriteMarkers

public TimelineServerBasedWriteMarkers(HoodieTable table,
                                       String instantTime)

Method Detail

deleteMarkerDir

public boolean deleteMarkerDir(HoodieEngineContext context,
                               int parallelism)

Description copied from class: WriteMarkers

Deletes the marker directory.

Specified by:

deleteMarkerDir in class WriteMarkers

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for deleting the marker files in the directory.

Returns:

if successful; otherwise.

doesMarkerDirExist

public boolean doesMarkerDirExist()

Specified by:

doesMarkerDirExist in class WriteMarkers

Returns:

if the marker directory exists in the file system; otherwise.

createdAndMergedDataPaths

public Set<String> createdAndMergedDataPaths(HoodieEngineContext context,
                                             int parallelism)
                                      throws IOException

Specified by:

createdAndMergedDataPaths in class WriteMarkers

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for reading the marker files in the directory.

Returns:

all the data file paths of write IO type "CREATE" and "MERGE"

Throws:

IOException

allMarkerFilePaths

public Set<String> allMarkerFilePaths()

Specified by:

allMarkerFilePaths in class WriteMarkers

Returns:

all the marker paths

create

protected Option<org.apache.hadoop.fs.Path> create(String partitionPath,
                                                   String dataFileName,
                                                   IOType type,
                                                   boolean checkIfExists)

Description copied from class: WriteMarkers

Creates a marker.

Parameters:

partitionPath - partition path in the table

dataFileName - data file name

type - write IO type

checkIfExists - whether to check if the marker already exists

Returns:

the marker path or empty option if already exists and checkIfExists is true

createWithEarlyConflictDetection

public Option<org.apache.hadoop.fs.Path> createWithEarlyConflictDetection(String partitionPath,
                                                                          String dataFileName,
                                                                          IOType type,
                                                                          boolean checkIfExists,
                                                                          HoodieWriteConfig config,
                                                                          String fileId,
                                                                          HoodieActiveTimeline activeTimeline)

Description copied from class: WriteMarkers

Creates a marker with early conflict detection for multi-writers. If conflict is detected, an exception is thrown to fail the write operation.

Specified by:

createWithEarlyConflictDetection in class WriteMarkers

Parameters:

partitionPath - partition path in the table.

dataFileName - data file name.

type - write IO type.

checkIfExists - whether to check if the marker already exists.

config - Hudi write configs.

fileId - File ID.

activeTimeline - Active timeline for the write operation.

Returns:

the marker path or empty option if already exists and checkIfExists is true.