org.apache.hudi.table.marker

Class DirectWriteMarkers

java.lang.Object

org.apache.hudi.table.marker.WriteMarkers

org.apache.hudi.table.marker.DirectWriteMarkers

All Implemented Interfaces:

Serializable

public class DirectWriteMarkers
extends WriteMarkers

Marker operations of directly accessing the file system to create and delete marker files. Each data file has a corresponding marker file.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.table.marker.WriteMarkers

basePath, instantTime, markerDirPath

Constructor Summary

Constructors

Constructor and Description
DirectWriteMarkers(org.apache.hadoop.fs.FileSystem fs, String basePath, String markerFolderPath, String instantTime)
DirectWriteMarkers(HoodieTable table, String instantTime)

Method Summary

Modifier and Type	Method and Description
Set<String>	allMarkerFilePaths()
Option<org.apache.hadoop.fs.Path>	create(String markerName) Creates a marker file based on the full marker name excluding the base path and instant.
protected Option<org.apache.hadoop.fs.Path>	create(String partitionPath, String dataFileName, IOType type, boolean checkIfExists) Creates a marker.
Set<String>	createdAndMergedDataPaths(HoodieEngineContext context, int parallelism)
Option<org.apache.hadoop.fs.Path>	createWithEarlyConflictDetection(String partitionPath, String dataFileName, IOType type, boolean checkIfExists, HoodieWriteConfig config, String fileId, HoodieActiveTimeline activeTimeline) Creates a marker with early conflict detection for multi-writers.
boolean	deleteMarkerDir(HoodieEngineContext context, int parallelism) Deletes Marker directory corresponding to an instant.
boolean	doesMarkerDirExist()

Methods inherited from class org.apache.hudi.table.marker.WriteMarkers

create, create, createIfNotExists, createIfNotExists, getMarkerFileName, getMarkerPath, quietDeleteMarkerDir, stripMarkerSuffix

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DirectWriteMarkers

public DirectWriteMarkers(org.apache.hadoop.fs.FileSystem fs,
                          String basePath,
                          String markerFolderPath,
                          String instantTime)

DirectWriteMarkers

public DirectWriteMarkers(HoodieTable table,
                          String instantTime)

Method Detail

deleteMarkerDir

public boolean deleteMarkerDir(HoodieEngineContext context,
                               int parallelism)

Deletes Marker directory corresponding to an instant.

Specified by:

deleteMarkerDir in class WriteMarkers

Parameters:

context - HoodieEngineContext.

parallelism - parallelism for deletion.

Returns:

if successful; otherwise.

doesMarkerDirExist

public boolean doesMarkerDirExist()
                           throws IOException

Specified by:

doesMarkerDirExist in class WriteMarkers

Returns:

true if marker directory exists; false otherwise.

Throws:

IOException

createdAndMergedDataPaths

public Set<String> createdAndMergedDataPaths(HoodieEngineContext context,
                                             int parallelism)
                                      throws IOException

Specified by:

createdAndMergedDataPaths in class WriteMarkers

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for reading the marker files in the directory.

Returns:

all the data file paths of write IO type "CREATE" and "MERGE"

Throws:

IOException

allMarkerFilePaths

public Set<String> allMarkerFilePaths()
                               throws IOException

Specified by:

allMarkerFilePaths in class WriteMarkers

Returns:

all the marker paths

Throws:

IOException

create

public Option<org.apache.hadoop.fs.Path> create(String markerName)

Creates a marker file based on the full marker name excluding the base path and instant.

Parameters:

markerName - the full marker name, e.g., "2021/08/13/file1.marker.CREATE"

Returns:

path of the marker file

create

protected Option<org.apache.hadoop.fs.Path> create(String partitionPath,
                                                   String dataFileName,
                                                   IOType type,
                                                   boolean checkIfExists)

Description copied from class: WriteMarkers

Creates a marker.

Parameters:

partitionPath - partition path in the table

dataFileName - data file name

type - write IO type

checkIfExists - whether to check if the marker already exists

Returns:

the marker path or empty option if already exists and checkIfExists is true

createWithEarlyConflictDetection

public Option<org.apache.hadoop.fs.Path> createWithEarlyConflictDetection(String partitionPath,
                                                                          String dataFileName,
                                                                          IOType type,
                                                                          boolean checkIfExists,
                                                                          HoodieWriteConfig config,
                                                                          String fileId,
                                                                          HoodieActiveTimeline activeTimeline)

Description copied from class: WriteMarkers

Creates a marker with early conflict detection for multi-writers. If conflict is detected, an exception is thrown to fail the write operation.

Specified by:

createWithEarlyConflictDetection in class WriteMarkers

Parameters:

partitionPath - partition path in the table.

dataFileName - data file name.

type - write IO type.

checkIfExists - whether to check if the marker already exists.

config - Hudi write configs.

fileId - File ID.

activeTimeline - Active timeline for the write operation.

Returns:

the marker path or empty option if already exists and checkIfExists is true.