org.apache.hudi.table.marker

Class WriteMarkers

java.lang.Object

org.apache.hudi.table.marker.WriteMarkers

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

DirectWriteMarkers, TimelineServerBasedWriteMarkers

public abstract class WriteMarkers
extends Object
implements Serializable

Operates on markers for a given write action (commit, delta commit, compaction). This abstract class provides abstract methods of different marker operations, so that different marker write mechanism can be implemented.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected String	basePath
protected String	instantTime
protected org.apache.hadoop.fs.Path	markerDirPath

Constructor Summary

Constructors

Constructor and Description
WriteMarkers(String basePath, String markerFolderPath, String instantTime)

Method Summary

Modifier and Type	Method and Description
abstract Set<String>	allMarkerFilePaths()
Option<org.apache.hadoop.fs.Path>	create(String partitionPath, String dataFileName, IOType type) Creates a marker without checking if the marker already exists.
Option<org.apache.hadoop.fs.Path>	create(String partitionPath, String dataFileName, IOType type, HoodieWriteConfig writeConfig, String fileId, HoodieActiveTimeline activeTimeline) Creates a marker without checking if the marker already exists.
abstract Set<String>	createdAndMergedDataPaths(HoodieEngineContext context, int parallelism)
Option<org.apache.hadoop.fs.Path>	createIfNotExists(String partitionPath, String dataFileName, IOType type) Creates a marker if the marker does not exist.
Option<org.apache.hadoop.fs.Path>	createIfNotExists(String partitionPath, String fileName, IOType type, HoodieWriteConfig writeConfig, String fileId, HoodieActiveTimeline activeTimeline) Creates a marker if the marker does not exist.
abstract Option<org.apache.hadoop.fs.Path>	createWithEarlyConflictDetection(String partitionPath, String dataFileName, IOType type, boolean checkIfExists, HoodieWriteConfig config, String fileId, HoodieActiveTimeline activeTimeline) Creates a marker with early conflict detection for multi-writers.
abstract boolean	deleteMarkerDir(HoodieEngineContext context, int parallelism) Deletes the marker directory.
abstract boolean	doesMarkerDirExist()
protected String	getMarkerFileName(String dataFileName, IOType type) Gets the marker file name, in the format of "[data_file_name].marker.[IO_type]".
protected org.apache.hadoop.fs.Path	getMarkerPath(String partitionPath, String dataFileName, IOType type) Returns the marker path.
void	quietDeleteMarkerDir(HoodieEngineContext context, int parallelism) Quietly deletes the marker directory.
static String	stripMarkerSuffix(String path) Strips the marker file suffix from the input path, i.e., ".marker.[IO_type]".

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

basePath

protected final String basePath

markerDirPath

protected final transient org.apache.hadoop.fs.Path markerDirPath

instantTime

protected final String instantTime

Constructor Detail

WriteMarkers

public WriteMarkers(String basePath,
                    String markerFolderPath,
                    String instantTime)

Method Detail

create

public Option<org.apache.hadoop.fs.Path> create(String partitionPath,
                                                String dataFileName,
                                                IOType type)

Creates a marker without checking if the marker already exists.

Parameters:

partitionPath - partition path in the table.

dataFileName - data file name.

type - write IO type.

Returns:

the marker path.

create

public Option<org.apache.hadoop.fs.Path> create(String partitionPath,
                                                String dataFileName,
                                                IOType type,
                                                HoodieWriteConfig writeConfig,
                                                String fileId,
                                                HoodieActiveTimeline activeTimeline)

Creates a marker without checking if the marker already exists. This can invoke marker-based early conflict detection when enabled for multi-writers.

Parameters:

partitionPath - partition path in the table

dataFileName - data file name

type - write IO type

writeConfig - Hudi write configs.

fileId - File ID.

activeTimeline - Active timeline for the write operation.

Returns:

the marker path.

createIfNotExists

public Option<org.apache.hadoop.fs.Path> createIfNotExists(String partitionPath,
                                                           String dataFileName,
                                                           IOType type)

Creates a marker if the marker does not exist.

Parameters:

partitionPath - partition path in the table

dataFileName - data file name

type - write IO type

Returns:

the marker path or empty option if already exists

createIfNotExists

public Option<org.apache.hadoop.fs.Path> createIfNotExists(String partitionPath,
                                                           String fileName,
                                                           IOType type,
                                                           HoodieWriteConfig writeConfig,
                                                           String fileId,
                                                           HoodieActiveTimeline activeTimeline)

Creates a marker if the marker does not exist. This can invoke marker-based early conflict detection when enabled for multi-writers.

Parameters:

partitionPath - partition path in the table

fileName - file name

type - write IO type

writeConfig - Hudi write configs.

fileId - File ID.

activeTimeline - Active timeline for the write operation.

Returns:

the marker path.

quietDeleteMarkerDir

public void quietDeleteMarkerDir(HoodieEngineContext context,
                                 int parallelism)

Quietly deletes the marker directory.

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for deleting the marker files in the directory.

stripMarkerSuffix

public static String stripMarkerSuffix(String path)

Strips the marker file suffix from the input path, i.e., ".marker.[IO_type]".

Parameters:

path - file path

Returns:

Stripped path

getMarkerFileName

protected String getMarkerFileName(String dataFileName,
                                   IOType type)

Gets the marker file name, in the format of "[data_file_name].marker.[IO_type]".

Parameters:

dataFileName - data file name

type - IO type

Returns:

the marker file name

getMarkerPath

protected org.apache.hadoop.fs.Path getMarkerPath(String partitionPath,
                                                  String dataFileName,
                                                  IOType type)

Returns the marker path. Would create the partition path first if not exists

Parameters:

partitionPath - The partition path

dataFileName - The data file name

type - The IO type

Returns:

path of the marker file

deleteMarkerDir

public abstract boolean deleteMarkerDir(HoodieEngineContext context,
                                        int parallelism)

Deletes the marker directory.

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for deleting the marker files in the directory.

Returns:

if successful; otherwise.

doesMarkerDirExist

public abstract boolean doesMarkerDirExist()
                                    throws IOException

Returns:

if the marker directory exists in the file system; otherwise.

Throws:

IOException

createdAndMergedDataPaths

public abstract Set<String> createdAndMergedDataPaths(HoodieEngineContext context,
                                                      int parallelism)
                                               throws IOException

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for reading the marker files in the directory.

Returns:

all the data file paths of write IO type "CREATE" and "MERGE"

Throws:

IOException

allMarkerFilePaths

public abstract Set<String> allMarkerFilePaths()
                                        throws IOException

Returns:

all the marker paths

Throws:

IOException

createWithEarlyConflictDetection

public abstract Option<org.apache.hadoop.fs.Path> createWithEarlyConflictDetection(String partitionPath,
                                                                                   String dataFileName,
                                                                                   IOType type,
                                                                                   boolean checkIfExists,
                                                                                   HoodieWriteConfig config,
                                                                                   String fileId,
                                                                                   HoodieActiveTimeline activeTimeline)

Creates a marker with early conflict detection for multi-writers. If conflict is detected, an exception is thrown to fail the write operation.

Parameters:

partitionPath - partition path in the table.

dataFileName - data file name.

type - write IO type.

checkIfExists - whether to check if the marker already exists.

config - Hudi write configs.

fileId - File ID.

activeTimeline - Active timeline for the write operation.

Returns:

the marker path or empty option if already exists and checkIfExists is true.