org.apache.hudi.table.marker

Class WriteMarkers

java.lang.Object

org.apache.hudi.table.marker.WriteMarkers

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

DirectWriteMarkers, TimelineServerBasedWriteMarkers

public abstract class WriteMarkers
extends Object
implements Serializable

Operates on markers for a given write action (commit, delta commit, compaction). This abstract class provides abstract methods of different marker operations, so that different marker write mechanism can be implemented.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected String	basePath
protected String	instantTime
protected org.apache.hadoop.fs.Path	markerDirPath

Constructor Summary

Constructors

Constructor and Description
WriteMarkers(String basePath, String markerFolderPath, String instantTime)

Method Summary

Modifier and Type	Method and Description
abstract Set<String>	allMarkerFilePaths()
Option<org.apache.hadoop.fs.Path>	create(String partitionPath, String dataFileName, IOType type) Creates a marker without checking if the marker already exists.
abstract Set<String>	createdAndMergedDataPaths(HoodieEngineContext context, int parallelism)
Option<org.apache.hadoop.fs.Path>	createIfNotExists(String partitionPath, String dataFileName, IOType type) Creates a marker if the marker does not exist.
abstract boolean	deleteMarkerDir(HoodieEngineContext context, int parallelism) Deletes the marker directory.
abstract boolean	doesMarkerDirExist()
protected String	getMarkerFileName(String dataFileName, IOType type) Gets the marker file name, in the format of "[data_file_name].marker.[IO_type]".
protected org.apache.hadoop.fs.Path	getMarkerPath(String partitionPath, String dataFileName, IOType type) Returns the marker path.
void	quietDeleteMarkerDir(HoodieEngineContext context, int parallelism) Quietly deletes the marker directory.
static String	stripMarkerSuffix(String path) Strips the marker file suffix from the input path, i.e., ".marker.[IO_type]".

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

basePath

protected final String basePath

markerDirPath

protected final transient org.apache.hadoop.fs.Path markerDirPath

instantTime

protected final String instantTime

Constructor Detail

WriteMarkers

public WriteMarkers(String basePath,
                    String markerFolderPath,
                    String instantTime)

Method Detail

create

public Option<org.apache.hadoop.fs.Path> create(String partitionPath,
                                                String dataFileName,
                                                IOType type)

Creates a marker without checking if the marker already exists.

Parameters:

partitionPath - partition path in the table

dataFileName - data file name

type - write IO type

Returns:

the marker path

createIfNotExists

public Option<org.apache.hadoop.fs.Path> createIfNotExists(String partitionPath,
                                                           String dataFileName,
                                                           IOType type)

Creates a marker if the marker does not exist.

Parameters:

partitionPath - partition path in the table

dataFileName - data file name

type - write IO type

Returns:

the marker path or empty option if already exists

quietDeleteMarkerDir

public void quietDeleteMarkerDir(HoodieEngineContext context,
                                 int parallelism)

Quietly deletes the marker directory.

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for deleting the marker files in the directory.

stripMarkerSuffix

public static String stripMarkerSuffix(String path)

Strips the marker file suffix from the input path, i.e., ".marker.[IO_type]".

Parameters:

path - file path

Returns:

Stripped path

getMarkerFileName

protected String getMarkerFileName(String dataFileName,
                                   IOType type)

Gets the marker file name, in the format of "[data_file_name].marker.[IO_type]".

Parameters:

dataFileName - data file name

type - IO type

Returns:

the marker file name

getMarkerPath

protected org.apache.hadoop.fs.Path getMarkerPath(String partitionPath,
                                                  String dataFileName,
                                                  IOType type)

Returns the marker path. Would create the partition path first if not exists

Parameters:

partitionPath - The partition path

dataFileName - The data file name

type - The IO type

Returns:

path of the marker file

deleteMarkerDir

public abstract boolean deleteMarkerDir(HoodieEngineContext context,
                                        int parallelism)

Deletes the marker directory.

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for deleting the marker files in the directory.

Returns:

if successful; otherwise.

doesMarkerDirExist

public abstract boolean doesMarkerDirExist()
                                    throws IOException

Returns:

if the marker directory exists in the file system; otherwise.

Throws:

IOException

createdAndMergedDataPaths

public abstract Set<String> createdAndMergedDataPaths(HoodieEngineContext context,
                                                      int parallelism)
                                               throws IOException

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for reading the marker files in the directory.

Returns:

all the data file paths of write IO type "CREATE" and "MERGE"

Throws:

IOException

allMarkerFilePaths

public abstract Set<String> allMarkerFilePaths()
                                        throws IOException

Returns:

all the marker paths

Throws:

IOException