org.apache.hudi.common.util

Class MarkerUtils

java.lang.Object

org.apache.hudi.common.util.MarkerUtils

public class MarkerUtils
extends Object

A utility class for marker related operations.

Field Summary

Fields

Modifier and Type	Field and Description
static String	MARKER_TYPE_FILENAME
static String	MARKERS_FILENAME_PREFIX

Constructor Summary

Constructors

Constructor and Description
MarkerUtils()

Method Summary

Modifier and Type	Method and Description
static void	deleteMarkerTypeFile(org.apache.hadoop.fs.FileSystem fileSystem, String markerDir) Deletes `MARKERS.type` file.
static boolean	doesMarkerTypeFileExist(org.apache.hadoop.fs.FileSystem fileSystem, String markerDir)
static Set<String>	readMarkersFromFile(org.apache.hadoop.fs.Path markersFilePath, SerializableConfiguration conf) Reads the markers stored in the underlying file.
static Option<MarkerType>	readMarkerType(org.apache.hadoop.fs.FileSystem fileSystem, String markerDir) Reads the marker type from `MARKERS.type` file.
static Map<String,Set<String>>	readTimelineServerBasedMarkersFromFileSystem(String markerDir, org.apache.hadoop.fs.FileSystem fileSystem, HoodieEngineContext context, int parallelism) Reads files containing the markers written by timeline-server-based marker mechanism.
static String	stripMarkerFolderPrefix(String fullMarkerPath, String markerDir) Strips the marker folder prefix of any file path under the marker directory.
static String	stripMarkerFolderPrefix(String fullMarkerPath, String basePath, String instantTime) Strips the folder prefix of the marker file path corresponding to a data file.
static void	writeMarkerTypeToFile(MarkerType markerType, org.apache.hadoop.fs.FileSystem fileSystem, String markerDir) Writes the marker type to the file `MARKERS.type`.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MARKERS_FILENAME_PREFIX

public static final String MARKERS_FILENAME_PREFIX

See Also:

Constant Field Values

MARKER_TYPE_FILENAME

public static final String MARKER_TYPE_FILENAME

See Also:

Constant Field Values

Constructor Detail

MarkerUtils

public MarkerUtils()

Method Detail

stripMarkerFolderPrefix

public static String stripMarkerFolderPrefix(String fullMarkerPath,
                                             String basePath,
                                             String instantTime)

Strips the folder prefix of the marker file path corresponding to a data file.

Parameters:

fullMarkerPath - the full path of the marker file

basePath - the base path

instantTime - instant of interest

Returns:

marker file name

stripMarkerFolderPrefix

public static String stripMarkerFolderPrefix(String fullMarkerPath,
                                             String markerDir)

Strips the marker folder prefix of any file path under the marker directory.

Parameters:

fullMarkerPath - the full path of the file

markerDir - marker directory

Returns:

file name

doesMarkerTypeFileExist

public static boolean doesMarkerTypeFileExist(org.apache.hadoop.fs.FileSystem fileSystem,
                                              String markerDir)
                                       throws IOException

Parameters:

fileSystem - file system to use.

markerDir - marker directory.

Returns:

true if the MARKERS.type file exists; false otherwise.

Throws:

IOException

readMarkerType

public static Option<MarkerType> readMarkerType(org.apache.hadoop.fs.FileSystem fileSystem,
                                                String markerDir)

Reads the marker type from `MARKERS.type` file.

Parameters:

fileSystem - file system to use.

markerDir - marker directory.

Returns:

the marker type, or empty if the marker type file does not exist.

writeMarkerTypeToFile

public static void writeMarkerTypeToFile(MarkerType markerType,
                                         org.apache.hadoop.fs.FileSystem fileSystem,
                                         String markerDir)

Writes the marker type to the file `MARKERS.type`.

Parameters:

markerType - marker type.

fileSystem - file system to use.

markerDir - marker directory.

deleteMarkerTypeFile

public static void deleteMarkerTypeFile(org.apache.hadoop.fs.FileSystem fileSystem,
                                        String markerDir)

Deletes `MARKERS.type` file.

Parameters:

fileSystem - file system to use.

markerDir - marker directory.

readTimelineServerBasedMarkersFromFileSystem

public static Map<String,Set<String>> readTimelineServerBasedMarkersFromFileSystem(String markerDir,
                                                                                   org.apache.hadoop.fs.FileSystem fileSystem,
                                                                                   HoodieEngineContext context,
                                                                                   int parallelism)

Reads files containing the markers written by timeline-server-based marker mechanism.

Parameters:

markerDir - marker directory.

fileSystem - file system to use.

context - instance of HoodieEngineContext to use

parallelism - parallelism to use

Returns:

A Map of file name to the set of markers stored in the file.

readMarkersFromFile

public static Set<String> readMarkersFromFile(org.apache.hadoop.fs.Path markersFilePath,
                                              SerializableConfiguration conf)

Reads the markers stored in the underlying file.

Parameters:

markersFilePath - file path for the markers

conf - serializable config

Returns:

markers in a Set of String.