org.apache.hudi.common.fs

Class FailSafeConsistencyGuard

java.lang.Object

org.apache.hudi.common.fs.FailSafeConsistencyGuard

All Implemented Interfaces:

ConsistencyGuard

Direct Known Subclasses:

OptimisticConsistencyGuard

public class FailSafeConsistencyGuard
extends Object
implements ConsistencyGuard

A consistency checker that fails if it is unable to meet the required condition within a specified timeout.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.fs.ConsistencyGuard

ConsistencyGuard.FileVisibility

Field Summary

Fields

Modifier and Type	Field and Description
protected ConsistencyGuardConfig	consistencyGuardConfig
protected org.apache.hadoop.fs.FileSystem	fs

Constructor Summary

Constructors

Constructor and Description
FailSafeConsistencyGuard(org.apache.hadoop.fs.FileSystem fs, ConsistencyGuardConfig consistencyGuardConfig)

Method Summary

Modifier and Type	Method and Description
protected boolean	checkFilesVisibility(int retryNum, org.apache.hadoop.fs.Path dir, List<String> files, ConsistencyGuard.FileVisibility event) Helper to check for file visibility based on ConsistencyGuard.FileVisibility event.
protected boolean	checkFileVisibility(org.apache.hadoop.fs.Path filePath, ConsistencyGuard.FileVisibility visibility) Helper to check of file visibility.
protected List<String>	getFilesWithoutSchemeAndAuthority(List<String> files) Generate file names without scheme and authority.
void	waitForFilesVisibility(String dirPath, List<String> files, ConsistencyGuard.FileVisibility event) Helper function to wait for all files belonging to single directory to appear.
void	waitTillAllFilesAppear(String dirPath, List<String> files) Wait till all passed files belonging to a directory shows up in the listing.
void	waitTillAllFilesDisappear(String dirPath, List<String> files) Wait till all passed files belonging to a directory disappears from listing.
void	waitTillFileAppears(org.apache.hadoop.fs.Path filePath) Wait for file to be listable based on configurable timeout.
void	waitTillFileDisappears(org.apache.hadoop.fs.Path filePath) Wait for file to be listable based on configurable timeout.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.common.fs.ConsistencyGuard

waitTill

Field Detail

fs

protected final org.apache.hadoop.fs.FileSystem fs

consistencyGuardConfig

protected final ConsistencyGuardConfig consistencyGuardConfig

Constructor Detail

FailSafeConsistencyGuard

public FailSafeConsistencyGuard(org.apache.hadoop.fs.FileSystem fs,
                                ConsistencyGuardConfig consistencyGuardConfig)

Method Detail

waitTillFileAppears

public void waitTillFileAppears(org.apache.hadoop.fs.Path filePath)
                         throws TimeoutException

Description copied from interface: ConsistencyGuard

Wait for file to be listable based on configurable timeout.

Specified by:

waitTillFileAppears in interface ConsistencyGuard

Throws:

TimeoutException - when retries exhausted

waitTillFileDisappears

public void waitTillFileDisappears(org.apache.hadoop.fs.Path filePath)
                            throws TimeoutException

Description copied from interface: ConsistencyGuard

Wait for file to be listable based on configurable timeout.

Specified by:

waitTillFileDisappears in interface ConsistencyGuard

Throws:

TimeoutException - when retries exhausted

waitTillAllFilesAppear

public void waitTillAllFilesAppear(String dirPath,
                                   List<String> files)
                            throws TimeoutException

Description copied from interface: ConsistencyGuard

Wait till all passed files belonging to a directory shows up in the listing.

Specified by:

waitTillAllFilesAppear in interface ConsistencyGuard

Throws:

TimeoutException

waitTillAllFilesDisappear

public void waitTillAllFilesDisappear(String dirPath,
                                      List<String> files)
                               throws TimeoutException

Description copied from interface: ConsistencyGuard

Wait till all passed files belonging to a directory disappears from listing.

Specified by:

waitTillAllFilesDisappear in interface ConsistencyGuard

Throws:

TimeoutException

waitForFilesVisibility

public void waitForFilesVisibility(String dirPath,
                                   List<String> files,
                                   ConsistencyGuard.FileVisibility event)
                            throws TimeoutException

Helper function to wait for all files belonging to single directory to appear.

Parameters:

dirPath - Dir Path

files - Files to appear/disappear

event - Appear/Disappear

Throws:

TimeoutException

checkFileVisibility

protected boolean checkFileVisibility(org.apache.hadoop.fs.Path filePath,
                                      ConsistencyGuard.FileVisibility visibility)
                               throws IOException

Helper to check of file visibility.

Parameters:

filePath - File Path

visibility - Visibility

Returns:

true (if file visible in Path), false (otherwise)

Throws:

IOException - -

checkFilesVisibility

protected boolean checkFilesVisibility(int retryNum,
                                       org.apache.hadoop.fs.Path dir,
                                       List<String> files,
                                       ConsistencyGuard.FileVisibility event)

Helper to check for file visibility based on ConsistencyGuard.FileVisibility event.

Parameters:

retryNum - retry attempt count.

dir - directory of interest in which list of files are checked for visibility

files - List of files to check for visibility

event - ConsistencyGuard.FileVisibility event of interest.

Returns:

true if condition succeeded. else false.

getFilesWithoutSchemeAndAuthority

protected List<String> getFilesWithoutSchemeAndAuthority(List<String> files)

Generate file names without scheme and authority.

Parameters:

files - list of files of interest.

Returns:

the filenames without scheme and authority.