org.apache.hudi.common.fs

Class OptimisticConsistencyGuard

java.lang.Object

org.apache.hudi.common.fs.FailSafeConsistencyGuard

org.apache.hudi.common.fs.OptimisticConsistencyGuard

All Implemented Interfaces:

ConsistencyGuard

public class OptimisticConsistencyGuard
extends FailSafeConsistencyGuard

A consistency guard which sleeps for configured period of time only on APPEAR. It is a no-op for DISAPPEAR. This is specifically for S3A filesystem and here is the rational. This guard is used when deleting data files corresponding to marker files that needs to be deleted. There are two tricky cases that needs to be considered. Case 1 : A data file creation is eventually consistent and hence when issuing deletes, it may not be found. Case 2: a data file was never created in the first place since the process crashed. In S3A, GET and LIST are eventually consistent, and delete() implementation internally does a LIST/EXISTS. Prior to this patch, hudi was leveraging FailSafeConsistencyGuard which was doing the following to delete data files. Step1: wait for all files to appear with linear backoff. Step2: issue deletes Step3: wait for all files to disappear with linear backoff. Step1 and Step2 is handled by FailSafeConsistencyGuard. We are simplifying these steps with OptimisticConsistencyGuard. Step1: Check if all files adhere to visibility event. If yes, proceed to Sptep 3. Step2: If not, Sleep for a configured threshold and then proceed to next step. Step3: issue deletes. With this, if any files that was created, should be available within configured threshold(eventual consistency). Delete() will return false if FileNotFound. So, both cases are taken care of this ConsistencyGuard.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.fs.ConsistencyGuard

ConsistencyGuard.FileVisibility

Field Summary

Fields inherited from class org.apache.hudi.common.fs.FailSafeConsistencyGuard

consistencyGuardConfig, fs

Constructor Summary

Constructors

Constructor and Description
OptimisticConsistencyGuard(org.apache.hadoop.fs.FileSystem fs, ConsistencyGuardConfig consistencyGuardConfig)

Method Summary

Modifier and Type	Method and Description
void	waitTillAllFilesAppear(String dirPath, List<String> files) Wait till all passed files belonging to a directory shows up in the listing.
void	waitTillAllFilesDisappear(String dirPath, List<String> files) Wait till all passed files belonging to a directory disappears from listing.
void	waitTillFileAppears(org.apache.hadoop.fs.Path filePath) Wait for file to be listable based on configurable timeout.
void	waitTillFileDisappears(org.apache.hadoop.fs.Path filePath) Wait for file to be listable based on configurable timeout.

Methods inherited from class org.apache.hudi.common.fs.FailSafeConsistencyGuard

checkFilesVisibility, checkFileVisibility, getFilesWithoutSchemeAndAuthority, waitForFilesVisibility

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.common.fs.ConsistencyGuard

waitTill

Constructor Detail

OptimisticConsistencyGuard

public OptimisticConsistencyGuard(org.apache.hadoop.fs.FileSystem fs,
                                  ConsistencyGuardConfig consistencyGuardConfig)

Method Detail

waitTillFileAppears

public void waitTillFileAppears(org.apache.hadoop.fs.Path filePath)
                         throws TimeoutException

Description copied from interface: ConsistencyGuard

Wait for file to be listable based on configurable timeout.

Specified by:

waitTillFileAppears in interface ConsistencyGuard

Overrides:

waitTillFileAppears in class FailSafeConsistencyGuard

Throws:

TimeoutException - when retries exhausted

waitTillFileDisappears

public void waitTillFileDisappears(org.apache.hadoop.fs.Path filePath)
                            throws TimeoutException

Description copied from interface: ConsistencyGuard

Wait for file to be listable based on configurable timeout.

Specified by:

waitTillFileDisappears in interface ConsistencyGuard

Overrides:

waitTillFileDisappears in class FailSafeConsistencyGuard

Throws:

TimeoutException - when retries exhausted

waitTillAllFilesAppear

public void waitTillAllFilesAppear(String dirPath,
                                   List<String> files)
                            throws TimeoutException

Description copied from interface: ConsistencyGuard

Wait till all passed files belonging to a directory shows up in the listing.

Specified by:

waitTillAllFilesAppear in interface ConsistencyGuard

Overrides:

waitTillAllFilesAppear in class FailSafeConsistencyGuard

Throws:

TimeoutException

waitTillAllFilesDisappear

public void waitTillAllFilesDisappear(String dirPath,
                                      List<String> files)
                               throws TimeoutException

Description copied from interface: ConsistencyGuard

Wait till all passed files belonging to a directory disappears from listing.

Specified by:

waitTillAllFilesDisappear in interface ConsistencyGuard

Overrides:

waitTillAllFilesDisappear in class FailSafeConsistencyGuard

Throws:

TimeoutException