org.apache.hadoop.fs.azurebfs.services

Class AbfsClient

java.lang.Object

org.apache.hadoop.fs.azurebfs.services.AbfsClient

All Implemented Interfaces:

Closeable, AutoCloseable

public class AbfsClient
extends Object
implements Closeable

AbfsClient.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
AbfsClient(URL baseUrl, SharedKeyCredentials sharedKeyCredentials, AbfsConfiguration abfsConfiguration, AccessTokenProvider tokenProvider, AbfsClientContext abfsClientContext)
AbfsClient(URL baseUrl, SharedKeyCredentials sharedKeyCredentials, AbfsConfiguration abfsConfiguration, SASTokenProvider sasTokenProvider, AbfsClientContext abfsClientContext)

Method Summary

Modifier and Type	Method and Description
AbfsRestOperation	acquireLease(String path, int duration)
<V> void	addCallback(org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListenableFuture<V> future, org.apache.hadoop.thirdparty.com.google.common.util.concurrent.FutureCallback<V> callback)
AbfsRestOperation	append(String path, byte[] buffer, AppendRequestParameters reqParams, String cachedSasToken)
boolean	appendSuccessCheckOp(AbfsRestOperation op, String path, long length)
AbfsRestOperation	breakLease(String path)
AbfsRestOperation	checkAccess(String path, String rwx) Talks to the server to check whether the permission specified in the rwx parameter is present for the path specified in the path parameter.
void	close()
AbfsRestOperation	createFilesystem()
AbfsRestOperation	createPath(String path, boolean isFile, boolean overwrite, String permission, String umask, boolean isAppendBlob, String eTag)
protected URL	createRequestUrl(String path, String query)
AbfsRestOperation	deleteFilesystem()
AbfsRestOperation	deleteIdempotencyCheckOp(AbfsRestOperation op) Check if the delete request failure is post a retry and if delete failure qualifies to be a success response assuming idempotency.
AbfsRestOperation	deletePath(String path, boolean recursive, String continuation)
AbfsRestOperation	flush(String path, long position, boolean retainUncommittedData, boolean isClose, String cachedSasToken, String leaseId)
protected AbfsCounters	getAbfsCounters() Getter for abfsCounters from AbfsClient.
protected AbfsPerfTracker	getAbfsPerfTracker()
String	getAccessToken()
AbfsRestOperation	getAclStatus(String path)
AbfsRestOperation	getAclStatus(String path, boolean useUPN)
AuthType	getAuthType()
static String	getDirectoryQueryParameter(String path) Get the directory query parameter used by the List Paths REST API and used as the path in the continuation token.
String	getFileSystem()
AbfsRestOperation	getFilesystemProperties()
int	getNumLeaseThreads()
AbfsRestOperation	getPathStatus(String path, boolean includeProperties)
SASTokenProvider	getSasTokenProvider()
AbfsRestOperation	listPath(String relativePath, boolean recursive, int listMaxResults, String continuation)
AbfsRestOperation	read(String path, long position, byte[] buffer, int bufferOffset, int bufferLength, String eTag, String cachedSasToken)
AbfsRestOperation	releaseLease(String path, String leaseId)
AbfsRestOperation	renameIdempotencyCheckOp(Instant renameRequestStartTime, AbfsRestOperation op, String destination) Check if the rename request failure is post a retry and if earlier rename request might have succeeded at back-end.
AbfsRestOperation	renamePath(String source, String destination, String continuation)
AbfsRestOperation	renewLease(String path, String leaseId)
<V> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListenableScheduledFuture<V>	schedule(Callable<V> callable, long delay, TimeUnit timeUnit)
AbfsRestOperation	setAcl(String path, String aclSpecString)
AbfsRestOperation	setAcl(String path, String aclSpecString, String eTag)
AbfsRestOperation	setFilesystemProperties(String properties)
AbfsRestOperation	setOwner(String path, String owner, String group)
AbfsRestOperation	setPathProperties(String path, String properties)
AbfsRestOperation	setPermission(String path, String permission)
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListenableFuture<?>	submit(Runnable runnable)
static String	urlEncode(String value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

AbfsClient

public AbfsClient(URL baseUrl,
                  SharedKeyCredentials sharedKeyCredentials,
                  AbfsConfiguration abfsConfiguration,
                  AccessTokenProvider tokenProvider,
                  AbfsClientContext abfsClientContext)
           throws IOException

Throws:

IOException

AbfsClient

public AbfsClient(URL baseUrl,
                  SharedKeyCredentials sharedKeyCredentials,
                  AbfsConfiguration abfsConfiguration,
                  SASTokenProvider sasTokenProvider,
                  AbfsClientContext abfsClientContext)
           throws IOException

Throws:

IOException

Method Detail

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getFileSystem

public String getFileSystem()

getAbfsPerfTracker

protected AbfsPerfTracker getAbfsPerfTracker()

createFilesystem

public AbfsRestOperation createFilesystem()
                                   throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

setFilesystemProperties

public AbfsRestOperation setFilesystemProperties(String properties)
                                          throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

listPath

public AbfsRestOperation listPath(String relativePath,
                                  boolean recursive,
                                  int listMaxResults,
                                  String continuation)
                           throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

getFilesystemProperties

public AbfsRestOperation getFilesystemProperties()
                                          throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

deleteFilesystem

public AbfsRestOperation deleteFilesystem()
                                   throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

createPath

public AbfsRestOperation createPath(String path,
                                    boolean isFile,
                                    boolean overwrite,
                                    String permission,
                                    String umask,
                                    boolean isAppendBlob,
                                    String eTag)
                             throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

acquireLease

public AbfsRestOperation acquireLease(String path,
                                      int duration)
                               throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

renewLease

public AbfsRestOperation renewLease(String path,
                                    String leaseId)
                             throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

releaseLease

public AbfsRestOperation releaseLease(String path,
                                      String leaseId)
                               throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

breakLease

public AbfsRestOperation breakLease(String path)
                             throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

renamePath

public AbfsRestOperation renamePath(String source,
                                    String destination,
                                    String continuation)
                             throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

renameIdempotencyCheckOp

public AbfsRestOperation renameIdempotencyCheckOp(Instant renameRequestStartTime,
                                                  AbfsRestOperation op,
                                                  String destination)
                                           throws AzureBlobFileSystemException

Check if the rename request failure is post a retry and if earlier rename request might have succeeded at back-end. If there is a parallel rename activity happening from any other store interface, the logic here will detect the rename to have happened due to the one initiated from this ABFS filesytem instance as it was retried. This should be a corner case hence going ahead with LMT check.

Parameters:

renameRequestStartTime - startTime for the rename request

op - Rename request REST operation response

destination - rename destination path

Returns:

REST operation response post idempotency check

Throws:

AzureBlobFileSystemException - if GetFileStatus hits any exception

append

public AbfsRestOperation append(String path,
                                byte[] buffer,
                                AppendRequestParameters reqParams,
                                String cachedSasToken)
                         throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

appendSuccessCheckOp

public boolean appendSuccessCheckOp(AbfsRestOperation op,
                                    String path,
                                    long length)
                             throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

flush

public AbfsRestOperation flush(String path,
                               long position,
                               boolean retainUncommittedData,
                               boolean isClose,
                               String cachedSasToken,
                               String leaseId)
                        throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

setPathProperties

public AbfsRestOperation setPathProperties(String path,
                                           String properties)
                                    throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

getPathStatus

public AbfsRestOperation getPathStatus(String path,
                                       boolean includeProperties)
                                throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

read

public AbfsRestOperation read(String path,
                              long position,
                              byte[] buffer,
                              int bufferOffset,
                              int bufferLength,
                              String eTag,
                              String cachedSasToken)
                       throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

deletePath

public AbfsRestOperation deletePath(String path,
                                    boolean recursive,
                                    String continuation)
                             throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

deleteIdempotencyCheckOp

public AbfsRestOperation deleteIdempotencyCheckOp(AbfsRestOperation op)

Check if the delete request failure is post a retry and if delete failure qualifies to be a success response assuming idempotency. There are below scenarios where delete could be incorrectly deducted as success post request retry: 1. Target was originally not existing and initial delete request had to be re-tried. 2. Parallel delete issued from any other store interface rather than delete issued from this filesystem instance. These are few corner cases and usually returning a success at this stage should help the job to continue.

Parameters:

op - Delete request REST operation response

Returns:

REST operation response post idempotency check

setOwner

public AbfsRestOperation setOwner(String path,
                                  String owner,
                                  String group)
                           throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

setPermission

public AbfsRestOperation setPermission(String path,
                                       String permission)
                                throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

setAcl

public AbfsRestOperation setAcl(String path,
                                String aclSpecString)
                         throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

setAcl

public AbfsRestOperation setAcl(String path,
                                String aclSpecString,
                                String eTag)
                         throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

getAclStatus

public AbfsRestOperation getAclStatus(String path)
                               throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

getAclStatus

public AbfsRestOperation getAclStatus(String path,
                                      boolean useUPN)
                               throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

checkAccess

public AbfsRestOperation checkAccess(String path,
                                     String rwx)
                              throws AzureBlobFileSystemException

Talks to the server to check whether the permission specified in the rwx parameter is present for the path specified in the path parameter.

Parameters:

path - Path for which access check needs to be performed

rwx - The permission to be checked on the path

Returns:

The AbfsRestOperation object for the operation

Throws:

AzureBlobFileSystemException - in case of bad requests

getDirectoryQueryParameter

public static String getDirectoryQueryParameter(String path)

Get the directory query parameter used by the List Paths REST API and used as the path in the continuation token. If the input path is null or the root path "/", empty string is returned. If the input path begins with '/', the return value is the substring beginning at offset 1. Otherwise, the input path is returned.

Parameters:

path - the path to be listed.

Returns:

the value of the directory query parameter

createRequestUrl

protected URL createRequestUrl(String path,
                               String query)
                        throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

urlEncode

public static String urlEncode(String value)
                        throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

getAccessToken

public String getAccessToken()
                      throws IOException

Throws:

IOException

getAuthType

public AuthType getAuthType()

getSasTokenProvider

public SASTokenProvider getSasTokenProvider()

getAbfsCounters

protected AbfsCounters getAbfsCounters()

Getter for abfsCounters from AbfsClient.

Returns:

AbfsCounters instance.

getNumLeaseThreads

public int getNumLeaseThreads()

schedule

public <V> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListenableScheduledFuture<V> schedule(Callable<V> callable,
                                                                                                                long delay,
                                                                                                                TimeUnit timeUnit)

submit

public org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListenableFuture<?> submit(Runnable runnable)

addCallback

public <V> void addCallback(org.apache.hadoop.thirdparty.com.google.common.util.concurrent.ListenableFuture<V> future,
                            org.apache.hadoop.thirdparty.com.google.common.util.concurrent.FutureCallback<V> callback)