org.apache.hadoop.fs.swift.snative

Class SwiftNativeFileSystemStore

java.lang.Object

org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore

public class SwiftNativeFileSystemStore
extends Object

File system store implementation. Makes REST requests, parses data from responses

Constructor Summary

Constructors

Constructor and Description
SwiftNativeFileSystemStore()

Method Summary

Modifier and Type	Method and Description
void	copy(org.apache.hadoop.fs.Path srcKey, org.apache.hadoop.fs.Path dstKey)
void	createDirectory(org.apache.hadoop.fs.Path path) Create a directory
void	createManifestForPartUpload(org.apache.hadoop.fs.Path path) Tell the Swift server to expect a multi-part upload by submitting a 0-byte file with the X-Object-Manifest header
boolean	delete(org.apache.hadoop.fs.Path absolutePath, boolean recursive) Delete the entire tree.
boolean	deleteObject(org.apache.hadoop.fs.Path path) deletes object from Swift
static List<URI>	extractUris(String json, org.apache.hadoop.fs.Path path) extracts URIs from json
long	getBlocksize() Get the default blocksize of this (bound) filesystem
int	getBufferSizeKB()
org.apache.hadoop.fs.Path	getCorrectSwiftPath(org.apache.hadoop.fs.Path path) Take a Hadoop path and return one which uses the URI prefix and authority of this FS.
HttpBodyContent	getObject(org.apache.hadoop.fs.Path path) Get the object as an input stream
HttpBodyContent	getObject(org.apache.hadoop.fs.Path path, long byteRangeStart, long length) Get the input stream starting from a specific point.
org.apache.http.Header[]	getObjectHeaders(org.apache.hadoop.fs.Path path, boolean newest) Get the HTTP headers, in case you really need the low-level metadata
List<URI>	getObjectLocation(org.apache.hadoop.fs.Path path) Try to find the specific server(s) on which the data lives
SwiftFileStatus	getObjectMetadata(org.apache.hadoop.fs.Path path) Get the metadata of an object
SwiftFileStatus	getObjectMetadata(org.apache.hadoop.fs.Path path, boolean newest) Get the metadata of an object
List<DurationStats>	getOperationStatistics() Get the current operation statistics
long	getPartsizeKB()
int	getThrottleDelay()
void	initialize(URI fsURI, org.apache.hadoop.conf.Configuration configuration) Initalize the filesystem store -this creates the REST client binding.
org.apache.hadoop.fs.FileStatus[]	listSubPaths(org.apache.hadoop.fs.Path path, boolean recursive, boolean newest) List all elements in this directory
boolean	objectExists(org.apache.hadoop.fs.Path path) Does the object exist
boolean	objectExists(SwiftObjectPath path) Does the object exist
void	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Rename through copy-and-delete.
boolean	rmdir(org.apache.hadoop.fs.Path path) deletes a directory from Swift.
void	throttle() Insert a throttled wait if the throttle delay > 0
String	toString()
void	uploadFile(org.apache.hadoop.fs.Path path, InputStream inputStream, long length) Upload a file/input stream of a specific length.
void	uploadFilePart(org.apache.hadoop.fs.Path path, int partNumber, InputStream inputStream, long length) Upload part of a larger file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

SwiftNativeFileSystemStore

public SwiftNativeFileSystemStore()

Method Detail

initialize

public void initialize(URI fsURI,
                       org.apache.hadoop.conf.Configuration configuration)
                throws IOException

Initalize the filesystem store -this creates the REST client binding.

Parameters:

fsURI - URI of the filesystem, which is used to map to the filesystem-specific options in the configuration file

configuration - configuration

Throws:

IOException - on any failure.

toString

public String toString()

Overrides:

toString in class Object

getBlocksize

public long getBlocksize()

Get the default blocksize of this (bound) filesystem

Returns:

the blocksize returned for all FileStatus queries, which is used by the MapReduce splitter.

getPartsizeKB

public long getPartsizeKB()

getBufferSizeKB

public int getBufferSizeKB()

getThrottleDelay

public int getThrottleDelay()

uploadFile

public void uploadFile(org.apache.hadoop.fs.Path path,
                       InputStream inputStream,
                       long length)
                throws IOException

Upload a file/input stream of a specific length.

Parameters:

path - destination path in the swift filesystem

inputStream - input data. This is closed afterwards, always

length - length of the data

Throws:

IOException - on a problem

uploadFilePart

public void uploadFilePart(org.apache.hadoop.fs.Path path,
                           int partNumber,
                           InputStream inputStream,
                           long length)
                    throws IOException

Upload part of a larger file.

Parameters:

path - destination path

partNumber - item number in the path

inputStream - input data

length - length of the data

Throws:

IOException - on a problem

createManifestForPartUpload

public void createManifestForPartUpload(org.apache.hadoop.fs.Path path)
                                 throws IOException

Tell the Swift server to expect a multi-part upload by submitting a 0-byte file with the X-Object-Manifest header

Parameters:

path - path of final final

Throws:

IOException

getObjectMetadata

public SwiftFileStatus getObjectMetadata(org.apache.hadoop.fs.Path path)
                                  throws IOException

Get the metadata of an object

Parameters:

path - path

Returns:

file metadata. -or null if no headers were received back from the server.

Throws:

IOException - on a problem

FileNotFoundException - if there is nothing at the end

getObjectHeaders

public org.apache.http.Header[] getObjectHeaders(org.apache.hadoop.fs.Path path,
                                                 boolean newest)
                                          throws IOException,
                                                 FileNotFoundException

Get the HTTP headers, in case you really need the low-level metadata

Parameters:

path - path to probe

newest - newest or oldest?

Returns:

the header list

Throws:

IOException - IO problem

FileNotFoundException - if there is nothing at the end

getObjectMetadata

public SwiftFileStatus getObjectMetadata(org.apache.hadoop.fs.Path path,
                                         boolean newest)
                                  throws IOException,
                                         FileNotFoundException

Get the metadata of an object

Parameters:

path - path

newest - flag to say "set the newest header", otherwise take any entry

Returns:

file metadata. -or null if no headers were received back from the server.

Throws:

IOException - on a problem

FileNotFoundException - if there is nothing at the end

getObject

public HttpBodyContent getObject(org.apache.hadoop.fs.Path path)
                          throws IOException

Get the object as an input stream

Parameters:

path - object path

Returns:

the input stream -this must be closed to terminate the connection

Throws:

IOException - IO problems

FileNotFoundException - path doesn't resolve to an object

getObject

public HttpBodyContent getObject(org.apache.hadoop.fs.Path path,
                                 long byteRangeStart,
                                 long length)
                          throws IOException

Get the input stream starting from a specific point.

Parameters:

path - path to object

byteRangeStart - starting point

length - no. of bytes

Returns:

an input stream that must be closed

Throws:

IOException - IO problems

listSubPaths

public org.apache.hadoop.fs.FileStatus[] listSubPaths(org.apache.hadoop.fs.Path path,
                                                      boolean recursive,
                                                      boolean newest)
                                               throws IOException

List all elements in this directory

Parameters:

path - path to work with

recursive - do a recursive get

newest - ask for the newest, or can some out of date data work?

Returns:

the file statuses, or an empty array if there are no children

Throws:

IOException - on IO problems

FileNotFoundException - if the path is nonexistent

createDirectory

public void createDirectory(org.apache.hadoop.fs.Path path)
                     throws IOException

Create a directory

Parameters:

path - path

Throws:

IOException

getObjectLocation

public List<URI> getObjectLocation(org.apache.hadoop.fs.Path path)
                            throws IOException

Try to find the specific server(s) on which the data lives

Parameters:

path - path to probe

Returns:

a possibly empty list of locations

Throws:

IOException - on problems determining the locations

deleteObject

public boolean deleteObject(org.apache.hadoop.fs.Path path)
                     throws IOException

deletes object from Swift

Parameters:

path - path to delete

Returns:

true if the path was deleted by this specific operation.

Throws:

IOException - on a failure

rmdir

public boolean rmdir(org.apache.hadoop.fs.Path path)
              throws IOException

deletes a directory from Swift. This is not recursive

Parameters:

path - path to delete

Returns:

true if the path was deleted by this specific operation -or the path was root and not acted on.

Throws:

IOException - on a failure

objectExists

public boolean objectExists(org.apache.hadoop.fs.Path path)
                     throws IOException

Does the object exist

Parameters:

path - object path

Returns:

true if the metadata of an object could be retrieved

Throws:

IOException - IO problems other than FileNotFound, which is downgraded to an object does not exist return code

objectExists

public boolean objectExists(SwiftObjectPath path)
                     throws IOException

Does the object exist

Parameters:

path - swift object path

Returns:

true if the metadata of an object could be retrieved

Throws:

IOException - IO problems other than FileNotFound, which is downgraded to an object does not exist return code

rename

public void rename(org.apache.hadoop.fs.Path src,
                   org.apache.hadoop.fs.Path dst)
            throws FileNotFoundException,
                   SwiftOperationFailedException,
                   IOException

Rename through copy-and-delete. this is a consequence of the Swift filesystem using the path as the hash into the Distributed Hash Table, "the ring" of filenames.

Because of the nature of the operation, it is not atomic.

Parameters:

src - source file/dir

dst - destination

Throws:

IOException - IO failure

SwiftOperationFailedException - if the rename failed

FileNotFoundException - if the source directory is missing, or the parent directory of the destination

copy

public void copy(org.apache.hadoop.fs.Path srcKey,
                 org.apache.hadoop.fs.Path dstKey)
          throws IOException

Throws:

IOException

getCorrectSwiftPath

public org.apache.hadoop.fs.Path getCorrectSwiftPath(org.apache.hadoop.fs.Path path)
                                              throws SwiftException

Take a Hadoop path and return one which uses the URI prefix and authority of this FS. It doesn't make a relative path absolute

Parameters:

path - path in

Returns:

path with a URI bound to this FS

Throws:

SwiftException - URI cannot be created.

extractUris

public static List<URI> extractUris(String json,
                                    org.apache.hadoop.fs.Path path)
                             throws SwiftOperationFailedException

extracts URIs from json

Parameters:

json - json to parse

path - path (used in exceptions)

Returns:

URIs

Throws:

SwiftOperationFailedException - on any problem parsing the JSON

throttle

public void throttle()
              throws InterruptedIOException

Insert a throttled wait if the throttle delay > 0

Throws:

InterruptedIOException - if interrupted during sleep

getOperationStatistics

public List<DurationStats> getOperationStatistics()

Get the current operation statistics

Returns:

a snapshot of the statistics

delete

public boolean delete(org.apache.hadoop.fs.Path absolutePath,
                      boolean recursive)
               throws IOException

Delete the entire tree. This is an internal one with slightly different behavior: if an entry is missing, a FileNotFoundException is raised. This lets the caller distinguish a file not found with other reasons for failure, so handles race conditions in recursive directory deletes better.

The problem being addressed is: caller A requests a recursive directory of directory /dir ; caller B requests a delete of a file /dir/file, between caller A enumerating the files contents, and requesting a delete of /dir/file. We want to recognise the special case "directed file is no longer there" and not convert that into a failure

Parameters:

absolutePath - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception if the directory is not empty case of a file the recursive can be set to either true or false.

Returns:

true if the object was deleted

Throws:

IOException - IO problems

FileNotFoundException - if a file/dir being deleted is not there - this includes entries below the specified path, (if the path is a dir and recursive is true)