com.google.cloud.hadoop.gcsio

Class GoogleCloudStorageImpl

java.lang.Object

com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl

All Implemented Interfaces:

GoogleCloudStorage

public class GoogleCloudStorageImpl
extends Object
implements GoogleCloudStorage

Provides read/write access to Google Cloud Storage (GCS), using Java nio channel semantics. This is a basic implementation of the GoogleCloudStorage interface which mostly delegates through to the appropriate API call(s) via the generated JSON API client, while adding reliability and performance features such as setting up low-level retries, translating low-level exceptions, and request batching.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	GoogleCloudStorageImpl.BackOffFactory A factory for producing BackOff objects.

Field Summary

Fields

Modifier and Type	Field and Description
static int	BUCKET_EMPTY_MAX_RETRIES
static int	BUCKET_EMPTY_WAIT_TIME_MS
static String	PATH_DELIMITER

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	GoogleCloudStorageImpl()
	GoogleCloudStorageImpl(GoogleCloudStorageOptions options, com.google.api.client.auth.oauth2.Credential credential) Constructs an instance of GoogleCloudStorageImpl.
	GoogleCloudStorageImpl(GoogleCloudStorageOptions options, com.google.api.services.storage.Storage gcs) Constructs an instance of GoogleCloudStorageImpl.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() See GoogleCloudStorage.close() for details about expected behavior.
void	copy(String srcBucketName, List<String> srcObjectNames, String dstBucketName, List<String> dstObjectNames) See GoogleCloudStorage.copy(String, List, String, List) for details about expected behavior.
WritableByteChannel	create(StorageResourceId resourceId) See GoogleCloudStorage.create(StorageResourceId) for details about expected behavior.
WritableByteChannel	create(StorageResourceId resourceId, CreateObjectOptions options) Creates and opens an object for writing.
void	create(String bucketName) See GoogleCloudStorage.create(String) for details about expected behavior.
void	createEmptyObject(StorageResourceId resourceId) See GoogleCloudStorage.createEmptyObject(StorageResourceId) for details about expected behavior.
void	createEmptyObject(StorageResourceId resourceId, CreateObjectOptions options) Creates an empty object, useful for placeholders representing, for example, directories.
void	createEmptyObjects(List<StorageResourceId> resourceIds) See GoogleCloudStorage.createEmptyObjects(List) for details about expected behavior.
void	createEmptyObjects(List<StorageResourceId> resourceIds, CreateObjectOptions options) Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method.
void	deleteBuckets(List<String> bucketNames) See GoogleCloudStorage#deleteBuckets(List) for details about expected behavior.
void	deleteObjects(List<StorageResourceId> fullObjectNames) See GoogleCloudStorage#deleteObjects(List) for details about expected behavior.
GoogleCloudStorageItemInfo	getItemInfo(StorageResourceId resourceId) See GoogleCloudStorage.getItemInfo(StorageResourceId) for details about expected behavior.
List<GoogleCloudStorageItemInfo>	getItemInfos(List<StorageResourceId> resourceIds) See GoogleCloudStorage#getItemInfos(List) for details about expected behavior.
List<GoogleCloudStorageItemInfo>	listBucketInfo() See GoogleCloudStorage.listBucketInfo() for details about expected behavior.
List<String>	listBucketNames() See GoogleCloudStorage.listBucketNames() for details about expected behavior.
List<GoogleCloudStorageItemInfo>	listObjectInfo(String bucketName, String objectNamePrefix, String delimiter) See GoogleCloudStorage.listObjectInfo(String, String, String) for details about expected behavior.
List<String>	listObjectNames(String bucketName, String objectNamePrefix, String delimiter) See GoogleCloudStorage.listObjectNames(String, String, String) for details about expected behavior.
SeekableReadableByteChannel	open(StorageResourceId resourceId) See GoogleCloudStorage.open(StorageResourceId) for details about expected behavior.
List<GoogleCloudStorageItemInfo>	updateItems(List<UpdatableItemInfo> itemInfoList) Attempt to update metadata of the objects referenced within the passed itemInfo objects.
void	waitForBucketEmpty(String bucketName) See GoogleCloudStorage.waitForBucketEmpty(String) for details about expected behavior.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATH_DELIMITER

public static final String PATH_DELIMITER

See Also:

Constant Field Values

BUCKET_EMPTY_MAX_RETRIES

public static final int BUCKET_EMPTY_MAX_RETRIES

See Also:

Constant Field Values

BUCKET_EMPTY_WAIT_TIME_MS

public static final int BUCKET_EMPTY_WAIT_TIME_MS

See Also:

Constant Field Values

Constructor Detail

GoogleCloudStorageImpl

public GoogleCloudStorageImpl(GoogleCloudStorageOptions options,
                      com.google.api.client.auth.oauth2.Credential credential)
                       throws IOException

Constructs an instance of GoogleCloudStorageImpl.

Parameters:

credential - OAuth2 credential that allows access to GCS

Throws:

IOException - on IO error

GoogleCloudStorageImpl

public GoogleCloudStorageImpl(GoogleCloudStorageOptions options,
                      com.google.api.services.storage.Storage gcs)

Constructs an instance of GoogleCloudStorageImpl.

Parameters:

gcs - Preconstructed Storage to use for I/O.

GoogleCloudStorageImpl

protected GoogleCloudStorageImpl()

Method Detail

create

public WritableByteChannel create(StorageResourceId resourceId,
                         CreateObjectOptions options)
                           throws IOException

Description copied from interface: GoogleCloudStorage

Creates and opens an object for writing. The bucket must already exist.

Specified by:

create in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - Options to use when creating the object

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

create

public WritableByteChannel create(StorageResourceId resourceId)
                           throws IOException

See GoogleCloudStorage.create(StorageResourceId) for details about expected behavior.

Specified by:

create in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId,
                     CreateObjectOptions options)
                       throws IOException

Description copied from interface: GoogleCloudStorage

Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - options to use when creating the object

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId)
                       throws IOException

See GoogleCloudStorage.createEmptyObject(StorageResourceId) for details about expected behavior.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Throws:

IOException - on IO error

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds,
                      CreateObjectOptions options)
                        throws IOException

Description copied from interface: GoogleCloudStorage

Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method. Implementations may use different flow than the single-item version for greater efficiency.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds)
                        throws IOException

See GoogleCloudStorage.createEmptyObjects(List) for details about expected behavior.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

open

public SeekableReadableByteChannel open(StorageResourceId resourceId)
                                 throws IOException

See GoogleCloudStorage.open(StorageResourceId) for details about expected behavior.

Specified by:

open in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Returns:

a channel for reading from the given object

Throws:

IOException - if object exists but cannot be opened

create

public void create(String bucketName)
            throws IOException

See GoogleCloudStorage.create(String) for details about expected behavior.

Specified by:

create in interface GoogleCloudStorage

Parameters:

bucketName - name of the bucket to create

Throws:

IOException - on IO error

deleteBuckets

public void deleteBuckets(List<String> bucketNames)
                   throws IOException

See GoogleCloudStorage#deleteBuckets(List) for details about expected behavior.

Specified by:

deleteBuckets in interface GoogleCloudStorage

Parameters:

bucketNames - name of the buckets to delete

Throws:

IOException - on IO error

deleteObjects

public void deleteObjects(List<StorageResourceId> fullObjectNames)
                   throws IOException

See GoogleCloudStorage#deleteObjects(List) for details about expected behavior.

Specified by:

deleteObjects in interface GoogleCloudStorage

Parameters:

fullObjectNames - names of objects to delete with their respective bucketNames.

Throws:

IOException - if object exists but cannot be deleted

copy

public void copy(String srcBucketName,
        List<String> srcObjectNames,
        String dstBucketName,
        List<String> dstObjectNames)
          throws IOException

See GoogleCloudStorage.copy(String, List, String, List) for details about expected behavior.

Specified by:

copy in interface GoogleCloudStorage

Parameters:

srcBucketName - name of the bucket containing the objects to copy

srcObjectNames - names of the objects to copy

dstBucketName - name of the bucket to copy to

dstObjectNames - names of the objects after copy

Throws:

IOException - in all other error cases

listBucketNames

public List<String> listBucketNames()
                             throws IOException

See GoogleCloudStorage.listBucketNames() for details about expected behavior.

Specified by:

listBucketNames in interface GoogleCloudStorage

Throws:

IOException

listBucketInfo

public List<GoogleCloudStorageItemInfo> listBucketInfo()
                                                throws IOException

See GoogleCloudStorage.listBucketInfo() for details about expected behavior.

Specified by:

listBucketInfo in interface GoogleCloudStorage

Throws:

IOException

listObjectNames

public List<String> listObjectNames(String bucketName,
                           String objectNamePrefix,
                           String delimiter)
                             throws IOException

See GoogleCloudStorage.listObjectNames(String, String, String) for details about expected behavior.

Specified by:

listObjectNames in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

delimiter - delimiter to use (typically "/"), otherwise null

Returns:

list of object names

Throws:

IOException - on IO error

listObjectInfo

public List<GoogleCloudStorageItemInfo> listObjectInfo(String bucketName,
                                              String objectNamePrefix,
                                              String delimiter)
                                                throws IOException

See GoogleCloudStorage.listObjectInfo(String, String, String) for details about expected behavior.

Specified by:

listObjectInfo in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

delimiter - delimiter to use (typically "/"), otherwise null

Returns:

list of object info

Throws:

IOException - on IO error

getItemInfos

public List<GoogleCloudStorageItemInfo> getItemInfos(List<StorageResourceId> resourceIds)
                                              throws IOException

See GoogleCloudStorage#getItemInfos(List) for details about expected behavior.

Specified by:

getItemInfos in interface GoogleCloudStorage

Parameters:

resourceIds - names of the GCS StorageObjects or Buckets for which to retrieve info.

Returns:

information about the given resourceIds.

Throws:

IOException - on IO error

updateItems

public List<GoogleCloudStorageItemInfo> updateItems(List<UpdatableItemInfo> itemInfoList)
                                             throws IOException

Description copied from interface: GoogleCloudStorage

Attempt to update metadata of the objects referenced within the passed itemInfo objects.

Specified by:

updateItems in interface GoogleCloudStorage

Returns:

Updated GoogleCloudStorageItemInfo objects for the referenced objects.

Throws:

IOException - on IO error

getItemInfo

public GoogleCloudStorageItemInfo getItemInfo(StorageResourceId resourceId)
                                       throws IOException

See GoogleCloudStorage.getItemInfo(StorageResourceId) for details about expected behavior.

Specified by:

getItemInfo in interface GoogleCloudStorage

Parameters:

resourceId - identifies either root, a Bucket, or a StorageObject

Returns:

information about the given item

Throws:

IOException - on IO error

close

public void close()

See GoogleCloudStorage.close() for details about expected behavior.

Specified by:

close in interface GoogleCloudStorage

waitForBucketEmpty

public void waitForBucketEmpty(String bucketName)
                        throws IOException

See GoogleCloudStorage.waitForBucketEmpty(String) for details about expected behavior.

Specified by:

waitForBucketEmpty in interface GoogleCloudStorage

Throws:

IOException