com.google.cloud.hadoop.gcsio

Class CacheSupplementedGoogleCloudStorage

java.lang.Object

com.google.cloud.hadoop.gcsio.CacheSupplementedGoogleCloudStorage

All Implemented Interfaces:

GoogleCloudStorage

public class CacheSupplementedGoogleCloudStorage
extends Object
implements GoogleCloudStorage

CacheSupplementedGoogleCloudStorage adds additional book-keeping to a GoogleCloudStorage instance using a DirectoryListCache and wraps the create/copy/delete/list methods to provide immediate same-client consistency for "list" operations following a "create/copy/delete". See DirectoryListCache for details of consistency semantics.

Field Summary

Fields inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage

PATH_DELIMITER

Constructor Summary

Constructors

Constructor and Description

CacheSupplementedGoogleCloudStorage(GoogleCloudStorage gcsDelegate, DirectoryListCache resourceCache) Constructs a CacheSupplementedGoogleCloudStorage which should be usable anywhere a GoogleCloudStorage interface is used and which supplements missing listObject/listBucket results from an in-memory cache of known GCS resources which may not have propagated into the eventually-consistent remote "list" index yet.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Pure pass-through.
void	copy(String srcBucketName, List<String> srcObjectNames, String dstBucketName, List<String> dstObjectNames) Adds the copied destination items to the list cache, without their associated metadata; supplementing with the cache will have to populate the metadata on-demand.
WritableByteChannel	create(StorageResourceId resourceId) Wraps the delegate's returned WritableByteChannel in a helper which will update the resourceCache when close() is called.
WritableByteChannel	create(StorageResourceId resourceId, CreateObjectOptions options) Creates and opens an object for writing.
void	create(String bucketName) Updates cache with bucketName.
void	createEmptyObject(StorageResourceId resourceId) Records the resourceId after delegating.
void	createEmptyObject(StorageResourceId resourceId, CreateObjectOptions options) Creates an empty object, useful for placeholders representing, for example, directories.
void	createEmptyObjects(List<StorageResourceId> resourceIds) Records the resourceIds after delegating.
void	createEmptyObjects(List<StorageResourceId> resourceIds, CreateObjectOptions options) Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method.
void	deleteBuckets(List<String> bucketNames) Removes buckets from cache, if they exist.
void	deleteObjects(List<StorageResourceId> fullObjectNames) Removes objects from cache, if they exist.
GoogleCloudStorageItemInfo	getItemInfo(StorageResourceId resourceId) Pure pass-through.
List<GoogleCloudStorageItemInfo>	getItemInfos(List<StorageResourceId> resourceIds) Pure pass-through.
List<GoogleCloudStorageItemInfo>	listBucketInfo() Supplements the list returned by the delegate with cached bucket infos; may trigger fetching of any metadata not already available in the cache.
List<String>	listBucketNames() Supplements the list returned by the delegate with cached bucket names; won't trigger any fetching of metadata.
List<GoogleCloudStorageItemInfo>	listObjectInfo(String bucketName, String objectNamePrefix, String delimiter) Supplements the list returned by the delegate with cached object infos; may trigger fetching of any metadata not already available in the cache.
List<String>	listObjectNames(String bucketName, String objectNamePrefix, String delimiter) Supplements the list returned by the delegate with cached object names; won't trigger any fetching of metadata.
SeekableReadableByteChannel	open(StorageResourceId resourceId) Pure pass-through.
List<GoogleCloudStorageItemInfo>	updateItems(List<UpdatableItemInfo> itemInfoList) Attempt to update metadata of the objects referenced within the passed itemInfo objects.
void	waitForBucketEmpty(String bucketName) Pure pass-through.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CacheSupplementedGoogleCloudStorage

public CacheSupplementedGoogleCloudStorage(GoogleCloudStorage gcsDelegate,
                                   DirectoryListCache resourceCache)

Constructs a CacheSupplementedGoogleCloudStorage which should be usable anywhere a GoogleCloudStorage interface is used and which supplements missing listObject/listBucket results from an in-memory cache of known GCS resources which may not have propagated into the eventually-consistent remote "list" index yet.

Parameters:

gcsDelegate - The GoogleCloudStorage to be used for normal API interactions, before supplementing with in-memory info.

Method Detail

create

public WritableByteChannel create(StorageResourceId resourceId)
                           throws IOException

Wraps the delegate's returned WritableByteChannel in a helper which will update the resourceCache when close() is called.

Specified by:

create in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

create

public WritableByteChannel create(StorageResourceId resourceId,
                         CreateObjectOptions options)
                           throws IOException

Description copied from interface: GoogleCloudStorage

Creates and opens an object for writing. The bucket must already exist.

Specified by:

create in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - Options to use when creating the object

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId)
                       throws IOException

Records the resourceId after delegating.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId,
                     CreateObjectOptions options)
                       throws IOException

Description copied from interface: GoogleCloudStorage

Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - options to use when creating the object

Throws:

IOException - on IO error

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds)
                        throws IOException

Records the resourceIds after delegating.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds,
                      CreateObjectOptions options)
                        throws IOException

Description copied from interface: GoogleCloudStorage

Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method. Implementations may use different flow than the single-item version for greater efficiency.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

open

public SeekableReadableByteChannel open(StorageResourceId resourceId)
                                 throws IOException

Pure pass-through.

Specified by:

open in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Returns:

a channel for reading from the given object

Throws:

IOException - if object exists but cannot be opened

create

public void create(String bucketName)
            throws IOException

Updates cache with bucketName.

Specified by:

create in interface GoogleCloudStorage

Parameters:

bucketName - name of the bucket to create

Throws:

IOException - on IO error

deleteBuckets

public void deleteBuckets(List<String> bucketNames)
                   throws IOException

Removes buckets from cache, if they exist.

Specified by:

deleteBuckets in interface GoogleCloudStorage

Parameters:

bucketNames - name of the buckets to delete

Throws:

IOException - on IO error

deleteObjects

public void deleteObjects(List<StorageResourceId> fullObjectNames)
                   throws IOException

Removes objects from cache, if they exist.

Specified by:

deleteObjects in interface GoogleCloudStorage

Parameters:

fullObjectNames - names of objects to delete with their respective bucketNames.

Throws:

IOException - if object exists but cannot be deleted

copy

public void copy(String srcBucketName,
        List<String> srcObjectNames,
        String dstBucketName,
        List<String> dstObjectNames)
          throws IOException

Adds the copied destination items to the list cache, without their associated metadata; supplementing with the cache will have to populate the metadata on-demand.

Specified by:

copy in interface GoogleCloudStorage

Parameters:

srcBucketName - name of the bucket containing the objects to copy

srcObjectNames - names of the objects to copy

dstBucketName - name of the bucket to copy to

dstObjectNames - names of the objects after copy

Throws:

IOException - in all other error cases

listBucketNames

public List<String> listBucketNames()
                             throws IOException

Supplements the list returned by the delegate with cached bucket names; won't trigger any fetching of metadata.

Specified by:

listBucketNames in interface GoogleCloudStorage

Throws:

IOException

listBucketInfo

public List<GoogleCloudStorageItemInfo> listBucketInfo()
                                                throws IOException

Supplements the list returned by the delegate with cached bucket infos; may trigger fetching of any metadata not already available in the cache. If a delegate-returned item is also in the cache and the cache doesn't already have the metadata, it will be opportunistically updated with the retrieved metadata.

Specified by:

listBucketInfo in interface GoogleCloudStorage

Throws:

IOException

listObjectNames

public List<String> listObjectNames(String bucketName,
                           String objectNamePrefix,
                           String delimiter)
                             throws IOException

Supplements the list returned by the delegate with cached object names; won't trigger any fetching of metadata.

Specified by:

listObjectNames in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

delimiter - delimiter to use (typically "/"), otherwise null

Returns:

list of object names

Throws:

IOException - on IO error

listObjectInfo

public List<GoogleCloudStorageItemInfo> listObjectInfo(String bucketName,
                                              String objectNamePrefix,
                                              String delimiter)
                                                throws IOException

Supplements the list returned by the delegate with cached object infos; may trigger fetching of any metadata not already available in the cache. If a delegate-returned item is also in the cache and the cache doesn't already have the metadata, it will be opportunistically updated with the retrieved metadata.

Specified by:

listObjectInfo in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

delimiter - delimiter to use (typically "/"), otherwise null

Returns:

list of object info

Throws:

IOException - on IO error

getItemInfos

public List<GoogleCloudStorageItemInfo> getItemInfos(List<StorageResourceId> resourceIds)
                                              throws IOException

Pure pass-through.

Specified by:

getItemInfos in interface GoogleCloudStorage

Parameters:

resourceIds - names of the GCS StorageObjects or Buckets for which to retrieve info.

Returns:

information about the given resourceIds.

Throws:

IOException - on IO error

updateItems

public List<GoogleCloudStorageItemInfo> updateItems(List<UpdatableItemInfo> itemInfoList)
                                             throws IOException

Description copied from interface: GoogleCloudStorage

Attempt to update metadata of the objects referenced within the passed itemInfo objects.

Specified by:

updateItems in interface GoogleCloudStorage

Returns:

Updated GoogleCloudStorageItemInfo objects for the referenced objects.

Throws:

IOException - on IO error

getItemInfo

public GoogleCloudStorageItemInfo getItemInfo(StorageResourceId resourceId)
                                       throws IOException

Pure pass-through.

Specified by:

getItemInfo in interface GoogleCloudStorage

Parameters:

resourceId - identifies either root, a Bucket, or a StorageObject

Returns:

information about the given item

Throws:

IOException - on IO error

close

public void close()

Pure pass-through.

Specified by:

close in interface GoogleCloudStorage

waitForBucketEmpty

public void waitForBucketEmpty(String bucketName)
                        throws IOException

Pure pass-through.

Specified by:

waitForBucketEmpty in interface GoogleCloudStorage

Throws:

IOException