com.google.cloud.hadoop.gcsio

Class MetadataReadOnlyGoogleCloudStorage

java.lang.Object

com.google.cloud.hadoop.gcsio.MetadataReadOnlyGoogleCloudStorage

All Implemented Interfaces:

GoogleCloudStorage

public class MetadataReadOnlyGoogleCloudStorage
extends Object
implements GoogleCloudStorage

MetadataReadOnlyGoogleCloudStorage holds a collection of Storage object/bucket metadata entries and serves listObjectNames, listObjectInfo, getItemInfos, and getItemInfo exclusively from the in-memory cache. All other operations will throw an UnsupportedOperationException. For the time being, such unsupported operations include bucket info list operations like listBucketInfo and listBucketNames. This instance will supplement fake GoogleCloudStorageItemInfos for pure implicit directories if there is no metadata entry for the directory object itself. This means that some directory objects will list info that is inconsistent with an actual listing.

Field Summary

Fields

Modifier and Type	Field and Description
static LogUtil	log

Fields inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage

PATH_DELIMITER

Constructor Summary

Constructors

Constructor and Description
MetadataReadOnlyGoogleCloudStorage(Collection<GoogleCloudStorageItemInfo> itemInfos) Constructs a MetadataReadOnlyGoogleCloudStorage which can be used for temporary contexts where only object metadata read operations will be used through the GoogleCloudStorage interface.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Releases resources used by this instance.
void	copy(String srcBucketName, List<String> srcObjectNames, String dstBucketName, List<String> dstObjectNames) Copies metadata of the given objects.
WritableByteChannel	create(StorageResourceId resourceId) Creates and opens an object for writing.
WritableByteChannel	create(StorageResourceId resourceId, CreateObjectOptions options) Creates and opens an object for writing.
void	create(String bucketName) Creates a bucket.
void	createEmptyObject(StorageResourceId resourceId) Creates an empty object, useful for placeholders representing, for example, directories.
void	createEmptyObject(StorageResourceId resourceId, CreateObjectOptions options) Creates an empty object, useful for placeholders representing, for example, directories.
void	createEmptyObjects(List<StorageResourceId> resourceIds) Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method.
void	createEmptyObjects(List<StorageResourceId> resourceIds, CreateObjectOptions options) Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method.
void	deleteBuckets(List<String> bucketNames) Deletes a list of buckets.
void	deleteObjects(List<StorageResourceId> fullObjectNames) Deletes the given objects.
GoogleCloudStorageItemInfo	getItemInfo(StorageResourceId resourceId) Pure fetch from cache.
List<GoogleCloudStorageItemInfo>	getItemInfos(List<StorageResourceId> resourceIds) Pure fetch from cache.
List<GoogleCloudStorageItemInfo>	listBucketInfo() Gets a list of GoogleCloudStorageItemInfo for all buckets of this project.
List<String>	listBucketNames() Gets a list of names of buckets in this project.
List<GoogleCloudStorageItemInfo>	listObjectInfo(String bucketName, String objectNamePrefix, String delimiter) Uses shared prefix-matching logic to filter entries from the metadata cache.
List<String>	listObjectNames(String bucketName, String objectNamePrefix, String delimiter) Gets names of objects contained in the given bucket and whose names begin with the given prefix.
SeekableReadableByteChannel	open(StorageResourceId resourceId) Opens an object for reading.
List<GoogleCloudStorageItemInfo>	updateItems(List<UpdatableItemInfo> itemInfoList) Attempt to update metadata of the objects referenced within the passed itemInfo objects.
void	waitForBucketEmpty(String bucketName) Waits for the given bucket to be empty.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

public static final LogUtil log

Constructor Detail

MetadataReadOnlyGoogleCloudStorage

public MetadataReadOnlyGoogleCloudStorage(Collection<GoogleCloudStorageItemInfo> itemInfos)
                                   throws IOException

Constructs a MetadataReadOnlyGoogleCloudStorage which can be used for temporary contexts where only object metadata read operations will be used through the GoogleCloudStorage interface.

Parameters:

itemInfos - The collection of item infos with which to serve all list/get object requests.

Throws:

IOException

Method Detail

create

public WritableByteChannel create(StorageResourceId resourceId)
                           throws IOException

Description copied from interface: GoogleCloudStorage

Creates and opens an object for writing. The bucket must already exist. If the object already exists, it is deleted.

Specified by:

create in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

create

public WritableByteChannel create(StorageResourceId resourceId,
                         CreateObjectOptions options)
                           throws IOException

Description copied from interface: GoogleCloudStorage

Creates and opens an object for writing. The bucket must already exist.

Specified by:

create in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - Options to use when creating the object

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId)
                       throws IOException

Description copied from interface: GoogleCloudStorage

Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId,
                     CreateObjectOptions options)
                       throws IOException

Description copied from interface: GoogleCloudStorage

Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - options to use when creating the object

Throws:

IOException - on IO error

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds)
                        throws IOException

Description copied from interface: GoogleCloudStorage

Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method. Implementations may use different flow than the single-item version for greater efficiency.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds,
                      CreateObjectOptions options)
                        throws IOException

Description copied from interface: GoogleCloudStorage

Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method. Implementations may use different flow than the single-item version for greater efficiency.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

open

public SeekableReadableByteChannel open(StorageResourceId resourceId)
                                 throws IOException

Description copied from interface: GoogleCloudStorage

Opens an object for reading.

Specified by:

open in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Returns:

a channel for reading from the given object

Throws:

IOException - if object exists but cannot be opened

create

public void create(String bucketName)
            throws IOException

Description copied from interface: GoogleCloudStorage

Creates a bucket.

Specified by:

create in interface GoogleCloudStorage

Parameters:

bucketName - name of the bucket to create

Throws:

IOException - on IO error

deleteBuckets

public void deleteBuckets(List<String> bucketNames)
                   throws IOException

Description copied from interface: GoogleCloudStorage

Deletes a list of buckets. Does not throw any exception for "bucket not found" errors.

Specified by:

deleteBuckets in interface GoogleCloudStorage

Parameters:

bucketNames - name of the buckets to delete

Throws:

IOException - on IO error

deleteObjects

public void deleteObjects(List<StorageResourceId> fullObjectNames)
                   throws IOException

Description copied from interface: GoogleCloudStorage

Deletes the given objects. Does not throw any exception for "object not found" errors.

Specified by:

deleteObjects in interface GoogleCloudStorage

Parameters:

fullObjectNames - names of objects to delete with their respective bucketNames.

Throws:

IOException - if object exists but cannot be deleted

copy

public void copy(String srcBucketName,
        List<String> srcObjectNames,
        String dstBucketName,
        List<String> dstObjectNames)
          throws IOException

Description copied from interface: GoogleCloudStorage

Copies metadata of the given objects. After the copy is successfully complete, each object blob is reachable by two different names. Copying between two different locations or between two different storage classes is not allowed.

Specified by:

copy in interface GoogleCloudStorage

Parameters:

srcBucketName - name of the bucket containing the objects to copy

srcObjectNames - names of the objects to copy

dstBucketName - name of the bucket to copy to

dstObjectNames - names of the objects after copy

Throws:

IOException - in all other error cases

listBucketNames

public List<String> listBucketNames()
                             throws IOException

Description copied from interface: GoogleCloudStorage

Gets a list of names of buckets in this project.

Specified by:

listBucketNames in interface GoogleCloudStorage

Throws:

IOException

listBucketInfo

public List<GoogleCloudStorageItemInfo> listBucketInfo()
                                                throws IOException

Description copied from interface: GoogleCloudStorage

Gets a list of GoogleCloudStorageItemInfo for all buckets of this project. This is no more expensive than calling listBucketNames(), since the list API for buckets already retrieves all the relevant bucket metadata.

Specified by:

listBucketInfo in interface GoogleCloudStorage

Throws:

IOException

listObjectNames

public List<String> listObjectNames(String bucketName,
                           String objectNamePrefix,
                           String delimiter)
                             throws IOException

Description copied from interface: GoogleCloudStorage

Gets names of objects contained in the given bucket and whose names begin with the given prefix. Note: Although GCS does not implement a file system, it treats objects that contain a delimiter as different from other objects when listing objects. This will be clearer with an example. Consider a bucket with objects: o1, d1/, d1/o1, d1/o2 With prefix == null and delimiter == /, we get: d1/, o1 With prefix == null and delimiter == null, we get: o1, d1/, d1/o1, d1/o2 Thus when delimiter is null, the entire key name is considered an opaque string, otherwise only the part up to the first delimiter is considered.

Specified by:

listObjectNames in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

delimiter - delimiter to use (typically "/"), otherwise null

Returns:

list of object names

Throws:

IOException - on IO error

listObjectInfo

public List<GoogleCloudStorageItemInfo> listObjectInfo(String bucketName,
                                              String objectNamePrefix,
                                              String delimiter)
                                                throws IOException

Uses shared prefix-matching logic to filter entries from the metadata cache. For implicit prefix matches with no corresponding real directory object, adds a fake directory object with creationTime == 0.

Specified by:

listObjectInfo in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

delimiter - delimiter to use (typically "/"), otherwise null

Returns:

list of object info

Throws:

IOException - on IO error

getItemInfos

public List<GoogleCloudStorageItemInfo> getItemInfos(List<StorageResourceId> resourceIds)
                                              throws IOException

Pure fetch from cache.

Specified by:

getItemInfos in interface GoogleCloudStorage

Parameters:

resourceIds - names of the GCS StorageObjects or Buckets for which to retrieve info.

Returns:

information about the given resourceIds.

Throws:

IOException - on IO error

updateItems

public List<GoogleCloudStorageItemInfo> updateItems(List<UpdatableItemInfo> itemInfoList)
                                             throws IOException

Description copied from interface: GoogleCloudStorage

Attempt to update metadata of the objects referenced within the passed itemInfo objects.

Specified by:

updateItems in interface GoogleCloudStorage

Returns:

Updated GoogleCloudStorageItemInfo objects for the referenced objects.

Throws:

IOException - on IO error

getItemInfo

public GoogleCloudStorageItemInfo getItemInfo(StorageResourceId resourceId)
                                       throws IOException

Pure fetch from cache.

Specified by:

getItemInfo in interface GoogleCloudStorage

Parameters:

resourceId - identifies either root, a Bucket, or a StorageObject

Returns:

information about the given item

Throws:

IOException - on IO error

close

public void close()

Description copied from interface: GoogleCloudStorage

Releases resources used by this instance.

Specified by:

close in interface GoogleCloudStorage

waitForBucketEmpty

public void waitForBucketEmpty(String bucketName)
                        throws IOException

Description copied from interface: GoogleCloudStorage

Waits for the given bucket to be empty. Note: GCS only supports eventual consistency of object lists. When a user deletes a top-level directory recursively, the fact that all items have gone away is not reflected instantly. We retry and wait for that to happen.

Specified by:

waitForBucketEmpty in interface GoogleCloudStorage

Throws:

IOException