com.google.cloud.hadoop.gcsio

Class InMemoryDirectoryListCache

java.lang.Object

com.google.cloud.hadoop.gcsio.DirectoryListCache

com.google.cloud.hadoop.gcsio.InMemoryDirectoryListCache

public class InMemoryDirectoryListCache
extends DirectoryListCache

InMemoryDirectoryListCache provides in-memory accounting of full paths for directories and files created and deleted in GoogleCloudStorageFileSystem within a single running process.

This in-memory layer guarantees same-client consistency between file-creation and a subsequent "list" request. Same-client deletes will remove associated entries from the cache, so that same-client create-followed-by-delete will not erroneously supplement a nonexistent object. List-after-delete consistency is implemented by blacklisting associated entries. Cache staleness in the face of cross-client operations is mitigated by the fact that the cached GoogleCloudStorageItemInfo can be configured to expire with a shorter lifespan than the cache entry itself; when the GoogleCloudStorageItemInfo is fetched lazily, 404s allow pre-emptively removing stale deleted CacheEntries, and likewise, non-404s allow removing an incorrect cache-blacklist entry.

The tradeoff of using this cache is that cross-client deletes may take longer to become visible, if the associated GoogleCloudStorageItemInfo is still active in the cache after another client deletes the associated object. To summarize:

1. Same-client 'create' followed by 'list' will be immediately consistent. 2. Cross-client 'delete' followed by 'list' will be worse than pure-GCS. 3. Same-client 'delete' followed by 'list' is unchanged with respect to eventual consistency.

This class is thread-safe.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.hadoop.gcsio.DirectoryListCache

DirectoryListCache.Config, DirectoryListCache.Type

Field Summary

Fields inherited from class com.google.cloud.hadoop.gcsio.DirectoryListCache

cacheConfig, clock

Constructor Summary

Constructors

Constructor and Description
InMemoryDirectoryListCache() Callers should usually only obtain an instance via getInstance() so that cache info is shared process-wide, but instances can be created for temporary caches.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	containsEntriesForImplicitDirectories() We don't inspect StorageResourceIds in putResourceId to auto-insert entries for parent directories, nor do we return fake entries in getObjectList when a delimiter implies a pure-prefix match.
List<CacheEntry>	getBucketList()
CacheEntry	getCacheEntry(StorageResourceId resourceId) Returns the CacheEntry associated with resourceId, or null if it doesn't exist.
static DirectoryListCache	getInstance() Accessor for shared singleton instance of DirectoryListCache.
int	getInternalNumBuckets() Gets the internal number of CachedBucket entries, which may not be equal to the size of getBucketList() if there are expired entries.
int	getInternalNumObjects() Gets the internal total count of cached StorageObject entries.
List<CacheEntry>	getObjectList(String bucketName, String objectNamePrefix, String delimiter, Set<String> returnedPrefixes)
List<CacheEntry>	getRawBucketList()
CacheEntry	putResourceId(StorageResourceId resourceId) Adds the names of the Bucket or StorageObject referenced by resourceId to the cache, with no attached metadata.
void	removeResourceId(StorageResourceId resourceId) Removes CacheEntry associated with resourceId, if it exists.
boolean	supportsCacheEntryByReference() We use in-memory data structures to hold CacheEntry items, and thus manage them in a shared manner; returned CacheEntry items are shared references, and updating their cached info effectively updates the entry's info for all users of the cache.

Methods inherited from class com.google.cloud.hadoop.gcsio.DirectoryListCache

getMutableConfig, isCacheEntryExpired, maybeInvalidateExpiredInfo, setClock, validateResourceId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InMemoryDirectoryListCache

public InMemoryDirectoryListCache()

Callers should usually only obtain an instance via getInstance() so that cache info is shared process-wide, but instances can be created for temporary caches.

Method Detail

getInstance

public static DirectoryListCache getInstance()

Accessor for shared singleton instance of DirectoryListCache.

supportsCacheEntryByReference

public boolean supportsCacheEntryByReference()

We use in-memory data structures to hold CacheEntry items, and thus manage them in a shared manner; returned CacheEntry items are shared references, and updating their cached info effectively updates the entry's info for all users of the cache.

Specified by:

supportsCacheEntryByReference in class DirectoryListCache

containsEntriesForImplicitDirectories

public boolean containsEntriesForImplicitDirectories()

We don't inspect StorageResourceIds in putResourceId to auto-insert entries for parent directories, nor do we return fake entries in getObjectList when a delimiter implies a pure-prefix match.

Specified by:

containsEntriesForImplicitDirectories in class DirectoryListCache

putResourceId

public CacheEntry putResourceId(StorageResourceId resourceId)

Description copied from class: DirectoryListCache

Adds the names of the Bucket or StorageObject referenced by resourceId to the cache, with no attached metadata. If the entry already exists, then nothing is modified. If resourceId is a StorageObject, the parent Bucket name is also added to the cache, if it doesn't already exist. TODO(user): Even if the entry exists, it might be correct to invalidate any existing metadata and force a refresh next time it is fetched.

Specified by:

putResourceId in class DirectoryListCache

Returns:

The CacheEntry corresponding to the item added.

getCacheEntry

public CacheEntry getCacheEntry(StorageResourceId resourceId)

Description copied from class: DirectoryListCache

Returns the CacheEntry associated with resourceId, or null if it doesn't exist. This returns the real mutable CacheEntry (rather than a copy of the data) so that the caller may efficiently update the info stored in the CacheEntry if necessary.

Specified by:

getCacheEntry in class DirectoryListCache

removeResourceId

public void removeResourceId(StorageResourceId resourceId)

Description copied from class: DirectoryListCache

Removes CacheEntry associated with resourceId, if it exists. Cached GoogleCloudStorageItemInfo associated with the resourceId is also removed, if it exists. If resourceId denotes a non-empty bucket, then all the cached StorageObject children of that bucket will also be removed from the cache; it is the caller's responsibility to ensure that the bucket should really be removed. Note that normal expiration of a CachedBucket will *not* remove the actual CachedBucket, even though the bucketName will stop appearing in calls to getBucketList().

Specified by:

removeResourceId in class DirectoryListCache

getBucketList

public List<CacheEntry> getBucketList()

Specified by:

getBucketList in class DirectoryListCache

Returns:

List of CacheEntry corresponding to Buckets known by this cache; includes buckets added to the cache automatically when caching a StorageObject contained in the corresponding bucket. Will not return null. Hides/removes expired bucket entries and clears any expired GoogleCloudStorageItemInfo associated with buckets.

getRawBucketList

public List<CacheEntry> getRawBucketList()

Specified by:

getRawBucketList in class DirectoryListCache

Returns:

List of *all* Bucket CacheEntries, including ones which might be expired. Doesn't actively invalidate, set expiration, or delete expired entries. Can be used when the internal bucket list is needed without wanting to cause any mutations in the cache.

getObjectList

public List<CacheEntry> getObjectList(String bucketName,
                             String objectNamePrefix,
                             String delimiter,
                             Set<String> returnedPrefixes)

Specified by:

getObjectList in class DirectoryListCache

Parameters:

bucketName - The bucket inside of which to list objects.

objectNamePrefix - The prefix to be used to match object names to return.

delimiter - The character for specifying 'directory' boundaries, or null.

returnedPrefixes - A container to be populated with implied "directory objects" that come from some object which includes the prefix, followed by some string, then followed by the 'delimiter'. This may or may not be a duplicate of one of the actual returned directory objects. For example, if gs://foo/bar/baz.txt exists and we query with parameters ("foo", "ba", "/", {}) then the returnedPrefixes will be populated with the string "bar/" by virtue of existence of the "bar/baz.txt" file. May be null if the caller doesn't desire fetching such returnedPrefixes.

Returns:

List of CacheEntrys for StorageObjects residing in bucket bucketName which match the provided objectNamePrefix and delimiter, or possibly null if no such objects are present. May also return an empty list.

getInternalNumBuckets

public int getInternalNumBuckets()

Description copied from class: DirectoryListCache

Gets the internal number of CachedBucket entries, which may not be equal to the size of getBucketList() if there are expired entries. Does not mutate the cache.

Specified by:

getInternalNumBuckets in class DirectoryListCache

getInternalNumObjects

public int getInternalNumObjects()

Description copied from class: DirectoryListCache

Gets the internal total count of cached StorageObject entries. Does not mutate the cache.

Specified by:

getInternalNumObjects in class DirectoryListCache