com.google.cloud.hadoop.gcsio

Class FileSystemBackedDirectoryListCache

java.lang.Object

com.google.cloud.hadoop.gcsio.DirectoryListCache

com.google.cloud.hadoop.gcsio.FileSystemBackedDirectoryListCache

public class FileSystemBackedDirectoryListCache
extends DirectoryListCache

FileSystemBackedDirectoryListCache mirrors all GCS directory and file creation into a matching tree of directories + empty files inside a configured local filesystem, as accessed via java.nio.Path and java.io.File, all under a base directory configured at construction-time. If a regular local filesystem path is used, then this cache thus enables immediate list consistency for all processes which use cooperating GCS-connector classes on the same machine; alternatively, if the path is an NFS mount point, then it is possible to enforce cluster-wide immediate list consistency as long as all cooperating nodes configure the same NFS directory.

This class does *not* support caching the GoogleCloudStorageItemInfo alongside filenames; it is strictly for supplementing listed filenames, and any returned CacheEntry should not be expected to contain a GoogleCloudStorageItemInfo regardless of info-expiration settings.

This class is thread-safe.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.hadoop.gcsio.DirectoryListCache

DirectoryListCache.Config, DirectoryListCache.Type

Field Summary

Fields inherited from class com.google.cloud.hadoop.gcsio.DirectoryListCache

cacheConfig, clock

Constructor Summary

Constructors

Constructor and Description
FileSystemBackedDirectoryListCache(String basePathStr)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	containsEntriesForImplicitDirectories() Since our backing store has no notion of "implicit" directories, automatically cache directory objects for 'implicit' directories, by virtue of having to create those real directories to insert the child object at all.
List<CacheEntry>	getBucketList()
CacheEntry	getCacheEntry(StorageResourceId resourceId) Returns the CacheEntry associated with resourceId, or null if it doesn't exist.
int	getInternalNumBuckets() Gets the internal number of CachedBucket entries, which may not be equal to the size of getBucketList() if there are expired entries.
int	getInternalNumObjects() Gets the internal total count of cached StorageObject entries.
List<CacheEntry>	getObjectList(String bucketName, String objectNamePrefix, String delimiter, Set<String> returnedPrefixes)
List<CacheEntry>	getRawBucketList()
static FileSystemBackedDirectoryListCache	getUninitializedInstanceForTest() Returns an instance not bound to a basePath for testing purposes.
CacheEntry	putResourceId(StorageResourceId resourceId) Adds the names of the Bucket or StorageObject referenced by resourceId to the cache, with no attached metadata.
void	removeResourceId(StorageResourceId resourceId) Removes CacheEntry associated with resourceId, if it exists.
void	setBasePath(String basePathStr) Only exposed for testing purposes; shouldn't be called outside of tests.
boolean	supportsCacheEntryByReference() We use a real filesystem as our authoritative cache, and thus CacheEntries are not shared references.

Methods inherited from class com.google.cloud.hadoop.gcsio.DirectoryListCache

getMutableConfig, isCacheEntryExpired, maybeInvalidateExpiredInfo, setClock, validateResourceId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileSystemBackedDirectoryListCache

public FileSystemBackedDirectoryListCache(String basePathStr)

Parameters:

basePathStr - The absolute path under which to place all mirrored directory trees; Paths.get(basePathStr).isAbsolute() must return true.

Method Detail

getUninitializedInstanceForTest

public static FileSystemBackedDirectoryListCache getUninitializedInstanceForTest()

Returns an instance not bound to a basePath for testing purposes.

supportsCacheEntryByReference

public boolean supportsCacheEntryByReference()

We use a real filesystem as our authoritative cache, and thus CacheEntries are not shared references.

Specified by:

supportsCacheEntryByReference in class DirectoryListCache

containsEntriesForImplicitDirectories

public boolean containsEntriesForImplicitDirectories()

Since our backing store has no notion of "implicit" directories, automatically cache directory objects for 'implicit' directories, by virtue of having to create those real directories to insert the child object at all.

Specified by:

containsEntriesForImplicitDirectories in class DirectoryListCache

putResourceId

public CacheEntry putResourceId(StorageResourceId resourceId)
                         throws IOException

Description copied from class: DirectoryListCache

Adds the names of the Bucket or StorageObject referenced by resourceId to the cache, with no attached metadata. If the entry already exists, then nothing is modified. If resourceId is a StorageObject, the parent Bucket name is also added to the cache, if it doesn't already exist. TODO(user): Even if the entry exists, it might be correct to invalidate any existing metadata and force a refresh next time it is fetched.

Specified by:

putResourceId in class DirectoryListCache

Returns:

The CacheEntry corresponding to the item added.

Throws:

IOException

getCacheEntry

public CacheEntry getCacheEntry(StorageResourceId resourceId)
                         throws IOException

Description copied from class: DirectoryListCache

Returns the CacheEntry associated with resourceId, or null if it doesn't exist. This returns the real mutable CacheEntry (rather than a copy of the data) so that the caller may efficiently update the info stored in the CacheEntry if necessary.

Specified by:

getCacheEntry in class DirectoryListCache

Throws:

IOException

removeResourceId

public void removeResourceId(StorageResourceId resourceId)
                      throws IOException

Description copied from class: DirectoryListCache

Removes CacheEntry associated with resourceId, if it exists. Cached GoogleCloudStorageItemInfo associated with the resourceId is also removed, if it exists. If resourceId denotes a non-empty bucket, then all the cached StorageObject children of that bucket will also be removed from the cache; it is the caller's responsibility to ensure that the bucket should really be removed. Note that normal expiration of a CachedBucket will *not* remove the actual CachedBucket, even though the bucketName will stop appearing in calls to getBucketList().

Specified by:

removeResourceId in class DirectoryListCache

Throws:

IOException

getBucketList

public List<CacheEntry> getBucketList()
                               throws IOException

Specified by:

getBucketList in class DirectoryListCache

Returns:

List of CacheEntry corresponding to Buckets known by this cache; includes buckets added to the cache automatically when caching a StorageObject contained in the corresponding bucket. Will not return null. Hides/removes expired bucket entries and clears any expired GoogleCloudStorageItemInfo associated with buckets.

Throws:

IOException

getRawBucketList

public List<CacheEntry> getRawBucketList()
                                  throws IOException

Specified by:

getRawBucketList in class DirectoryListCache

Returns:

List of *all* Bucket CacheEntries, including ones which might be expired. Doesn't actively invalidate, set expiration, or delete expired entries. Can be used when the internal bucket list is needed without wanting to cause any mutations in the cache.

Throws:

IOException

getObjectList

public List<CacheEntry> getObjectList(String bucketName,
                             String objectNamePrefix,
                             String delimiter,
                             Set<String> returnedPrefixes)
                               throws IOException

Specified by:

getObjectList in class DirectoryListCache

Parameters:

bucketName - The bucket inside of which to list objects.

objectNamePrefix - The prefix to be used to match object names to return.

delimiter - The character for specifying 'directory' boundaries, or null.

returnedPrefixes - A container to be populated with implied "directory objects" that come from some object which includes the prefix, followed by some string, then followed by the 'delimiter'. This may or may not be a duplicate of one of the actual returned directory objects. For example, if gs://foo/bar/baz.txt exists and we query with parameters ("foo", "ba", "/", {}) then the returnedPrefixes will be populated with the string "bar/" by virtue of existence of the "bar/baz.txt" file. May be null if the caller doesn't desire fetching such returnedPrefixes.

Returns:

List of CacheEntrys for StorageObjects residing in bucket bucketName which match the provided objectNamePrefix and delimiter, or possibly null if no such objects are present. May also return an empty list.

Throws:

IOException

getInternalNumBuckets

public int getInternalNumBuckets()
                          throws IOException

Description copied from class: DirectoryListCache

Gets the internal number of CachedBucket entries, which may not be equal to the size of getBucketList() if there are expired entries. Does not mutate the cache.

Specified by:

getInternalNumBuckets in class DirectoryListCache

Throws:

IOException

getInternalNumObjects

public int getInternalNumObjects()
                          throws IOException

Description copied from class: DirectoryListCache

Gets the internal total count of cached StorageObject entries. Does not mutate the cache.

Specified by:

getInternalNumObjects in class DirectoryListCache

Throws:

IOException

setBasePath

public void setBasePath(String basePathStr)

Only exposed for testing purposes; shouldn't be called outside of tests.