com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopFileSystemBase

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

All Implemented Interfaces:

FileSystemDescriptor, Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

GoogleHadoopFileSystem, GoogleHadoopGlobalRootedFileSystem

public abstract class GoogleHadoopFileSystemBase
extends org.apache.hadoop.fs.FileSystem
implements FileSystemDescriptor

This class provides a Hadoop compatible File System on top of Google Cloud Storage (GCS). It is implemented as a thin abstraction layer on top of GCS. The layer hides any specific characteristics of the underlying store and exposes FileSystem interface understood by the Hadoop engine.

Users interact with the files in the storage using fully qualified URIs. The file system exposed by this class is identified using the 'gs' scheme. For example, gs://dir1/dir2/file1.txt.

This implementation translates paths between hadoop Path and GCS URI with the convention that the Hadoop root directly corresponds to the GCS "root", e.g. gs:/. This is convenient for many reasons, such as data portability and close equivalence to gsutil paths, but imposes certain inherited constraints, such as files not being allowed in root (only 'directories' can be placed in root), and directory names inside root have a more limited set of allowed characters.

One of the main goals of this implementation is to maintain compatibility with behavior of HDFS implementation when accessed through FileSystem interface. HDFS implementation is not very consistent about the cases when it throws versus the cases when methods return false. We run GHFS tests and HDFS tests against the same test data and use that as a guide to decide whether to throw or to return false.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GoogleHadoopFileSystemBase.Counter Defines names of counters we track for each operation.
protected static class	GoogleHadoopFileSystemBase.ListStatusFileNotFoundBehavior Behavior of listStatus when a path is not found.
static class	GoogleHadoopFileSystemBase.ParentTimestampUpdateIncludePredicate A predicate that processes individual directory paths and evaluates the conditions set in fs.gs.parent.timestamp.update.enable, fs.gs.parent.timestamp.update.substrings.include and fs.gs.parent.timestamp.update.substrings.exclude to determine if a path should be ignored when running directory timestamp updates.

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static String	AUTHENTICATION_PREFIX
static int	BLOCK_SIZE_DEFAULT
static String	BLOCK_SIZE_KEY
static int	BUFFERSIZE_DEFAULT
static String	BUFFERSIZE_KEY
protected com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,AtomicLong>	counters
static org.apache.hadoop.fs.PathFilter	DEFAULT_FILTER
protected long	defaultBlockSize
static String	ENABLE_GCE_SERVICE_ACCOUNT_AUTH_KEY
static String	GCS_CLIENT_ID_KEY
static String	GCS_CLIENT_SECRET_KEY
static boolean	GCS_CREATE_SYSTEM_BUCKET_DEFAULT
static String	GCS_CREATE_SYSTEM_BUCKET_KEY
static boolean	GCS_ENABLE_FLAT_GLOB_DEFAULT
static String	GCS_ENABLE_FLAT_GLOB_KEY
static boolean	GCS_ENABLE_MARKER_FILE_CREATION_DEFAULT
static String	GCS_ENABLE_MARKER_FILE_CREATION_KEY
static boolean	GCS_ENABLE_METADATA_CACHE_DEFAULT
static String	GCS_ENABLE_METADATA_CACHE_KEY
static boolean	GCS_ENABLE_REPAIR_IMPLICIT_DIRECTORIES_DEFAULT
static String	GCS_ENABLE_REPAIR_IMPLICIT_DIRECTORIES_KEY
static String	GCS_FILE_SIZE_LIMIT_250GB
static boolean	GCS_FILE_SIZE_LIMIT_250GB_DEFAULT
static String	GCS_METADATA_CACHE_DIRECTORY_DEFAULT
static String	GCS_METADATA_CACHE_DIRECTORY_KEY
static String	GCS_METADATA_CACHE_TYPE_DEFAULT
static String	GCS_METADATA_CACHE_TYPE_KEY
static boolean	GCS_PARENT_TIMESTAMP_UPDATE_ENABLE_DEFAULT
static String	GCS_PARENT_TIMESTAMP_UPDATE_ENABLE_KEY
static String	GCS_PARENT_TIMESTAMP_UPDATE_EXCLUDES_DEFAULT
static String	GCS_PARENT_TIMESTAMP_UPDATE_EXCLUDES_KEY
static String	GCS_PARENT_TIMESTAMP_UPDATE_INCLUDES_DEFAULT
static String	GCS_PARENT_TIMESTAMP_UPDATE_INCLUDES_KEY
static String	GCS_PROJECT_ID_KEY
static String	GCS_SYSTEM_BUCKET_KEY
static String	GCS_WORKING_DIRECTORY_KEY
protected GoogleCloudStorageFileSystem	gcsfs
static String	GHFS_ID
protected URI	initUri
protected GoogleHadoopFileSystemBase.ListStatusFileNotFoundBehavior	listStatusFileNotFoundBehavior
static LogUtil	log
static String	MR_JOB_HISTORY_DONE_DIR_KEY
static String	MR_JOB_HISTORY_INTERMEDIATE_DONE_DIR_KEY
static String	PROPERTIES_FILE
static short	REPLICATION_FACTOR_DEFAULT
static String	SERVICE_ACCOUNT_AUTH_EMAIL_KEY
static String	SERVICE_ACCOUNT_AUTH_KEYFILE_KEY
protected String	systemBucket Deprecated.
static String	UNKNOWN_VERSION
static String	VERSION
static String	VERSION_PROPERTY
static int	WRITE_BUFFERSIZE_DEFAULT
static String	WRITE_BUFFERSIZE_KEY

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
GoogleHadoopFileSystemBase() Constructs an instance of GoogleHadoopFileSystemBase; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.
GoogleHadoopFileSystemBase(GoogleCloudStorageFileSystem gcsfs) Constructs an instance of GoogleHadoopFileSystemBase using the provided GoogleCloudStorageFileSystem; initialize() will not re-initialize it.

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path hadoopPath, int bufferSize, org.apache.hadoop.util.Progressable progress) Appends to an existing file (optional operation).
protected void	checkPath(org.apache.hadoop.fs.Path path)
void	close()
void	completeLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile)
void	configureBuckets(String systemBucketName, boolean createSystemBucket) Validates and possibly creates the system bucket.
void	copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path[] srcs, org.apache.hadoop.fs.Path dst)
void	copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	copyToLocalFile(boolean delSrc, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Opens the given file for writing.
protected com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,AtomicLong>	createCounterMap()
boolean	delete(org.apache.hadoop.fs.Path f) Deprecated. Use delete(Path, boolean) instead
boolean	delete(org.apache.hadoop.fs.Path hadoopPath, boolean recursive) Deletes the given file or directory.
boolean	deleteOnExit(org.apache.hadoop.fs.Path f)
String	getCanonicalServiceName() Returns null, because GHFS does not use security tokens.
org.apache.hadoop.fs.ContentSummary	getContentSummary(org.apache.hadoop.fs.Path f)
long	getDefaultBlockSize()
protected int	getDefaultPort() The default port is listed as -1 as an indication that ports are not used.
short	getDefaultReplication() Gets the default replication factor.
abstract org.apache.hadoop.fs.Path	getDefaultWorkingDirectory() Gets the default value of working directory.
org.apache.hadoop.security.token.Token<?>	getDelegationToken(String renewer)
org.apache.hadoop.fs.FileChecksum	getFileChecksum(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path hadoopPath) Gets status of the given path item.
abstract org.apache.hadoop.fs.Path	getFileSystemRoot() Returns the Hadoop path representing the root of the FileSystem associated with this FileSystemDescriptor.
abstract URI	getGcsPath(org.apache.hadoop.fs.Path hadoopPath) Gets GCS path corresponding to the given Hadoop path, which can be relative or absolute, and can have either gs:// or gs:/ forms.
abstract org.apache.hadoop.fs.Path	getHadoopPath(URI gcsPath) Gets Hadoop path corresponding to the given GCS path.
String	getHadoopScheme() Deprecated.
org.apache.hadoop.fs.Path	getHomeDirectory() Returns home directory of the current user.
protected abstract String	getHomeDirectorySubpath() Returns an unqualified path without any leading slash, relative to the filesystem root, which serves as the home directory of the current user; see getHomeDirectory for a description of what the home directory means.
abstract String	getScheme() Returns the URI scheme for the Hadoop FileSystem associated with this FileSystemDescriptor.
URI	getUri() Returns a URI of the root of this FileSystem.
long	getUsed()
org.apache.hadoop.fs.Path	getWorkingDirectory() Gets the current working directory.
org.apache.hadoop.fs.FileStatus[]	globStatus(org.apache.hadoop.fs.Path pathPattern) Returns an array of FileStatus objects whose path names match pathPattern.
org.apache.hadoop.fs.FileStatus[]	globStatus(org.apache.hadoop.fs.Path pathPattern, org.apache.hadoop.fs.PathFilter filter) Returns an array of FileStatus objects whose path names match pathPattern and is accepted by the user-supplied path filter.
void	initialize(URI path, org.apache.hadoop.conf.Configuration config) See initialize(URI, Configuration, boolean) for details; calls with third arg defaulting to 'true' for initializing the superclass.
void	initialize(URI path, org.apache.hadoop.conf.Configuration config, boolean initSuperclass) Initializes this file system instance.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path hadoopPath) Lists file status.
org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.Path path) Overridden to make root it's own parent.
boolean	mkdirs(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission) Makes the given path and all non-existent parents directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path hadoopPath, int bufferSize) Opens the given file for reading.
protected void	processDeleteOnExit()
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Renames src to dst.
protected void	setListStatusFileNotFoundBehavior(GoogleHadoopFileSystemBase.ListStatusFileNotFoundBehavior behavior)
void	setOwner(org.apache.hadoop.fs.Path p, String username, String groupname)
void	setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission)
void	setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime)
void	setVerifyChecksum(boolean verifyChecksum)
void	setWorkingDirectory(org.apache.hadoop.fs.Path hadoopPath) Sets the current working directory to the given path.
org.apache.hadoop.fs.Path	startLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile)

Methods inherited from class org.apache.hadoop.fs.FileSystem

addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, concat, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getBlockSize, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileBlockLocations, getFileLinkStatus, getFileSystemClass, getFSofPath, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setReplication, setWriteChecksum, supportsSymlinks

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

public static final LogUtil log

REPLICATION_FACTOR_DEFAULT

public static final short REPLICATION_FACTOR_DEFAULT

See Also:

Constant Field Values

BUFFERSIZE_KEY

public static final String BUFFERSIZE_KEY

See Also:

Constant Field Values

BUFFERSIZE_DEFAULT

public static final int BUFFERSIZE_DEFAULT

See Also:

Constant Field Values

WRITE_BUFFERSIZE_KEY

public static final String WRITE_BUFFERSIZE_KEY

See Also:

Constant Field Values

WRITE_BUFFERSIZE_DEFAULT

public static final int WRITE_BUFFERSIZE_DEFAULT

See Also:

Constant Field Values

BLOCK_SIZE_KEY

public static final String BLOCK_SIZE_KEY

See Also:

Constant Field Values

BLOCK_SIZE_DEFAULT

public static final int BLOCK_SIZE_DEFAULT

See Also:

Constant Field Values

AUTHENTICATION_PREFIX

public static final String AUTHENTICATION_PREFIX

See Also:

Constant Field Values

ENABLE_GCE_SERVICE_ACCOUNT_AUTH_KEY

public static final String ENABLE_GCE_SERVICE_ACCOUNT_AUTH_KEY

See Also:

Constant Field Values

SERVICE_ACCOUNT_AUTH_EMAIL_KEY

public static final String SERVICE_ACCOUNT_AUTH_EMAIL_KEY

See Also:

Constant Field Values

SERVICE_ACCOUNT_AUTH_KEYFILE_KEY

public static final String SERVICE_ACCOUNT_AUTH_KEYFILE_KEY

See Also:

Constant Field Values

GCS_PROJECT_ID_KEY

public static final String GCS_PROJECT_ID_KEY

See Also:

Constant Field Values

GCS_CLIENT_ID_KEY

public static final String GCS_CLIENT_ID_KEY

See Also:

Constant Field Values

GCS_CLIENT_SECRET_KEY

public static final String GCS_CLIENT_SECRET_KEY

See Also:

Constant Field Values

GCS_SYSTEM_BUCKET_KEY

public static final String GCS_SYSTEM_BUCKET_KEY

See Also:

Constant Field Values

GCS_CREATE_SYSTEM_BUCKET_KEY

public static final String GCS_CREATE_SYSTEM_BUCKET_KEY

See Also:

Constant Field Values

GCS_CREATE_SYSTEM_BUCKET_DEFAULT

public static final boolean GCS_CREATE_SYSTEM_BUCKET_DEFAULT

See Also:

Constant Field Values

GCS_WORKING_DIRECTORY_KEY

public static final String GCS_WORKING_DIRECTORY_KEY

See Also:

Constant Field Values

GCS_FILE_SIZE_LIMIT_250GB

public static final String GCS_FILE_SIZE_LIMIT_250GB

See Also:

Constant Field Values

GCS_FILE_SIZE_LIMIT_250GB_DEFAULT

public static final boolean GCS_FILE_SIZE_LIMIT_250GB_DEFAULT

See Also:

Constant Field Values

GCS_ENABLE_METADATA_CACHE_KEY

public static final String GCS_ENABLE_METADATA_CACHE_KEY

See Also:

Constant Field Values

GCS_ENABLE_METADATA_CACHE_DEFAULT

public static final boolean GCS_ENABLE_METADATA_CACHE_DEFAULT

See Also:

Constant Field Values

GCS_PARENT_TIMESTAMP_UPDATE_ENABLE_KEY

public static final String GCS_PARENT_TIMESTAMP_UPDATE_ENABLE_KEY

See Also:

Constant Field Values

GCS_PARENT_TIMESTAMP_UPDATE_ENABLE_DEFAULT

public static final boolean GCS_PARENT_TIMESTAMP_UPDATE_ENABLE_DEFAULT

See Also:

Constant Field Values

GCS_METADATA_CACHE_TYPE_KEY

public static final String GCS_METADATA_CACHE_TYPE_KEY

See Also:

Constant Field Values

GCS_METADATA_CACHE_TYPE_DEFAULT

public static final String GCS_METADATA_CACHE_TYPE_DEFAULT

See Also:

Constant Field Values

GCS_METADATA_CACHE_DIRECTORY_KEY

public static final String GCS_METADATA_CACHE_DIRECTORY_KEY

See Also:

Constant Field Values

GCS_METADATA_CACHE_DIRECTORY_DEFAULT

public static final String GCS_METADATA_CACHE_DIRECTORY_DEFAULT

See Also:

Constant Field Values

GCS_PARENT_TIMESTAMP_UPDATE_EXCLUDES_KEY

public static final String GCS_PARENT_TIMESTAMP_UPDATE_EXCLUDES_KEY

See Also:

Constant Field Values

GCS_PARENT_TIMESTAMP_UPDATE_EXCLUDES_DEFAULT

public static final String GCS_PARENT_TIMESTAMP_UPDATE_EXCLUDES_DEFAULT

See Also:

Constant Field Values

MR_JOB_HISTORY_INTERMEDIATE_DONE_DIR_KEY

public static final String MR_JOB_HISTORY_INTERMEDIATE_DONE_DIR_KEY

See Also:

Constant Field Values

MR_JOB_HISTORY_DONE_DIR_KEY

public static final String MR_JOB_HISTORY_DONE_DIR_KEY

See Also:

Constant Field Values

GCS_PARENT_TIMESTAMP_UPDATE_INCLUDES_KEY

public static final String GCS_PARENT_TIMESTAMP_UPDATE_INCLUDES_KEY

See Also:

Constant Field Values

GCS_PARENT_TIMESTAMP_UPDATE_INCLUDES_DEFAULT

public static final String GCS_PARENT_TIMESTAMP_UPDATE_INCLUDES_DEFAULT

GCS_ENABLE_REPAIR_IMPLICIT_DIRECTORIES_KEY

public static final String GCS_ENABLE_REPAIR_IMPLICIT_DIRECTORIES_KEY

See Also:

Constant Field Values

GCS_ENABLE_REPAIR_IMPLICIT_DIRECTORIES_DEFAULT

public static final boolean GCS_ENABLE_REPAIR_IMPLICIT_DIRECTORIES_DEFAULT

See Also:

Constant Field Values

GCS_ENABLE_FLAT_GLOB_KEY

public static final String GCS_ENABLE_FLAT_GLOB_KEY

See Also:

Constant Field Values

GCS_ENABLE_FLAT_GLOB_DEFAULT

public static final boolean GCS_ENABLE_FLAT_GLOB_DEFAULT

See Also:

Constant Field Values

GCS_ENABLE_MARKER_FILE_CREATION_KEY

public static final String GCS_ENABLE_MARKER_FILE_CREATION_KEY

See Also:

Constant Field Values

GCS_ENABLE_MARKER_FILE_CREATION_DEFAULT

public static final boolean GCS_ENABLE_MARKER_FILE_CREATION_DEFAULT

See Also:

Constant Field Values

DEFAULT_FILTER

public static final org.apache.hadoop.fs.PathFilter DEFAULT_FILTER

PROPERTIES_FILE

public static final String PROPERTIES_FILE

See Also:

Constant Field Values

VERSION_PROPERTY

public static final String VERSION_PROPERTY

See Also:

Constant Field Values

UNKNOWN_VERSION

public static final String UNKNOWN_VERSION

See Also:

Constant Field Values

VERSION

public static final String VERSION

GHFS_ID

public static final String GHFS_ID

initUri

protected URI initUri

systemBucket

@Deprecated
protected String systemBucket

Deprecated.

gcsfs

protected GoogleCloudStorageFileSystem gcsfs

defaultBlockSize

protected long defaultBlockSize

counters

protected final com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,AtomicLong> counters

listStatusFileNotFoundBehavior

protected GoogleHadoopFileSystemBase.ListStatusFileNotFoundBehavior listStatusFileNotFoundBehavior

Constructor Detail

GoogleHadoopFileSystemBase

public GoogleHadoopFileSystemBase()

Constructs an instance of GoogleHadoopFileSystemBase; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

GoogleHadoopFileSystemBase

public GoogleHadoopFileSystemBase(GoogleCloudStorageFileSystem gcsfs)

Constructs an instance of GoogleHadoopFileSystemBase using the provided GoogleCloudStorageFileSystem; initialize() will not re-initialize it.

Method Detail

createCounterMap

protected com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,AtomicLong> createCounterMap()

setListStatusFileNotFoundBehavior

protected void setListStatusFileNotFoundBehavior(GoogleHadoopFileSystemBase.ListStatusFileNotFoundBehavior behavior)

getHomeDirectorySubpath

protected abstract String getHomeDirectorySubpath()

Returns an unqualified path without any leading slash, relative to the filesystem root, which serves as the home directory of the current user; see getHomeDirectory for a description of what the home directory means.

getHadoopPath

public abstract org.apache.hadoop.fs.Path getHadoopPath(URI gcsPath)

Gets Hadoop path corresponding to the given GCS path.

Parameters:

gcsPath - Fully-qualified GCS path, of the form gs:///