com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopSyncableOutputStream

java.lang.Object

java.io.OutputStream

com.google.cloud.hadoop.fs.gcs.GoogleHadoopSyncableOutputStream

All Implemented Interfaces:

java.io.Closeable, java.io.Flushable, java.lang.AutoCloseable, org.apache.hadoop.fs.Syncable

public class GoogleHadoopSyncableOutputStream
extends java.io.OutputStream
implements org.apache.hadoop.fs.Syncable

GoogleHadoopSyncableOutputStream implements the Syncable interface by composing objects created in separate underlying streams for each hsync() call.

Prior to the first hsync(), sync() or close() call, this channel will behave the same way as a basic non-syncable channel, writing directly to the destination file.

On the first call to hsync()/sync(), the destination file is committed and a new temporary file using a hidden-file prefix (underscore) is created with an additional suffix which differs for each subsequent temporary file in the series; during this time readers can read the data committed to the destination file, but not the bytes written to the temporary file since the last hsync() call.

On each subsequent hsync()/sync() call, the temporary file closed(), composed onto the destination file, then deleted, and a new temporary file is opened under a new filename for further writes.

Caveat: each hsync()/sync() requires many underlying read and mutation requests occurring sequentially, so latency is expected to be fairly high.

If errors occur mid-stream, there may be one or more temporary files failing to be cleaned up, and require manual intervention to discover and delete any such unused files. Data written prior to the most recent successful hsync() is persistent and safe in such a case.

If multiple writers are attempting to write to the same destination file, generation ids used with low-level precondition checks will cause all but a one writer to fail their precondition checks during writes, and a single remaining writer will safely occupy the stream.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	TEMPFILE_PREFIX

Constructor Summary

Constructors

Constructor and Description
GoogleHadoopSyncableOutputStream(GoogleHadoopFileSystemBase ghfs, java.net.URI gcsPath, org.apache.hadoop.fs.FileSystem.Statistics statistics, CreateFileOptions createFileOptions) Creates a new GoogleHadoopSyncableOutputStream with initial stream initialized and expected to begin at file-offset 0.

Method Summary

Modifier and Type	Method and Description
void	close()
void	hflush() There is no way to flush data to become available for readers without a full-fledged hsync(), so this method is a no-op.
void	hsync() This overrides Syncable.hsync(), but is not annotated as such because the method doesn't exist in Hadoop 1.
void	sync()
void	write(byte[] b, int offset, int len)
void	write(int b)

Methods inherited from class java.io.OutputStream

flush, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEMPFILE_PREFIX

public static final java.lang.String TEMPFILE_PREFIX

See Also:

Constant Field Values

Constructor Detail

GoogleHadoopSyncableOutputStream

public GoogleHadoopSyncableOutputStream(GoogleHadoopFileSystemBase ghfs,
                                        java.net.URI gcsPath,
                                        org.apache.hadoop.fs.FileSystem.Statistics statistics,
                                        CreateFileOptions createFileOptions)
                                 throws java.io.IOException

Creates a new GoogleHadoopSyncableOutputStream with initial stream initialized and expected to begin at file-offset 0. This constructor is not suitable for "appending" to already existing files.

Throws:

java.io.IOException

Method Detail

write

public void write(int b)
           throws java.io.IOException

Specified by:

write in class java.io.OutputStream

Throws:

java.io.IOException

write

public void write(byte[] b,
                  int offset,
                  int len)
           throws java.io.IOException

Overrides:

write in class java.io.OutputStream

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Overrides:

close in class java.io.OutputStream

Throws:

java.io.IOException

sync

public void sync()
          throws java.io.IOException

Specified by:

sync in interface org.apache.hadoop.fs.Syncable

Throws:

java.io.IOException

hflush

public void hflush()
            throws java.io.IOException

There is no way to flush data to become available for readers without a full-fledged hsync(), so this method is a no-op. This overrides Syncable.hflush(), but is not annotated as such because the method doesn't exist in Hadoop 1.

Throws:

java.io.IOException

hsync

public void hsync()
           throws java.io.IOException

This overrides Syncable.hsync(), but is not annotated as such because the method doesn't exist in Hadoop 1.

Throws:

CompositeLimitExceededException - if this hsync() call would require any future close() call to exceed the component limit. If CompositeLimitExceededException is thrown, no actual GCS operations are taken and it's safe to subsequently call close() on this stream as normal; it just means data written since the last successful hsync() has not yet been committed.

java.io.IOException