org.apache.hadoop.hbase.regionserver.wal

Class AsyncProtobufLogWriter

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter

org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter

All Implemented Interfaces:

Closeable, AutoCloseable, AsyncFSWALProvider.AsyncWriter, WALProvider.AsyncWriter, WALProvider.WriterBase

Direct Known Subclasses:

SecureAsyncProtobufLogWriter

@InterfaceAudience.Private
public class AsyncProtobufLogWriter
extends AbstractProtobufLogWriter
implements AsyncFSWALProvider.AsyncWriter

AsyncWriter for protobuf-based WAL.

Field Summary

Fields inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter

cellEncoder, compressionContext, compressor, conf, length, trailer, trailerWarnSize, trailerWritten

Constructor Summary

Constructors

Constructor and Description
AsyncProtobufLogWriter(org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup eventLoopGroup, Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel> channelClass)

Method Summary

Modifier and Type	Method and Description
void	append(WAL.Entry entry)
void	close()
protected void	closeOutputIfNecessary() It is straight forward to close the output, do not need to write trailer like the Writer.close
AsyncFSOutput	getOutput()
protected OutputStream	getOutputStreamForCellEncoder()
long	getSyncedLength() NOTE: We add this method for WALFileLengthProvider used for replication, considering the case if we use AsyncFSWAL,we write to 3 DNs concurrently, according to the visibility guarantee of HDFS, the data will be available immediately when arriving at DN since all the DNs will be considered as the last one in pipeline.
protected String	getWriterClassName()
protected void	initOutput(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean overwritable, int bufferSize, short replication, long blockSize, StreamSlowMonitor monitor)
CompletableFuture<Long>	sync(boolean forceSync)
protected long	writeMagicAndWALHeader(byte[] magic, org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header) return the file length after written.
protected long	writeWALTrailerAndMagic(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer trailer, byte[] magic)

Methods inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter

buildSecureWALHeader, buildWALHeader, getLength, init, initAfterHeader, secureInitAfterHeader, setEncryptor, writeWALTrailer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.wal.AsyncFSWALProvider.AsyncWriter

init

Methods inherited from interface org.apache.hadoop.hbase.wal.WALProvider.WriterBase

getLength

Constructor Detail

AsyncProtobufLogWriter

public AsyncProtobufLogWriter(org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup eventLoopGroup,
                              Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel> channelClass)

Method Detail

getWriterClassName

protected String getWriterClassName()

Overrides:

getWriterClassName in class AbstractProtobufLogWriter

append

public void append(WAL.Entry entry)

Specified by:

append in interface WALProvider.AsyncWriter

sync

public CompletableFuture<Long> sync(boolean forceSync)

Specified by:

sync in interface WALProvider.AsyncWriter

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getOutput

public AsyncFSOutput getOutput()

initOutput

protected void initOutput(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path path,
                          boolean overwritable,
                          int bufferSize,
                          short replication,
                          long blockSize,
                          StreamSlowMonitor monitor)
                   throws IOException,
                          CommonFSUtils.StreamLacksCapabilityException

Specified by:

initOutput in class AbstractProtobufLogWriter

Throws:

IOException

CommonFSUtils.StreamLacksCapabilityException

closeOutputIfNecessary

protected void closeOutputIfNecessary()

Description copied from class: AbstractProtobufLogWriter

It is straight forward to close the output, do not need to write trailer like the Writer.close

Overrides:

closeOutputIfNecessary in class AbstractProtobufLogWriter

writeMagicAndWALHeader

protected long writeMagicAndWALHeader(byte[] magic,
                                      org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header)
                               throws IOException

Description copied from class: AbstractProtobufLogWriter

return the file length after written.

Specified by:

writeMagicAndWALHeader in class AbstractProtobufLogWriter

Throws:

IOException

writeWALTrailerAndMagic

protected long writeWALTrailerAndMagic(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer trailer,
                                       byte[] magic)
                                throws IOException

Specified by:

writeWALTrailerAndMagic in class AbstractProtobufLogWriter

Throws:

IOException

getOutputStreamForCellEncoder

protected OutputStream getOutputStreamForCellEncoder()

Specified by:

getOutputStreamForCellEncoder in class AbstractProtobufLogWriter

getSyncedLength

public long getSyncedLength()

Description copied from interface: WALProvider.WriterBase

NOTE: We add this method for WALFileLengthProvider used for replication, considering the case if we use AsyncFSWAL,we write to 3 DNs concurrently, according to the visibility guarantee of HDFS, the data will be available immediately when arriving at DN since all the DNs will be considered as the last one in pipeline. This means replication may read uncommitted data and replicate it to the remote cluster and cause data inconsistency. The method WALProvider.WriterBase.getLength() may return length which just in hdfs client buffer and not successfully synced to HDFS, so we use this method to return the length successfully synced to HDFS and replication thread could only read writing WAL file limited by this length. see also HBASE-14004 and this document for more details: https://docs.google.com/document/d/11AyWtGhItQs6vsLRIx32PwTxmBY3libXwGXI25obVEY/edit#

Specified by:

getSyncedLength in interface WALProvider.WriterBase

Returns:

byteSize successfully synced to underlying filesystem.