org.apache.hadoop.hbase.regionserver.wal

Class AbstractProtobufWALReader

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufWALReader

All Implemented Interfaces:

Closeable, AutoCloseable, AbstractFSWALProvider.Initializer

Direct Known Subclasses:

ProtobufWALStreamReader, ProtobufWALTailingReader

@InterfaceAudience.Private
public abstract class AbstractProtobufWALReader
extends Object
implements AbstractFSWALProvider.Initializer, Closeable

Base class for reading protobuf based wal reader

Field Summary

Fields

Modifier and Type	Field and Description
protected WALCellCodec.ByteStringUncompressor	byteStringUncompressor
protected Codec.Decoder	cellDecoder
protected String	codecClsName
protected CompressionContext	compressionCtx
protected org.apache.hadoop.conf.Configuration	conf
protected Decryptor	decryptor
protected long	fileLength
protected org.apache.hadoop.fs.FileSystem	fs
protected boolean	hasCompression
protected boolean	hasTagCompression
protected boolean	hasValueCompression
protected org.apache.hadoop.fs.FSDataInputStream	inputStream
protected org.apache.hadoop.fs.Path	path
static byte[]	PB_WAL_COMPLETE_MAGIC
static byte[]	PB_WAL_MAGIC
protected org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer	trailer
protected boolean	trailerPresent
protected int	trailerWarnSize
protected Compression.Algorithm	valueCompressionType
protected long	walEditsStopOffset

Constructor Summary

Constructors

Constructor and Description
AbstractProtobufWALReader()

Method Summary

Modifier and Type	Method and Description
void	close()
protected IOException	extractHiddenEof(Exception ex)
protected abstract InputStream	getCellCodecInputStream(org.apache.hadoop.fs.FSDataInputStream stream) Get or create the input stream used by cell decoder.
String	getCodecClsName() Returns the cell codec classname
long	getPosition()
protected String	getPositionQuietly()
List<String>	getWriterClsNames() Returns names of the accepted writer classes
void	init(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, long startPosition) A method to initialize a WAL reader.
protected void	initWALCellCodec(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header, InputStream inputStream)
protected boolean	isWALTrailer(long startPosition) This is used to determine whether we have already reached the WALTrailer.
protected Pair<org.apache.hadoop.fs.FSDataInputStream,org.apache.hadoop.fs.FileStatus>	open()
protected boolean	reachWALEditsStopOffset(long pos)
protected org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader	readHeader(org.apache.hadoop.fs.FSDataInputStream stream)
protected void	readTrailer(org.apache.hadoop.fs.FSDataInputStream stream, org.apache.hadoop.fs.FileStatus stat)
protected abstract void	skipTo(long position) Skip to the given position.
long	trailerSize()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PB_WAL_MAGIC

public static final byte[] PB_WAL_MAGIC

PB_WAL_COMPLETE_MAGIC

public static final byte[] PB_WAL_COMPLETE_MAGIC

conf

protected org.apache.hadoop.conf.Configuration conf

fs

protected org.apache.hadoop.fs.FileSystem fs

path

protected org.apache.hadoop.fs.Path path

fileLength

protected long fileLength

inputStream

protected org.apache.hadoop.fs.FSDataInputStream inputStream

compressionCtx

protected CompressionContext compressionCtx

hasCompression

protected boolean hasCompression

hasTagCompression

protected boolean hasTagCompression

hasValueCompression

protected boolean hasValueCompression

valueCompressionType

protected Compression.Algorithm valueCompressionType

cellDecoder

protected Codec.Decoder cellDecoder

byteStringUncompressor

protected WALCellCodec.ByteStringUncompressor byteStringUncompressor

walEditsStopOffset

protected long walEditsStopOffset

trailerPresent

protected boolean trailerPresent

trailer

protected org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer trailer

trailerWarnSize

protected int trailerWarnSize

codecClsName

protected String codecClsName

decryptor

protected Decryptor decryptor

Constructor Detail

AbstractProtobufWALReader

public AbstractProtobufWALReader()

Method Detail

getCellCodecInputStream

protected abstract InputStream getCellCodecInputStream(org.apache.hadoop.fs.FSDataInputStream stream)

Get or create the input stream used by cell decoder.

For implementing replication, we may need to limit the bytes we can read, so here we provide a method so subclasses can wrap the original input stream.

skipTo

protected abstract void skipTo(long position)
                        throws IOException

Skip to the given position.

Throws:

IOException

init

public void init(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.fs.Path path,
                 org.apache.hadoop.conf.Configuration conf,
                 long startPosition)
          throws IOException

Description copied from interface: AbstractFSWALProvider.Initializer

A method to initialize a WAL reader.

Specified by:

init in interface AbstractFSWALProvider.Initializer

startPosition - the start position you want to read from, -1 means start reading from the first WAL entry. Notice that, the first entry is not started at position as we have several headers, so typically you should not pass 0 here.

Throws:

IOException

open

protected final Pair<org.apache.hadoop.fs.FSDataInputStream,org.apache.hadoop.fs.FileStatus> open()
                                                                                           throws IOException

Throws:

IOException

readHeader

protected final org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader readHeader(org.apache.hadoop.fs.FSDataInputStream stream)
                                                                                          throws IOException

Throws:

IOException

initWALCellCodec

protected final void initWALCellCodec(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header,
                                      InputStream inputStream)
                               throws IOException

Throws:

IOException

readTrailer

protected final void readTrailer(org.apache.hadoop.fs.FSDataInputStream stream,
                                 org.apache.hadoop.fs.FileStatus stat)
                          throws IOException

Throws:

IOException

reachWALEditsStopOffset

protected final boolean reachWALEditsStopOffset(long pos)

getWriterClsNames

public List<String> getWriterClsNames()

Returns names of the accepted writer classes

getCodecClsName

public String getCodecClsName()

Returns the cell codec classname

getPosition

public long getPosition()
                 throws IOException

Throws:

IOException

trailerSize

public long trailerSize()

getPositionQuietly

protected final String getPositionQuietly()

extractHiddenEof

protected final IOException extractHiddenEof(Exception ex)

isWALTrailer

protected final boolean isWALTrailer(long startPosition)
                              throws IOException

This is used to determine whether we have already reached the WALTrailer. As the size and magic are at the end of the WAL file, it is possible that these two options are missing while writing, so we will consider there is no trailer. And when we actually reach the WALTrailer, we will try to decode it as WALKey and we will fail but the error could be varied as it is parsing WALTrailer actually.

Returns:

whether this is a WALTrailer and we should throw EOF to upper layer the file is done

Throws:

IOException

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable