org.apache.hadoop.hbase.regionserver.wal

类 AsyncFSWAL

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL<WALProvider.AsyncWriter>

org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL

所有已实现的接口:

Closeable, AutoCloseable, WALFileLengthProvider, WAL

@InterfaceAudience.LimitedPrivate(value="Configuration")
public class AsyncFSWAL
extends AbstractFSWAL<WALProvider.AsyncWriter>

An asynchronous implementation of FSWAL.

Here 'waitingConsumePayloads' acts as the RingBuffer in FSHLog.

For append, we process it as follow:

1. In the caller thread(typically, in the rpc handler thread):
   1. Insert the entry into 'waitingConsumePayloads'. Use ringbuffer sequence as txid.
   2. Schedule the consumer task if needed. See shouldScheduleConsumer() for more details.
2. In the consumer task(executed in a single threaded thread pool)
   1. Poll the entry from waitingConsumePayloads and insert it into toWriteAppends
   2. Poll the entry from toWriteAppends, append it to the AsyncWriter, and insert it into unackedAppends
   3. If the buffered size reaches batchSize, or there is a sync request, then we call sync on the AsyncWriter.
   4. In the callback methods:
      • If succeeded, poll the entry from unackedAppends and drop it.
      • If failed, add all the entries in unackedAppends back to toWriteAppends and wait for writing them again.

For sync, the processing stages are almost same. And different from FSHLog, we will open a new writer and rewrite unacked entries to the new writer and sync again if we hit a sync error.

Here we only describe the logic of doReplaceWriter. The main logic of rollWriter is same with FSHLog.
For a normal roll request(for example, we have reached the log roll size):

1. In the log roller thread, we will set waitingRoll(int) to true and readyForRolling to false, and then wait on readyForRolling(see waitForSafePoint()).
2. In the consumer thread, we will stop polling entries from waitingConsumePayloads if waitingRoll(int) is true, and also stop writing the entries in toWriteAppends out.
3. If there are unflush data in the writer, sync them.
4. When all out-going sync request is finished, i.e, the unackedAppends is empty, signal the readyForRollingCond.
5. Back to the log roller thread, now we can confirm that there are no out-going entries, i.e., we reach a safe point. So it is safe to replace old writer with new writer now.
6. Set writerBroken(int) and waitingRoll(int) to false.
7. Schedule the consumer task.
8. Schedule a background task to close the old writer.

For a broken writer roll request, the only difference is that we can bypass the wait for safe point stage.

嵌套类概要

从接口继承的嵌套类/接口 org.apache.hadoop.hbase.wal.WAL

WAL.Entry, WAL.Reader

字段概要

字段

限定符和类型	字段和说明
static String	ASYNC_WAL_USE_SHARED_EVENT_LOOP
static String	ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS
static boolean	DEFAULT_ASYNC_WAL_USE_SHARED_EVENT_LOOP
static int	DEFAULT_ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS
static long	DEFAULT_WAL_BATCH_SIZE
static String	WAL_BATCH_SIZE

从类继承的字段 org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL

blocksize, closed, conf, coprocessorHost, DEFAULT_SLOW_SYNC_TIME_MS, filenum, fs, highestSyncedTxid, highestUnsyncedTxid, implClassName, listeners, logrollsize, maxLogs, numEntries, ourFiles, prefixPathStr, rollRequested, rollWriterLock, sequenceIdAccounting, shutdown, slowSyncNs, totalLogSize, useHsync, walArchiveDir, walDir, walFile2Props, walFilePrefix, walFileSuffix

构造器概要

构造器

构造器和说明

AsyncFSWAL(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix, org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup eventLoopGroup, Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel> channelClass)

方法概要

限定符和类型	方法和说明
protected long	append(RegionInfo hri, WALKeyImpl key, WALEdit edits, boolean inMemstore) Append a set of edits to the WAL.
protected WALProvider.AsyncWriter	createWriterInstance(org.apache.hadoop.fs.Path path)
protected void	doAppend(WALProvider.AsyncWriter writer, org.apache.hadoop.hbase.regionserver.wal.FSWALEntry entry)
protected boolean	doCheckLogLowReplication()
protected void	doReplaceWriter(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath, WALProvider.AsyncWriter nextWriter) Notice that you need to clear the AbstractFSWAL.rollRequested flag in this method, as the new writer will begin to work before returning from this method.
protected void	doShutdown()
void	sync() Sync what we have in the WAL.
void	sync(boolean forceSync)
void	sync(long txid) Sync the WAL if the txId was not already sync'd.
void	sync(long txid, boolean forceSync)

从类继承的方法 org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL

abortCacheFlush, appendData, appendEntry, appendMarker, atHeadOfRingBufferEventHandlerAppend, blockOnSync, checkLogLowReplication, close, completeCacheFlush, computeFilename, getCoprocessorHost, getCurrentFileName, getEarliestMemStoreSeqNum, getEarliestMemStoreSeqNum, getFilenum, getFileNumFromFileName, getLogFileSize, getLogFileSizeIfBeingWritten, getNumLogFiles, getNumRolledLogFiles, getPreallocatedEventCount, getSyncFuture, getWALArchivePath, init, isLogRollRequested, logRollAndSetupWalProps, main, postSync, registerWALActionsListener, requestLogRoll, requestLogRoll, rollWriter, rollWriter, shutdown, stampSequenceIdAndPublishToRingBuffer, startCacheFlush, startCacheFlush, toString, unregisterWALActionsListener, updateStore

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

字段详细资料

WAL_BATCH_SIZE

public static final String WAL_BATCH_SIZE

另请参阅:

常量字段值

DEFAULT_WAL_BATCH_SIZE

public static final long DEFAULT_WAL_BATCH_SIZE

另请参阅:

常量字段值

ASYNC_WAL_USE_SHARED_EVENT_LOOP

public static final String ASYNC_WAL_USE_SHARED_EVENT_LOOP

另请参阅:

常量字段值

DEFAULT_ASYNC_WAL_USE_SHARED_EVENT_LOOP

public static final boolean DEFAULT_ASYNC_WAL_USE_SHARED_EVENT_LOOP

另请参阅:

常量字段值

ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS

public static final String ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS

另请参阅:

常量字段值

DEFAULT_ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS

public static final int DEFAULT_ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS

另请参阅:

常量字段值

构造器详细资料

AsyncFSWAL

public AsyncFSWAL(org.apache.hadoop.fs.FileSystem fs,
                  org.apache.hadoop.fs.Path rootDir,
                  String logDir,
                  String archiveDir,
                  org.apache.hadoop.conf.Configuration conf,
                  List<WALActionsListener> listeners,
                  boolean failIfWALExists,
                  String prefix,
                  String suffix,
                  org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup eventLoopGroup,
                  Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel> channelClass)
           throws FailedLogCloseException,
                  IOException

抛出:

FailedLogCloseException

IOException

方法详细资料

append

protected long append(RegionInfo hri,
                      WALKeyImpl key,
                      WALEdit edits,
                      boolean inMemstore)
               throws IOException

从类复制的说明: AbstractFSWAL

Append a set of edits to the WAL.

The WAL is not flushed/sync'd after this transaction completes BUT on return this edit must have its region edit/sequence id assigned else it messes up our unification of mvcc and sequenceid. On return key will have the region edit/sequence id filled in.

NOTE: This append, at a time that is usually after this call returns, starts an mvcc transaction by calling 'begin' wherein which we assign this update a sequenceid. At assignment time, we stamp all the passed in Cells inside WALEdit with their sequenceId. You must 'complete' the transaction this mvcc transaction by calling MultiVersionConcurrencyControl#complete(...) or a variant otherwise mvcc will get stuck. Do it in the finally of a try/finally block within which this append lives and any subsequent operations like sync or update of memstore, etc. Get the WriteEntry to pass mvcc out of the passed in WALKey walKey parameter. Be warned that the WriteEntry is not immediately available on return from this method. It WILL be available subsequent to a sync of this append; otherwise, you will just have to wait on the WriteEntry to get filled in.

指定者:

append 在类中 AbstractFSWAL<WALProvider.AsyncWriter>

参数:

hri - the regioninfo associated with append

key - Modified by this call; we add to it this edits region edit/sequence id.

edits - Edits to append. MAY CONTAIN NO EDITS for case where we want to get an edit sequence id that is after all currently appended edits.

inMemstore - Always true except for case where we are writing a region event meta marker edit, for example, a compaction completion record into the WAL or noting a Region Open event. In these cases the entry is just so we can finish an unfinished compaction after a crash when the new Server reads the WAL on recovery, etc. These transition event 'Markers' do not go via the memstore. When memstore is false, we presume a Marker event edit.

返回:

Returns a 'transaction id' and key will have the region edit/sequence id in it.

抛出:

IOException

sync

public void sync()
          throws IOException

从接口复制的说明: WAL

Sync what we have in the WAL.

抛出:

IOException

sync

public void sync(long txid)
          throws IOException

从接口复制的说明: WAL

Sync the WAL if the txId was not already sync'd.

参数:

txid - Transaction id to sync to.

抛出:

IOException

sync

public void sync(boolean forceSync)
          throws IOException

参数:

forceSync - Flag to force sync rather than flushing to the buffer. Example - Hadoop hflush vs hsync.

抛出:

IOException

sync

public void sync(long txid,
                 boolean forceSync)
          throws IOException

参数:

txid - Transaction id to sync to.

forceSync - Flag to force sync rather than flushing to the buffer. Example - Hadoop hflush vs hsync.

抛出:

IOException

createWriterInstance

protected WALProvider.AsyncWriter createWriterInstance(org.apache.hadoop.fs.Path path)
                                                throws IOException

指定者:

createWriterInstance 在类中 AbstractFSWAL<WALProvider.AsyncWriter>

抛出:

IOException

doReplaceWriter

protected void doReplaceWriter(org.apache.hadoop.fs.Path oldPath,
                               org.apache.hadoop.fs.Path newPath,
                               WALProvider.AsyncWriter nextWriter)
                        throws IOException

从类复制的说明: AbstractFSWAL

Notice that you need to clear the AbstractFSWAL.rollRequested flag in this method, as the new writer will begin to work before returning from this method. If we clear the flag after returning from this call, we may miss a roll request. The implementation class should choose a proper place to clear the AbstractFSWAL.rollRequested flag so we do not miss a roll request, typically before you start writing to the new writer.

指定者:

doReplaceWriter 在类中 AbstractFSWAL<WALProvider.AsyncWriter>

抛出:

IOException

doShutdown

protected void doShutdown()
                   throws IOException

指定者:

doShutdown 在类中 AbstractFSWAL<WALProvider.AsyncWriter>

抛出:

IOException

doAppend

protected void doAppend(WALProvider.AsyncWriter writer,
                        org.apache.hadoop.hbase.regionserver.wal.FSWALEntry entry)

指定者:

doAppend 在类中 AbstractFSWAL<WALProvider.AsyncWriter>

doCheckLogLowReplication

protected boolean doCheckLogLowReplication()

指定者:

doCheckLogLowReplication 在类中 AbstractFSWAL<WALProvider.AsyncWriter>