org.apache.hadoop.hbase.wal

Class WALSplitter

java.lang.Object

org.apache.hadoop.hbase.wal.WALSplitter

@InterfaceAudience.Private
public class WALSplitter
extends Object

Split RegionServer WAL files. Splits the WAL into new files, one per region, to be picked up on Region reopen. Deletes the split WAL when finished. Create an instance and call splitWAL(FileStatus, CancelableProgressable) per file or use static helper methods.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WALSplitter.PipelineController Contains some methods to control WAL-entries producer / consumer interactions

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
static boolean	DEFAULT_WAL_SPLIT_TO_HFILE
protected Map<String,Long>	lastFlushedSequenceIds
protected Map<String,Map<byte[],Long>>	regionMaxSeqIdInStores
protected LastSequenceId	sequenceIdChecker
static boolean	SPLIT_SKIP_ERRORS_DEFAULT By default we retry errors in splitting, rather than skipping.
static String	SPLIT_SKIP_ERRORS_KEY
static String	SPLIT_WAL_BUFFER_SIZE
static String	SPLIT_WAL_WRITER_THREADS
static String	SPLIT_WRITER_CREATION_BOUNDED True if we are to run with bounded amount of writers rather than let the count blossom.
static String	WAL_SPLIT_TO_HFILE Split WAL directly to hfiles instead of into intermediary 'recovered.edits' files.
protected org.apache.hadoop.fs.FileSystem	walFS
protected org.apache.hadoop.fs.Path	walRootDir

Method Summary

Modifier and Type	Method and Description
protected WALProvider.Writer	createWriter(org.apache.hadoop.fs.Path logfile) Create a new WALProvider.Writer for writing log splits.
protected WAL.Reader	getReader(org.apache.hadoop.fs.FileStatus walStatus, boolean skipErrors, CancelableProgressable cancel) Create a new WAL.Reader for reading logs to split.
static List<org.apache.hadoop.fs.Path>	split(org.apache.hadoop.fs.Path walRootDir, org.apache.hadoop.fs.Path walsDir, org.apache.hadoop.fs.Path archiveDir, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.conf.Configuration conf, WALFactory factory) Split a folder of WAL files.
static boolean	splitLogFile(org.apache.hadoop.fs.Path walDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter, LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination, WALFactory factory, RegionServerServices rsServices) Splits a WAL file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPLIT_SKIP_ERRORS_KEY

public static final String SPLIT_SKIP_ERRORS_KEY

See Also:

Constant Field Values

SPLIT_SKIP_ERRORS_DEFAULT

public static final boolean SPLIT_SKIP_ERRORS_DEFAULT

By default we retry errors in splitting, rather than skipping.

See Also:

Constant Field Values

walRootDir

protected final org.apache.hadoop.fs.Path walRootDir

walFS

protected final org.apache.hadoop.fs.FileSystem walFS

conf

protected final org.apache.hadoop.conf.Configuration conf

sequenceIdChecker

protected final LastSequenceId sequenceIdChecker

lastFlushedSequenceIds

protected Map<String,Long> lastFlushedSequenceIds

regionMaxSeqIdInStores

protected Map<String,Map<byte[],Long>> regionMaxSeqIdInStores

WAL_SPLIT_TO_HFILE

public static final String WAL_SPLIT_TO_HFILE

Split WAL directly to hfiles instead of into intermediary 'recovered.edits' files.

See Also:

Constant Field Values

DEFAULT_WAL_SPLIT_TO_HFILE

public static final boolean DEFAULT_WAL_SPLIT_TO_HFILE

See Also:

Constant Field Values

SPLIT_WRITER_CREATION_BOUNDED

public static final String SPLIT_WRITER_CREATION_BOUNDED

True if we are to run with bounded amount of writers rather than let the count blossom. Default is 'false'. Does not apply if you have set 'hbase.wal.split.to.hfile' as that is always bounded. Only applies when you are doing recovery to 'recovered.edits' files (the old default). Bounded writing tends to have higher throughput.

See Also:

Constant Field Values

SPLIT_WAL_BUFFER_SIZE

public static final String SPLIT_WAL_BUFFER_SIZE

See Also:

Constant Field Values

SPLIT_WAL_WRITER_THREADS

public static final String SPLIT_WAL_WRITER_THREADS

See Also:

Constant Field Values

Method Detail

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path walDir,
                                   org.apache.hadoop.fs.FileStatus logfile,
                                   org.apache.hadoop.fs.FileSystem walFS,
                                   org.apache.hadoop.conf.Configuration conf,
                                   CancelableProgressable reporter,
                                   LastSequenceId idChecker,
                                   SplitLogWorkerCoordination splitLogWorkerCoordination,
                                   WALFactory factory,
                                   RegionServerServices rsServices)
                            throws IOException

Splits a WAL file. Used by old SplitLogWorker and tests. Not used by new procedure-based WAL splitter.

Returns:

false if it is interrupted by the progress-able.

Throws:

IOException

split

public static List<org.apache.hadoop.fs.Path> split(org.apache.hadoop.fs.Path walRootDir,
                                                    org.apache.hadoop.fs.Path walsDir,
                                                    org.apache.hadoop.fs.Path archiveDir,
                                                    org.apache.hadoop.fs.FileSystem walFS,
                                                    org.apache.hadoop.conf.Configuration conf,
                                                    WALFactory factory)
                                             throws IOException

Split a folder of WAL files. Delete the directory when done. Used by tools and unit tests. It should be package private. It is public only because TestWALObserver is in a different package, which uses this method to do log splitting.

Returns:

List of output files created by the split.

Throws:

IOException

getReader

protected WAL.Reader getReader(org.apache.hadoop.fs.FileStatus walStatus,
                               boolean skipErrors,
                               CancelableProgressable cancel)
                        throws IOException,
                               org.apache.hadoop.hbase.wal.WALSplitter.CorruptedLogFileException

Create a new WAL.Reader for reading logs to split.

Returns:

Returns null if file has length zero or file can't be found.

Throws:

IOException

org.apache.hadoop.hbase.wal.WALSplitter.CorruptedLogFileException

createWriter

protected WALProvider.Writer createWriter(org.apache.hadoop.fs.Path logfile)
                                   throws IOException

Create a new WALProvider.Writer for writing log splits.

Returns:

a new Writer instance, caller should close

Throws:

IOException