org.apache.hadoop.hive.ql.txn.compactor

Class Worker

java.lang.Object

java.lang.Thread

org.apache.hadoop.hive.ql.txn.compactor.CompactorThread

org.apache.hadoop.hive.ql.txn.compactor.RemoteCompactorThread

org.apache.hadoop.hive.ql.txn.compactor.Worker

All Implemented Interfaces:

Runnable, org.apache.hadoop.conf.Configurable, MetaStoreThread

public class Worker
extends RemoteCompactorThread
implements MetaStoreThread

A class to do compactions. This will run in a separate thread. It will spin on the compaction queue and look for new work to do.

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Thread

Thread.State, Thread.UncaughtExceptionHandler

Field Summary

Fields inherited from class org.apache.hadoop.hive.ql.txn.compactor.RemoteCompactorThread

msc

Fields inherited from class org.apache.hadoop.hive.ql.txn.compactor.CompactorThread

conf, hostName, runtimeVersion, stop

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor and Description
Worker()

Method Summary

Modifier and Type	Method and Description
protected Boolean	findNextCompactionAndExecute(boolean collectGenericStats, boolean collectMrStats) Finds the next compaction and executes it.
CompactorMR	getMrCompactor()
void	init(AtomicBoolean stop)
static boolean	needsCleaning(AcidDirectory dir, org.apache.hadoop.hive.metastore.api.StorageDescriptor sd) Check for obsolete directories, and return true if any exist and Cleaner should be run.
void	run()

Methods inherited from class org.apache.hadoop.hive.ql.txn.compactor.RemoteCompactorThread

getPoolName, setPoolName

Methods inherited from class org.apache.hadoop.hive.ql.txn.compactor.CompactorThread

createLockRequest, getConf, getRuntimeVersion, initializeAndStartThread, isMinorCompactionSupported, replIsCompactionDisabledForTable, resolvePartition, resolveStorageDescriptor, runJobAsSelf, setConf, tableName

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.metastore.MetaStoreThread

start

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Constructor Detail

Worker

public Worker()

Method Detail

run

public void run()

Specified by:

run in interface Runnable

Overrides:

run in class Thread

init

public void init(AtomicBoolean stop)
          throws Exception

Specified by:

init in interface MetaStoreThread

Overrides:

init in class RemoteCompactorThread

Throws:

Exception

needsCleaning

public static boolean needsCleaning(AcidDirectory dir,
                                    org.apache.hadoop.hive.metastore.api.StorageDescriptor sd)

Check for obsolete directories, and return true if any exist and Cleaner should be run. For example if we insert overwrite into a table with only deltas, a new base file with the highest writeId is created so there will be no live delta directories, only obsolete ones. Compaction is not needed, but the cleaner should still be run.

Returns:

true if cleaning is needed

findNextCompactionAndExecute

protected Boolean findNextCompactionAndExecute(boolean collectGenericStats,
                                               boolean collectMrStats)

Finds the next compaction and executes it. The main thread might interrupt the execution of this method in case of timeout.

Parameters:

collectGenericStats - If true then for both MR and Query based compaction the stats are regenerated

collectMrStats - If true then for MR compaction the stats are regenerated

Returns:

Returns true, if there was compaction in the queue, and we started working on it.

getMrCompactor

public CompactorMR getMrCompactor()