org.apache.hadoop.hive.ql.txn.compactor

Class CompactorThread

java.lang.Object

java.lang.Thread

org.apache.hadoop.hive.ql.txn.compactor.CompactorThread

All Implemented Interfaces:

Runnable, org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

MetaStoreCompactorThread, RemoteCompactorThread

public abstract class CompactorThread
extends Thread
implements org.apache.hadoop.conf.Configurable

Superclass for all threads in the compactor.

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Thread

Thread.State, Thread.UncaughtExceptionHandler

Field Summary

Fields

Modifier and Type	Field and Description
protected HiveConf	conf
protected String	hostName
protected static org.slf4j.Logger	LOG
protected String	runtimeVersion
protected AtomicBoolean	stop

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor and Description
CompactorThread()

Method Summary

Modifier and Type	Method and Description
protected org.apache.hadoop.hive.metastore.api.LockRequest	createLockRequest(CompactionInfo ci, long txnId, org.apache.hadoop.hive.metastore.api.LockType lockType, org.apache.hadoop.hive.metastore.api.DataOperationType opType)
org.apache.hadoop.conf.Configuration	getConf()
protected String	getRuntimeVersion()
void	init(AtomicBoolean stop)
static void	initializeAndStartThread(CompactorThread thread, org.apache.hadoop.conf.Configuration conf)
protected boolean	isMinorCompactionSupported(Map<String,String> tblproperties, AcidDirectory dir) Check for that special case when minor compaction is supported or not.
protected boolean	replIsCompactionDisabledForTable(org.apache.hadoop.hive.metastore.api.Table tbl)
protected org.apache.hadoop.hive.metastore.api.Partition	resolvePartition(CompactionInfo ci) Get the partition being compacted.
protected org.apache.hadoop.hive.metastore.api.StorageDescriptor	resolveStorageDescriptor(org.apache.hadoop.hive.metastore.api.Table t, org.apache.hadoop.hive.metastore.api.Partition p) Get the storage descriptor for a compaction.
protected boolean	runJobAsSelf(String owner) Determine whether to run this job as the current user or whether we need a doAs to switch users.
void	setConf(org.apache.hadoop.conf.Configuration configuration)
protected String	tableName(org.apache.hadoop.hive.metastore.api.Table t)

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, run, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

conf

protected HiveConf conf

stop

protected AtomicBoolean stop

hostName

protected String hostName

runtimeVersion

protected String runtimeVersion

Constructor Detail

CompactorThread

public CompactorThread()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration configuration)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

init

public void init(AtomicBoolean stop)
          throws Exception

Throws:

Exception

resolvePartition

protected org.apache.hadoop.hive.metastore.api.Partition resolvePartition(CompactionInfo ci)
                                                                   throws org.apache.hadoop.hive.metastore.api.MetaException

Get the partition being compacted.

Parameters:

ci - compaction info returned from the compaction queue

Returns:

metastore partition, or null if there is not partition in this compaction info

Throws:

org.apache.hadoop.hive.metastore.api.MetaException - if underlying calls throw, or if the partition name resolves to more than one partition.

isMinorCompactionSupported

protected boolean isMinorCompactionSupported(Map<String,String> tblproperties,
                                             AcidDirectory dir)

Check for that special case when minor compaction is supported or not.

• The table is Insert-only OR
• Query based compaction is not enabled OR
• The table has only acid data in it.

Parameters:

tblproperties - The properties of the table to check

dir - The AcidDirectory instance pointing to the table's folder on the filesystem.

Returns:

Returns true if minor compaction is supported based on the given parameters, false otherwise.

resolveStorageDescriptor

protected org.apache.hadoop.hive.metastore.api.StorageDescriptor resolveStorageDescriptor(org.apache.hadoop.hive.metastore.api.Table t,
                                                                                          org.apache.hadoop.hive.metastore.api.Partition p)

Get the storage descriptor for a compaction.

Parameters:

t - table from resolveTable(org.apache.hadoop.hive.metastore.txn.CompactionInfo)

p - table from resolvePartition(org.apache.hadoop.hive.metastore.txn.CompactionInfo)

Returns:

metastore storage descriptor.

runJobAsSelf

protected boolean runJobAsSelf(String owner)

Determine whether to run this job as the current user or whether we need a doAs to switch users.

Parameters:

owner - of the directory we will be working in, as determined by TxnUtils.findUserToRunAs(String, Table, Configuration)

Returns:

true if the job should run as the current user, false if a doAs is needed.

tableName

protected String tableName(org.apache.hadoop.hive.metastore.api.Table t)

initializeAndStartThread

public static void initializeAndStartThread(CompactorThread thread,
                                            org.apache.hadoop.conf.Configuration conf)
                                     throws Exception

Throws:

Exception

replIsCompactionDisabledForTable

protected boolean replIsCompactionDisabledForTable(org.apache.hadoop.hive.metastore.api.Table tbl)

getRuntimeVersion

protected String getRuntimeVersion()

createLockRequest

protected org.apache.hadoop.hive.metastore.api.LockRequest createLockRequest(CompactionInfo ci,
                                                                             long txnId,
                                                                             org.apache.hadoop.hive.metastore.api.LockType lockType,
                                                                             org.apache.hadoop.hive.metastore.api.DataOperationType opType)