org.apache.hadoop.hive.ql.txn.compactor

Class MRCompactor

java.lang.Object

org.apache.hadoop.hive.ql.txn.compactor.MRCompactor

All Implemented Interfaces:

Compactor

public class MRCompactor
extends Object
implements Compactor

Class to do compactions via an MR job. This has to be in the ql package rather than metastore .compactions package with all of it's relatives because it needs access to the actual input and output formats, which are in ql. ql depends on metastore and we can't have a circular dependency.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MRCompactor.CompactorMap<V extends org.apache.hadoop.io.Writable>

Field Summary

Fields inherited from interface org.apache.hadoop.hive.ql.txn.compactor.Compactor

FINAL_LOCATION

Constructor Summary

Constructors

Constructor and Description
MRCompactor(IMetaStoreClient msc)

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapred.JobConf	createBaseJobConf(HiveConf conf, String jobName, org.apache.hadoop.hive.metastore.api.Table t, org.apache.hadoop.hive.metastore.api.StorageDescriptor sd, ValidWriteIdList writeIds, CompactionInfo ci)
org.apache.hadoop.mapred.JobConf	getMrJob()
boolean	run(CompactorContext context) Start a compaction.
void	run(HiveConf conf, org.apache.hadoop.hive.metastore.api.Table t, org.apache.hadoop.hive.metastore.api.StorageDescriptor sd, ValidWriteIdList writeIds, CompactionInfo ci, AcidDirectory dir) Run Compaction which may consist of several jobs on the cluster.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.ql.txn.compactor.Compactor

getCompactorTxnId

Constructor Detail

MRCompactor

public MRCompactor(IMetaStoreClient msc)

Method Detail

run

public boolean run(CompactorContext context)
            throws IOException,
                   HiveException,
                   InterruptedException

Description copied from interface: Compactor

Start a compaction.

Specified by:

run in interface Compactor

Parameters:

context - CompactionContext which contains information to perform compaction.

Throws:

IOException - compaction cannot be finished.

HiveException

InterruptedException

createBaseJobConf

public org.apache.hadoop.mapred.JobConf createBaseJobConf(HiveConf conf,
                                                          String jobName,
                                                          org.apache.hadoop.hive.metastore.api.Table t,
                                                          org.apache.hadoop.hive.metastore.api.StorageDescriptor sd,
                                                          ValidWriteIdList writeIds,
                                                          CompactionInfo ci)

run

public void run(HiveConf conf,
                org.apache.hadoop.hive.metastore.api.Table t,
                org.apache.hadoop.hive.metastore.api.StorageDescriptor sd,
                ValidWriteIdList writeIds,
                CompactionInfo ci,
                AcidDirectory dir)
         throws IOException

Run Compaction which may consist of several jobs on the cluster.

Parameters:

conf - Hive configuration file

t - metastore table

sd - metastore storage descriptor

writeIds - list of valid write ids

ci - CompactionInfo

Throws:

IOException - if the job fails

getMrJob

public org.apache.hadoop.mapred.JobConf getMrJob()