org.apache.hadoop.hive.ql.txn.compactor

Class CompactorMR

java.lang.Object

org.apache.hadoop.hive.ql.txn.compactor.CompactorMR

public class CompactorMR
extends Object

Class to do compactions via an MR job. This has to be in the ql package rather than metastore .compactions package with all of it's relatives because it needs access to the actual input and output formats, which are in ql. ql depends on metastore and we can't have a circular dependency.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CompactorMR.CompactorMap<V extends org.apache.hadoop.io.Writable>

Constructor Summary

Constructors

Constructor and Description
CompactorMR()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapred.JobConf	createBaseJobConf(HiveConf conf, String jobName, org.apache.hadoop.hive.metastore.api.Table t, org.apache.hadoop.hive.metastore.api.StorageDescriptor sd, ValidWriteIdList writeIds, CompactionInfo ci)
org.apache.hadoop.mapred.JobConf	getMrJob()
void	run(HiveConf conf, String jobName, org.apache.hadoop.hive.metastore.api.Table t, org.apache.hadoop.hive.metastore.api.Partition p, org.apache.hadoop.hive.metastore.api.StorageDescriptor sd, ValidWriteIdList writeIds, CompactionInfo ci, IMetaStoreClient msc, AcidDirectory dir) Run Compaction which may consist of several jobs on the cluster.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CompactorMR

public CompactorMR()

Method Detail

createBaseJobConf

public org.apache.hadoop.mapred.JobConf createBaseJobConf(HiveConf conf,
                                                          String jobName,
                                                          org.apache.hadoop.hive.metastore.api.Table t,
                                                          org.apache.hadoop.hive.metastore.api.StorageDescriptor sd,
                                                          ValidWriteIdList writeIds,
                                                          CompactionInfo ci)

run

public void run(HiveConf conf,
                String jobName,
                org.apache.hadoop.hive.metastore.api.Table t,
                org.apache.hadoop.hive.metastore.api.Partition p,
                org.apache.hadoop.hive.metastore.api.StorageDescriptor sd,
                ValidWriteIdList writeIds,
                CompactionInfo ci,
                IMetaStoreClient msc,
                AcidDirectory dir)
         throws IOException

Run Compaction which may consist of several jobs on the cluster.

Parameters:

conf - Hive configuration file

jobName - name to run this job with

t - metastore table

sd - metastore storage descriptor

writeIds - list of valid write ids

ci - CompactionInfo

Throws:

IOException - if the job fails

getMrJob

public org.apache.hadoop.mapred.JobConf getMrJob()