org.apache.hadoop.hive.ql.exec.tez

Class SplitGrouper

java.lang.Object

org.apache.hadoop.hive.ql.exec.tez.SplitGrouper

public class SplitGrouper
extends Object

SplitGrouper is used to combine splits based on head room and locality. It also enforces restrictions around schema, file format and bucketing.

Constructor Summary

Constructors

Constructor and Description
SplitGrouper()

Method Summary

Modifier and Type	Method and Description
List<org.apache.tez.dag.api.TaskLocationHint>	createTaskLocationHints(org.apache.hadoop.mapred.InputSplit[] splits, boolean consistentLocations) Create task location hints from a set of input splits
com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit>	generateGroupedSplits(org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapred.InputSplit[] splits, float waves, int availableSlots, org.apache.hadoop.mapred.split.SplitLocationProvider locationProvider) Generate groups of splits, separated by schema evolution boundaries
com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit>	generateGroupedSplits(org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapred.InputSplit[] splits, float waves, int availableSlots, String inputName, boolean groupAcrossFiles, org.apache.hadoop.mapred.split.SplitLocationProvider locationProvider) Generate groups of splits, separated by schema evolution boundaries OR When used from compactor, group splits based on the bucket number of the input files (in this case, splits for same logical bucket but different schema, end up in same group)
com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit>	group(org.apache.hadoop.conf.Configuration conf, com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> bucketSplitMultimap, int availableSlots, float waves, org.apache.hadoop.mapred.split.SplitLocationProvider splitLocationProvider) group splits for each bucket separately - while evenly filling all the available slots with tasks

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SplitGrouper

public SplitGrouper()

Method Detail

group

public com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> group(org.apache.hadoop.conf.Configuration conf,
                                                                                             com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> bucketSplitMultimap,
                                                                                             int availableSlots,
                                                                                             float waves,
                                                                                             org.apache.hadoop.mapred.split.SplitLocationProvider splitLocationProvider)
                                                                                      throws IOException

group splits for each bucket separately - while evenly filling all the available slots with tasks

Throws:

IOException

createTaskLocationHints

public List<org.apache.tez.dag.api.TaskLocationHint> createTaskLocationHints(org.apache.hadoop.mapred.InputSplit[] splits,
                                                                             boolean consistentLocations)
                                                                      throws IOException

Create task location hints from a set of input splits

Parameters:

splits - the actual splits

consistentLocations - whether to re-order locations for each split, if it's a file split

Returns:

taskLocationHints - 1 per input split specified

Throws:

IOException

generateGroupedSplits

public com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> generateGroupedSplits(org.apache.hadoop.mapred.JobConf jobConf,
                                                                                                             org.apache.hadoop.conf.Configuration conf,
                                                                                                             org.apache.hadoop.mapred.InputSplit[] splits,
                                                                                                             float waves,
                                                                                                             int availableSlots,
                                                                                                             org.apache.hadoop.mapred.split.SplitLocationProvider locationProvider)
                                                                                                      throws Exception

Generate groups of splits, separated by schema evolution boundaries

Throws:

Exception

generateGroupedSplits

public com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> generateGroupedSplits(org.apache.hadoop.mapred.JobConf jobConf,
                                                                                                             org.apache.hadoop.conf.Configuration conf,
                                                                                                             org.apache.hadoop.mapred.InputSplit[] splits,
                                                                                                             float waves,
                                                                                                             int availableSlots,
                                                                                                             String inputName,
                                                                                                             boolean groupAcrossFiles,
                                                                                                             org.apache.hadoop.mapred.split.SplitLocationProvider locationProvider)
                                                                                                      throws Exception

Generate groups of splits, separated by schema evolution boundaries OR When used from compactor, group splits based on the bucket number of the input files (in this case, splits for same logical bucket but different schema, end up in same group)

Throws:

Exception