org.apache.hadoop.hive.ql.io

Class HiveContextAwareRecordReader<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader<K,V>

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.mapred.RecordReader<K,V>

Direct Known Subclasses:

BucketizedHiveRecordReader, CombineHiveRecordReader, HiveRecordReader

public abstract class HiveContextAwareRecordReader<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
extends Object
implements org.apache.hadoop.mapred.RecordReader<K,V>

This class prepares an IOContext, and provides the ability to perform a binary search on the data. The binary search can be used by setting the value of inputFormatSorted in the MapreduceWork to true, but it should only be used if the data is going to a FilterOperator, which filters by comparing a value in the data with a constant, using one of the comparisons =, <, >, <=, >=. If the RecordReader's underlying format is an RCFile, this object can perform a binary search to find the block to begin reading from, and stop reading once it can be determined no other entries will match the filter.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	isSorted
protected org.apache.hadoop.mapred.JobConf	jobConf
protected org.apache.hadoop.mapred.RecordReader<K,V>	recordReader

Constructor Summary

Constructors

Constructor and Description
HiveContextAwareRecordReader(org.apache.hadoop.mapred.JobConf conf)
HiveContextAwareRecordReader(org.apache.hadoop.mapred.RecordReader<K,V> recordReader)
HiveContextAwareRecordReader(org.apache.hadoop.mapred.RecordReader<K,V> recordReader, org.apache.hadoop.mapred.JobConf conf)

Method Summary

Modifier and Type	Method and Description
void	close()
abstract void	doClose() Close this InputSplit to future operations.
boolean	doNext(K key, V value)
boolean	foundAllTargets() Returns true if the current comparison is in the list of stop comparisons, i.e.
protected FooterBuffer	getFooterBuffer()
IOContext	getIOContext()
float	getProgress()
void	initIOContext(org.apache.hadoop.mapred.FileSplit split, org.apache.hadoop.mapred.JobConf job, Class inputFormatClass)
void	initIOContext(org.apache.hadoop.mapred.FileSplit split, org.apache.hadoop.mapred.JobConf job, Class inputFormatClass, org.apache.hadoop.mapred.RecordReader recordReader)
void	initIOContextSortedProps(org.apache.hadoop.mapred.FileSplit split, org.apache.hadoop.mapred.RecordReader recordReader, org.apache.hadoop.mapred.JobConf job)
boolean	next(K key, V value)
protected void	setFooterBuffer(FooterBuffer buf)
void	setRecordReader(org.apache.hadoop.mapred.RecordReader<K,V> recordReader)
protected void	updateIOContext()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.mapred.RecordReader

createKey, createValue, getPos

Field Detail

recordReader

protected org.apache.hadoop.mapred.RecordReader<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable> recordReader

jobConf

protected org.apache.hadoop.mapred.JobConf jobConf

isSorted

protected boolean isSorted

Constructor Detail

HiveContextAwareRecordReader

public HiveContextAwareRecordReader(org.apache.hadoop.mapred.JobConf conf)
                             throws IOException

Throws:

IOException

HiveContextAwareRecordReader

public HiveContextAwareRecordReader(org.apache.hadoop.mapred.RecordReader<K,V> recordReader)

HiveContextAwareRecordReader

public HiveContextAwareRecordReader(org.apache.hadoop.mapred.RecordReader<K,V> recordReader,
                                    org.apache.hadoop.mapred.JobConf conf)
                             throws IOException

Throws:

IOException

Method Detail

setRecordReader

public void setRecordReader(org.apache.hadoop.mapred.RecordReader<K,V> recordReader)

doClose

public abstract void doClose()
                      throws IOException

Close this InputSplit to future operations.

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.hadoop.mapred.RecordReader<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Throws:

IOException

next

public boolean next(K key,
                    V value)
             throws IOException

Specified by:

next in interface org.apache.hadoop.mapred.RecordReader<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Throws:

IOException

updateIOContext

protected void updateIOContext()
                        throws IOException

Throws:

IOException

getIOContext

public IOContext getIOContext()

initIOContext

public void initIOContext(org.apache.hadoop.mapred.FileSplit split,
                          org.apache.hadoop.mapred.JobConf job,
                          Class inputFormatClass)
                   throws IOException

Throws:

IOException

initIOContext

public void initIOContext(org.apache.hadoop.mapred.FileSplit split,
                          org.apache.hadoop.mapred.JobConf job,
                          Class inputFormatClass,
                          org.apache.hadoop.mapred.RecordReader recordReader)
                   throws IOException

Throws:

IOException

initIOContextSortedProps

public void initIOContextSortedProps(org.apache.hadoop.mapred.FileSplit split,
                                     org.apache.hadoop.mapred.RecordReader recordReader,
                                     org.apache.hadoop.mapred.JobConf job)

getProgress

public float getProgress()
                  throws IOException

Specified by:

getProgress in interface org.apache.hadoop.mapred.RecordReader<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Throws:

IOException

getFooterBuffer

protected FooterBuffer getFooterBuffer()

setFooterBuffer

protected void setFooterBuffer(FooterBuffer buf)

doNext

public boolean doNext(K key,
                      V value)
               throws IOException

Throws:

IOException

foundAllTargets

public boolean foundAllTargets()

Returns true if the current comparison is in the list of stop comparisons, i.e. we've found all records which won't be filtered

Returns:

true if the current comparison is found