org.apache.hadoop.hive.ql.io.orc

Class VectorizedOrcAcidRowBatchReader

java.lang.Object

org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch>

public class VectorizedOrcAcidRowBatchReader
extends Object
implements org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch>

A fast vectorized batch reader class for ACID. Insert events are read directly from the base files/insert_only deltas in vectorized row batches. The deleted rows can then be easily indicated via the 'selected' field of the vectorized row batch. Refer HIVE-14233 for more details.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	VectorizedOrcAcidRowBatchReader.DeleteEventRegistry An interface that can determine which rows have been deleted from a given vectorized row batch.

Field Summary

Fields

Modifier and Type	Field and Description
protected Object[]	partitionValues
protected float	progress

Constructor Summary

Constructors

Constructor and Description
VectorizedOrcAcidRowBatchReader(OrcSplit inputSplit, org.apache.hadoop.mapred.JobConf conf, org.apache.hadoop.mapred.Reporter reporter, org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch> baseReader, VectorizedRowBatchCtx rbCtx, boolean isFlatPayload, MapWork mapWork) LLAP IO c'tor

Method Summary

Modifier and Type	Method and Description
void	close()
org.apache.hadoop.io.NullWritable	createKey()
VectorizedRowBatch	createValue()
long	getPos()
float	getProgress()
boolean	includeAcidColumns()
boolean	next(org.apache.hadoop.io.NullWritable key, VectorizedRowBatch value) There are 2 types of schema from the baseReader that this handles.
void	setBaseAndInnerReader(org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch> baseReader)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

progress

protected float progress

partitionValues

protected Object[] partitionValues

Constructor Detail

VectorizedOrcAcidRowBatchReader

public VectorizedOrcAcidRowBatchReader(OrcSplit inputSplit,
                                       org.apache.hadoop.mapred.JobConf conf,
                                       org.apache.hadoop.mapred.Reporter reporter,
                                       org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch> baseReader,
                                       VectorizedRowBatchCtx rbCtx,
                                       boolean isFlatPayload,
                                       MapWork mapWork)
                                throws IOException

LLAP IO c'tor

Throws:

IOException

Method Detail

includeAcidColumns

public boolean includeAcidColumns()

setBaseAndInnerReader

public void setBaseAndInnerReader(org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch> baseReader)

next

public boolean next(org.apache.hadoop.io.NullWritable key,
                    VectorizedRowBatch value)
             throws IOException

There are 2 types of schema from the baseReader that this handles. In the case the data was written to a transactional table from the start, every row is decorated with transaction related info and looks like <op, owid, writerId, rowid, cwid, <f1, ... fn>>. The other case is when data was written to non-transactional table and thus only has the user data: <f1, ... fn>. Then this table was then converted to a transactional table but the data files are not changed until major compaction. These are the "original" files. In this case we may need to decorate the outgoing data with transactional column values at read time. (It's done somewhat out of band via VectorizedRowBatchCtx - ask Teddy Choi). The "owid, writerId, rowid" columns represent RecordIdentifier. They are assigned each time the table is read in a way that needs to project VirtualColumn.ROWID. Major compaction will attach these values to each row permanently. It's critical that these generated column values are assigned exactly the same way by each read of the same row and by the Compactor. See CompactorMR and OrcRawRecordMerger.OriginalReaderPairToCompact for the Compactor read path. (Longer term should make compactor use this class) This only decorates original rows with metadata if something above is requesting these values or if there are Delete events to apply.

Specified by:

next in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch>

Returns:

false where there is no more data, i.e. value is empty

Throws:

IOException

createKey

public org.apache.hadoop.io.NullWritable createKey()

Specified by:

createKey in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch>

createValue

public VectorizedRowBatch createValue()

Specified by:

createValue in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch>

getPos

public long getPos()
            throws IOException

Specified by:

getPos in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch>

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch>

Throws:

IOException

getProgress

public float getProgress()
                  throws IOException

Specified by:

getProgress in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,VectorizedRowBatch>

Throws:

IOException