org.apache.hadoop.hive.ql.io.orc

Class OrcSplit

java.lang.Object

org.apache.hadoop.mapreduce.InputSplit

org.apache.hadoop.mapred.FileSplit

org.apache.hadoop.hive.ql.io.orc.OrcSplit

All Implemented Interfaces:

ColumnarSplit, LlapAwareSplit, org.apache.hadoop.io.Writable, org.apache.hadoop.mapred.InputSplit, org.apache.hadoop.mapred.InputSplitWithLocationInfo

public class OrcSplit
extends org.apache.hadoop.mapred.FileSplit
implements ColumnarSplit, LlapAwareSplit

OrcFileSplit. Holds file meta info

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	OrcSplit.OffsetAndBucketProperty Used for generating synthetic ROW__IDs for reading "original" files.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	OrcSplit()
	OrcSplit(org.apache.hadoop.fs.Path path, Object fileId, long offset, long length, String[] hosts, org.apache.orc.impl.OrcTail orcTail, boolean isOriginal, boolean hasBase, List<AcidInputFormat.DeltaMetaData> deltas, long projectedDataSize, long fileLen, org.apache.hadoop.fs.Path rootDir, OrcSplit.OffsetAndBucketProperty syntheticAcidProps)

Method Summary

Modifier and Type	Method and Description
boolean	canUseLlapIo(org.apache.hadoop.conf.Configuration conf)
int	getBucketId() Note: this is the bucket number as seen in the file name that contains this split.
long	getColumnarProjectionSize() Return the estimation size of the column projections that will be read from this split.
List<AcidInputFormat.DeltaMetaData>	getDeltas()
Object	getFileKey()
long	getFileLength()
org.apache.orc.impl.OrcTail	getOrcTail()
long	getProjectedColumnsUncompressedSize()
org.apache.hadoop.fs.Path	getRootDir()
int	getStatementId()
OrcSplit.OffsetAndBucketProperty	getSyntheticAcidProps()
long	getWriteId() Note: this is the write id as seen in the file name that contains this split For files that have min/max writeId, this is the starting one.
boolean	hasBase()
boolean	hasFooter()
boolean	isAcid() If this method returns true, then for sure it is ACID.
boolean	isOriginal()
void	parse(org.apache.hadoop.conf.Configuration conf)
void	parse(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path rootPath)
void	readFields(DataInput in)
String	toString()
void	write(DataOutput out)

Methods inherited from class org.apache.hadoop.mapred.FileSplit

getLength, getLocationInfo, getLocations, getPath, getStart

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

OrcSplit

protected OrcSplit()

OrcSplit

public OrcSplit(org.apache.hadoop.fs.Path path,
                Object fileId,
                long offset,
                long length,
                String[] hosts,
                org.apache.orc.impl.OrcTail orcTail,
                boolean isOriginal,
                boolean hasBase,
                List<AcidInputFormat.DeltaMetaData> deltas,
                long projectedDataSize,
                long fileLen,
                org.apache.hadoop.fs.Path rootDir,
                OrcSplit.OffsetAndBucketProperty syntheticAcidProps)

Method Detail

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Overrides:

write in class org.apache.hadoop.mapred.FileSplit

Throws:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Overrides:

readFields in class org.apache.hadoop.mapred.FileSplit

Throws:

IOException

getOrcTail

public org.apache.orc.impl.OrcTail getOrcTail()

hasFooter

public boolean hasFooter()

isOriginal

public boolean isOriginal()

Returns:

true if file schema doesn't have Acid metadata columns Such file may be in a delta_x_y/ or base_x due to being added via "load data" command. It could be at partition|table root due to table having been converted from non-acid to acid table. It could even be something like "warehouse/t/HIVE_UNION_SUBDIR_15/000000_0" if it was written by an "insert into t select ... from A union all select ... from B"

hasBase

public boolean hasBase()

getRootDir

public org.apache.hadoop.fs.Path getRootDir()

getDeltas

public List<AcidInputFormat.DeltaMetaData> getDeltas()

getFileLength

public long getFileLength()

isAcid

public boolean isAcid()

If this method returns true, then for sure it is ACID. However, if it returns false.. it could be ACID or non-ACID.

Returns:

getProjectedColumnsUncompressedSize

public long getProjectedColumnsUncompressedSize()

getFileKey

public Object getFileKey()

getSyntheticAcidProps

public OrcSplit.OffsetAndBucketProperty getSyntheticAcidProps()

getColumnarProjectionSize

public long getColumnarProjectionSize()

Description copied from interface: ColumnarSplit

Return the estimation size of the column projections that will be read from this split.

Specified by:

getColumnarProjectionSize in interface ColumnarSplit

Returns:

- estimated column projection size that will be read in bytes

canUseLlapIo

public boolean canUseLlapIo(org.apache.hadoop.conf.Configuration conf)

Specified by:

canUseLlapIo in interface LlapAwareSplit

getWriteId

public long getWriteId()

Note: this is the write id as seen in the file name that contains this split For files that have min/max writeId, this is the starting one.

Returns:

getStatementId

public int getStatementId()

getBucketId

public int getBucketId()

Note: this is the bucket number as seen in the file name that contains this split. Hive 3.0 encodes a bunch of info in the Acid schema's bucketId attribute. See: BucketCodec.V1 for details.

Returns:

parse

public void parse(org.apache.hadoop.conf.Configuration conf)
           throws IOException

Throws:

IOException

parse

public void parse(org.apache.hadoop.conf.Configuration conf,
                  org.apache.hadoop.fs.Path rootPath)
           throws IOException

Throws:

IOException

toString

public String toString()

Overrides:

toString in class org.apache.hadoop.mapred.FileSplit