org.apache.hadoop.hive.llap.io.api

Interface LlapIo<T>

public interface LlapIo<T>

Method Summary

Modifier and Type	Method and Description
void	close()
long	evictEntity(LlapDaemonProtocolProtos.EvictEntityRequestProto protoRequest) Handles request to evict entities specified in the request object.
LlapDaemonProtocolProtos.CacheEntryList	fetchCachedContentInfo() Extract and return the cache content metadata.
org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,T>	getInputFormat(org.apache.hadoop.mapred.InputFormat<?,?> sourceInputFormat, Deserializer serde)
String	getMemoryInfo()
org.apache.orc.impl.OrcTail	getOrcTailFromCache(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, CacheTag tag, Object fileKey) Returns a deserialized OrcTail instance associated with the ORC file on the given path.
org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers	getParquetFooterBuffersFromCache(org.apache.hadoop.fs.Path path, org.apache.hadoop.mapred.JobConf conf, Object fileKey) Returns the metadata buffers associated with the Parquet file on the given path.
void	initCacheOnlyInputFormat(org.apache.hadoop.mapred.InputFormat<?,?> inputFormat)
org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch>	llapVectorizedOrcReaderForPath(Object fileKey, org.apache.hadoop.fs.Path path, CacheTag tag, List<Integer> tableIncludedCols, org.apache.hadoop.mapred.JobConf conf, long offset, long length, org.apache.hadoop.mapred.Reporter reporter) Creates an LLAP record reader for a given file, by creating a split from this file, and passing it into a new LLAP record reader.
void	loadDataIntoCache(LlapDaemonProtocolProtos.CacheEntryList metadata) Load the actual data into the cache based on the provided metadata.
long	purge() purge is best effort and will just release the buffers that are unlocked (refCount == 0).
boolean	usingLowLevelCache()

Method Detail

getInputFormat

org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,T> getInputFormat(org.apache.hadoop.mapred.InputFormat<?,?> sourceInputFormat,
                                                                                         Deserializer serde)

close

void close()

getMemoryInfo

String getMemoryInfo()

purge

long purge()

purge is best effort and will just release the buffers that are unlocked (refCount == 0). This is typically called when the system is idle.

getOrcTailFromCache

org.apache.orc.impl.OrcTail getOrcTailFromCache(org.apache.hadoop.fs.Path path,
                                                org.apache.hadoop.conf.Configuration conf,
                                                CacheTag tag,
                                                @Nullable
                                                Object fileKey)
                                         throws IOException

Returns a deserialized OrcTail instance associated with the ORC file on the given path. Raw content is either obtained from cache, or from disk if there is a cache miss.

Parameters:

path - Orc file path

conf - jobConf

tag - a CacheTag instance must be provided as that's needed for cache insertion

fileKey - fileId of the ORC file (either the Long fileId of HDFS or the SyntheticFileId). Optional, if it is not provided, it will be generated, see: org.apache.hadoop.hive.ql.io.HdfsUtils#getFileId()

Returns:

The tail of the ORC file

Throws:

IOException - ex

getParquetFooterBuffersFromCache

org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers getParquetFooterBuffersFromCache(org.apache.hadoop.fs.Path path,
                                                                                                org.apache.hadoop.mapred.JobConf conf,
                                                                                                @Nullable
                                                                                                Object fileKey)
                                                                                         throws IOException

Returns the metadata buffers associated with the Parquet file on the given path. Content is either obtained from cache, or from disk if there is a cache miss.

Parameters:

path - Parquet file path

conf - jobConf

fileKey - fileId of the Parquet file (either the Long fileId of HDFS or the SyntheticFileId). Optional, if it is not provided, it will be generated, see: org.apache.hadoop.hive.ql.io.HdfsUtils#getFileId()

Returns:

Throws:

IOException

evictEntity

long evictEntity(LlapDaemonProtocolProtos.EvictEntityRequestProto protoRequest)

Handles request to evict entities specified in the request object.

Parameters:

protoRequest - lists Hive entities (DB, table, etc..) whose LLAP buffers should be evicted.

Returns:

number of evicted bytes.

initCacheOnlyInputFormat

void initCacheOnlyInputFormat(org.apache.hadoop.mapred.InputFormat<?,?> inputFormat)

llapVectorizedOrcReaderForPath

org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch> llapVectorizedOrcReaderForPath(Object fileKey,
                                                                                                                                                                 org.apache.hadoop.fs.Path path,
                                                                                                                                                                 CacheTag tag,
                                                                                                                                                                 List<Integer> tableIncludedCols,
                                                                                                                                                                 org.apache.hadoop.mapred.JobConf conf,
                                                                                                                                                                 long offset,
                                                                                                                                                                 long length,
                                                                                                                                                                 org.apache.hadoop.mapred.Reporter reporter)
                                                                                                                                                          throws IOException

Creates an LLAP record reader for a given file, by creating a split from this file, and passing it into a new LLAP record reader. May return null when attempting with unsupported schema evolution between reader and file schemas. Useful if a file is read multiple times as LLAP will cache it. Should be used for rather smaller files.

Parameters:

fileKey - - file ID, if null it will be determined during read but for additional performance cost

path - - path for the file to read

tag - - cache tag associated with this file (required for LLAP administrative purposes)

tableIncludedCols - - list of column #'s to be read from the file

conf - - job conf for this read. Schema serialized herein should be aligned with tableIncludedCols

offset - - required offset to start reading from

length - - required reading length

Returns:

Throws:

IOException

fetchCachedContentInfo

LlapDaemonProtocolProtos.CacheEntryList fetchCachedContentInfo()

Extract and return the cache content metadata.

loadDataIntoCache

void loadDataIntoCache(LlapDaemonProtocolProtos.CacheEntryList metadata)

Load the actual data into the cache based on the provided metadata.

usingLowLevelCache

boolean usingLowLevelCache()