org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast

Class VectorMapJoinFastBytesHashMapStore

java.lang.Object

org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMapStore

All Implemented Interfaces:

MemoryEstimate

public class VectorMapJoinFastBytesHashMapStore
extends Object
implements MemoryEstimate

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	VectorMapJoinFastBytesHashMapStore.HashMapResult A hash map result that can read values stored by the key and value store, one-by-one.

Constructor Summary

Constructors

Constructor and Description
VectorMapJoinFastBytesHashMapStore(int writeBuffersSize)

Method Summary

Modifier and Type	Method and Description
long	addFirst(long partialHashCode, byte[] keyBytes, int keyStart, int keyLength, byte[] valueBytes, int valueStart, int valueLength) Two 64-bit long result is the key and value reference.
long	addMore(long refWord, byte[] valueBytes, int valueStart, int valueLength, WriteBuffers.Position unsafeReadPos)
long	getEstimatedMemorySize()
void	getKey(long refWord, WriteBuffers.ByteSegmentRef keyByteSegmentRef, WriteBuffers.Position readPos)
WriteBuffers	getWriteBuffers() A store for a key and a list of 1 or more arbitrary length values in memory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

VectorMapJoinFastBytesHashMapStore

public VectorMapJoinFastBytesHashMapStore(int writeBuffersSize)

Method Detail

getWriteBuffers

public WriteBuffers getWriteBuffers()

A store for a key and a list of 1 or more arbitrary length values in memory. The memory is a "infinite" byte array as a WriteBuffers object. We give the client (e.g. hash map logic) a 64-bit key and value reference to keep that has the offset within the "infinite" byte array of the key. The 64 bits includes about half of the upper hash code to help during matching. We optimize the common case when the key length is short and store that information in the 64 bit reference. When there are more than 1 value, the zero padding is overwritten with a relative offset to the next value. The next value always includes the value length. Cases: 1) One element when key and is small (and stored in the reference word): Key and Value Reference | | absoluteOffset | --------------------------------- | v <5 0's for Next Relative Offset> <Key Bytes> <Value Length> <Value Bytes> NEXT (NONE) KEY VALUE NOTE: AbsoluteOffset.byteLength = 5 2) One element, general: shows optional big key length. Key and Value Reference | | absoluteOffset | --------------------------------- | v <5 0's for Next Relative Offset> [Big Key Length] <Key Bytes> <Value Length> <Value Bytes> NEXT (NONE) optional KEY VALUE 3) Two elements when key length is small and stored in reference word: Key and Value Reference | | absoluteOffset | ------------------------------------ | v <Next Value Rel Offset as 5 bytes> <Key Bytes> <Value Bytes> | NEXT KEY VALUE | | first record absolute offset + relative offset | -------- | v <5 0's Padding for Next Value Ref> <Value Length> <Value Bytes> NEXT (NONE) VALUE 4) Three elements showing how first record updated to point to new value and new value points to most recent (additional) value: Key and Value Reference | | absoluteOffset | ------------------------------------ | v <Next Value Rel Offset as 5 bytes> <Key Bytes> <Value Bytes> | NEXT KEY VALUE | | first record absolute offset + relative offset | | | <5 0's Padding for Next Value Ref> <Value Length> <Value Bytes> | ^ NEXT (NONE) VALUE | | | ------ | | | | new record absolute offset - (minus) relative offset | | -----><Next Value Rel Offset as 5 bytes> <Value Length> <Value Bytes> NEXT VALUE 5) Four elements showing how first record is again updated to point to new value and new value points to most recent (additional) value: Key and Value Reference | | absoluteOffset | ------------------------------------ | v <Next Value Rel Offset as 5 bytes> <Key Bytes> <Value Length> <Value Bytes> | NEXT KEY VALUE | | first record absolute offset + relative offset | | | <5 0's Padding for Next Value Ref> <Value Length> <Value Bytes> | ^ NEXT (NONE) VALUE | | | ------ | | record absolute offset - (minus) relative offset | | | <Next Value Rel Offset as 5 bytes> <Value Length> <Value Bytes> | ^ NEXT VALUE | | | ------ | | | | new record absolute offset - (minus) relative offset | | -----><Next Value Rel Offset as 5 bytes> <Value Length> <Value Bytes> NEXT VALUE You get the idea.

addFirst

public long addFirst(long partialHashCode,
                     byte[] keyBytes,
                     int keyStart,
                     int keyLength,
                     byte[] valueBytes,
                     int valueStart,
                     int valueLength)

Two 64-bit long result is the key and value reference.

Parameters:

partialHashCode -

keyBytes -

keyStart -

keyLength -

valueBytes -

valueStart -

valueLength -

addMore

public long addMore(long refWord,
                    byte[] valueBytes,
                    int valueStart,
                    int valueLength,
                    WriteBuffers.Position unsafeReadPos)

Parameters:

refWord -

valueBytes -

valueStart -

valueLength -

getKey

public void getKey(long refWord,
                   WriteBuffers.ByteSegmentRef keyByteSegmentRef,
                   WriteBuffers.Position readPos)

getEstimatedMemorySize

public long getEstimatedMemorySize()

Specified by:

getEstimatedMemorySize in interface MemoryEstimate