org.apache.hadoop.hive.ql.exec.persistence

Class BytesBytesMultiHashMap

java.lang.Object

org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap

All Implemented Interfaces:

MemoryEstimate

public final class BytesBytesMultiHashMap
extends Object
implements MemoryEstimate

HashMap that maps byte arrays to byte arrays with limited functionality necessary for MapJoin hash tables, with small memory overhead. Supports multiple values for single key. Values can be added for key (but cannot be removed); values can be gotten for the key. Some things (like entrySet) are easy to add; some e.g. deletion are pretty hard to do well. Additionally, for each key it contains a magic "state byte" which is not part of the key and can be updated on every put for that key. That is really silly, we use it to store aliasFilter. Magic byte could be removed for generality. Initially inspired by HPPC LongLongOpenHashMap; however, the code is almost completely reworked and there's very little in common left save for quadratic probing (and that with some changes).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	BytesBytesMultiHashMap.KvSource The source of keys and values to put into hashtable; avoids byte copying.
static class	BytesBytesMultiHashMap.Result The result of looking up a key in the multi-hash map.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
BytesBytesMultiHashMap(int initialCapacity, float loadFactor, int wbSize, long maxProbeSize)

Method Summary

Modifier and Type	Method and Description
void	clear()
void	debugDumpMetrics()
void	debugDumpTable() Writes the debug dump of the table into logs.
void	expandAndRehashToTarget(int estimateNewRowCount)
int	findNextNonMatched(int currentSlotNum, WriteBuffers.ByteSegmentRef keyRef, BytesBytesMultiHashMap.Result hashMapResult, MatchTracker matchTracker) Finds the next non matched Small Table key and value.
long	getEstimatedMemorySize()
int	getNumHashBuckets()
int	getNumValues() Number of values in the hashmap This is equal to or bigger than number of keys, since some values may share the same key
byte	getValueResult(byte[] key, int offset, int length, BytesBytesMultiHashMap.Result hashMapResult, MatchTracker matchTracker) Finds a key.
long	memorySize() Number of bytes used by the hashmap.
void	populateValue(WriteBuffers.ByteSegmentRef valueRef) Take the segment reference from getValueRefs(byte[],int,List) result and makes it self-contained - adds byte array where the value is stored, and updates the offset from "global" write buffers offset to offset within that array.
void	put(BytesBytesMultiHashMap.KvSource kv, int keyHashCode)
void	seal()
int	size() Number of keys in the hashmap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

BytesBytesMultiHashMap

public BytesBytesMultiHashMap(int initialCapacity,
                              float loadFactor,
                              int wbSize,
                              long maxProbeSize)

Method Detail

put

public void put(BytesBytesMultiHashMap.KvSource kv,
                int keyHashCode)
         throws SerDeException

Throws:

SerDeException

getValueResult

public byte getValueResult(byte[] key,
                           int offset,
                           int length,
                           BytesBytesMultiHashMap.Result hashMapResult,
                           MatchTracker matchTracker)

Finds a key. Values can be read with the supplied result object. Important Note: The caller is expected to pre-allocate the hashMapResult and not share it among other threads.

Parameters:

key - Key buffer.

offset - the offset to the key in the buffer

hashMapResult - The object to fill in that can read the values.

matchTracker - Opitional object for tracking key matches.

Returns:

The state byte.

populateValue

public void populateValue(WriteBuffers.ByteSegmentRef valueRef)

Take the segment reference from getValueRefs(byte[],int,List) result and makes it self-contained - adds byte array where the value is stored, and updates the offset from "global" write buffers offset to offset within that array.

findNextNonMatched

public int findNextNonMatched(int currentSlotNum,
                              WriteBuffers.ByteSegmentRef keyRef,
                              BytesBytesMultiHashMap.Result hashMapResult,
                              MatchTracker matchTracker)

Finds the next non matched Small Table key and value. Supports FULL OUTER MapJoin.

Parameters:

currentSlotNum - Start by specifying -1; the return index from the previous call.

keyRef - If the return value is not -1, a reference to the key bytes.

hashMapResult - If the return value is not -1, the key's values.

matchTracker - The object that tracks matches (non-shared).

Returns:

The current index of the non-matched key; or -1 if no more.

size

public int size()

Number of keys in the hashmap

Returns:

number of keys

getNumValues

public int getNumValues()

Number of values in the hashmap This is equal to or bigger than number of keys, since some values may share the same key

Returns:

number of values

getNumHashBuckets

public int getNumHashBuckets()

memorySize

public long memorySize()

Number of bytes used by the hashmap. There are two main components that take most memory: writeBuffers and refs Others include instance fields: 100

Returns:

number of bytes

getEstimatedMemorySize

public long getEstimatedMemorySize()

Specified by:

getEstimatedMemorySize in interface MemoryEstimate

seal

public void seal()

clear

public void clear()

expandAndRehashToTarget

public void expandAndRehashToTarget(int estimateNewRowCount)

debugDumpTable

public void debugDumpTable()

Writes the debug dump of the table into logs. Not thread-safe.

debugDumpMetrics

public void debugDumpMetrics()