org.apache.hudi.common.util.collection

Class ExternalSpillableMap<T extends Serializable,R extends Serializable>

java.lang.Object

org.apache.hudi.common.util.collection.ExternalSpillableMap<T,R>

All Implemented Interfaces:

Serializable, Map<T,R>

@NotThreadSafe
public class ExternalSpillableMap<T extends Serializable,R extends Serializable>
extends Object
implements Map<T,R>, Serializable

An external map that spills content to disk when there is insufficient space for it to grow.

This map holds 2 types of data structures :

(1) Key-Value pairs in a in-memory map (2) Key-ValueMetadata pairs in an in-memory map which keeps a marker to the values spilled to disk

NOTE : Values are only appended to disk. If a remove() is called, the entry is marked removed from the in-memory key-valueMetadata map but it's values will be lying around in the temp file on disk until the file is cleaned.

The setting of the spill threshold faces the following trade-off: If the spill threshold is too high, the in-memory map may occupy more memory than is available, resulting in OOM. However, if the spill threshold is too low, we spill frequently and incur unnecessary disk writes.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ExternalSpillableMap.DiskMapType The type of map to use for storing the Key, values on disk after it spills from memory in the ExternalSpillableMap.

Nested classes/interfaces inherited from interface java.util.Map

Map.Entry<K,V>

Constructor Summary

Constructors

Constructor and Description
ExternalSpillableMap(Long maxInMemorySizeInBytes, String baseFilePath, SizeEstimator<T> keySizeEstimator, SizeEstimator<R> valueSizeEstimator)
ExternalSpillableMap(Long maxInMemorySizeInBytes, String baseFilePath, SizeEstimator<T> keySizeEstimator, SizeEstimator<R> valueSizeEstimator, ExternalSpillableMap.DiskMapType diskMapType)
ExternalSpillableMap(Long maxInMemorySizeInBytes, String baseFilePath, SizeEstimator<T> keySizeEstimator, SizeEstimator<R> valueSizeEstimator, ExternalSpillableMap.DiskMapType diskMapType, boolean isCompressionEnabled)

Method Summary

Modifier and Type	Method and Description
void	clear()
void	close()
boolean	containsKey(Object key)
boolean	containsValue(Object value)
Set<Map.Entry<T,R>>	entrySet()
R	get(Object key)
long	getCurrentInMemoryMapSize() Approximate memory footprint of the in-memory map.
int	getDiskBasedMapNumEntries() Number of entries in BitCaskDiskMap.
int	getInMemoryMapNumEntries() Number of entries in InMemoryMap.
long	getSizeOfFileOnDiskInBytes() Number of bytes spilled to disk.
boolean	inDiskContainsKey(Object key)
boolean	inMemoryContainsKey(Object key)
boolean	isEmpty()
Iterator<R>	iterator() A custom iterator to wrap over iterating in-memory + disk spilled data.
Set<T>	keySet()
R	put(T key, R value)
void	putAll(Map<? extends T,? extends R> m)
R	remove(Object key)
int	size()
Collection<R>	values()
Stream<R>	valueStream()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Map

compute, computeIfAbsent, computeIfPresent, equals, forEach, getOrDefault, hashCode, merge, putIfAbsent, remove, replace, replace, replaceAll

Constructor Detail

ExternalSpillableMap

public ExternalSpillableMap(Long maxInMemorySizeInBytes,
                            String baseFilePath,
                            SizeEstimator<T> keySizeEstimator,
                            SizeEstimator<R> valueSizeEstimator)
                     throws IOException

Throws:

IOException

ExternalSpillableMap

public ExternalSpillableMap(Long maxInMemorySizeInBytes,
                            String baseFilePath,
                            SizeEstimator<T> keySizeEstimator,
                            SizeEstimator<R> valueSizeEstimator,
                            ExternalSpillableMap.DiskMapType diskMapType)
                     throws IOException

Throws:

IOException

ExternalSpillableMap

public ExternalSpillableMap(Long maxInMemorySizeInBytes,
                            String baseFilePath,
                            SizeEstimator<T> keySizeEstimator,
                            SizeEstimator<R> valueSizeEstimator,
                            ExternalSpillableMap.DiskMapType diskMapType,
                            boolean isCompressionEnabled)
                     throws IOException

Throws:

IOException

Method Detail

iterator

public Iterator<R> iterator()

A custom iterator to wrap over iterating in-memory + disk spilled data.

getDiskBasedMapNumEntries

public int getDiskBasedMapNumEntries()

Number of entries in BitCaskDiskMap.

getSizeOfFileOnDiskInBytes

public long getSizeOfFileOnDiskInBytes()

Number of bytes spilled to disk.

getInMemoryMapNumEntries

public int getInMemoryMapNumEntries()

Number of entries in InMemoryMap.

getCurrentInMemoryMapSize

public long getCurrentInMemoryMapSize()

Approximate memory footprint of the in-memory map.

size

public int size()

Specified by:

size in interface Map<T extends Serializable,R extends Serializable>

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface Map<T extends Serializable,R extends Serializable>

containsKey

public boolean containsKey(Object key)

Specified by:

containsKey in interface Map<T extends Serializable,R extends Serializable>

containsValue

public boolean containsValue(Object value)

Specified by:

containsValue in interface Map<T extends Serializable,R extends Serializable>

inMemoryContainsKey

public boolean inMemoryContainsKey(Object key)

inDiskContainsKey

public boolean inDiskContainsKey(Object key)

get

public R get(Object key)

Specified by:

get in interface Map<T extends Serializable,R extends Serializable>

put

public R put(T key,
             R value)

Specified by:

put in interface Map<T extends Serializable,R extends Serializable>

remove

public R remove(Object key)

Specified by:

remove in interface Map<T extends Serializable,R extends Serializable>

putAll

public void putAll(Map<? extends T,? extends R> m)

Specified by:

putAll in interface Map<T extends Serializable,R extends Serializable>

clear

public void clear()

Specified by:

clear in interface Map<T extends Serializable,R extends Serializable>

close

public void close()

keySet

public Set<T> keySet()

Specified by:

keySet in interface Map<T extends Serializable,R extends Serializable>

values

public Collection<R> values()

Specified by:

values in interface Map<T extends Serializable,R extends Serializable>

valueStream

public Stream<R> valueStream()

entrySet

public Set<Map.Entry<T,R>> entrySet()

Specified by:

entrySet in interface Map<T extends Serializable,R extends Serializable>