org.apache.druid.segment.data

Class GenericIndexed<T>

java.lang.Object

org.apache.druid.segment.data.GenericIndexed<T>

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<T>, HotLoopCallee, CloseableIndexed<T>, Indexed<T>, Serializer

public class GenericIndexed<T>
extends Object
implements CloseableIndexed<T>, Serializer

A generic, flat storage mechanism. Use static methods fromArray() or fromIterable() to construct. If input is sorted, supports binary search index lookups. If input is not sorted, only supports array-like index lookups.

V1 Storage Format:

byte 1: version (0x1) byte 2 == 0x1 =>; allowReverseLookup bytes 3-6 =>; numBytesUsed bytes 7-10 =>; numElements bytes 10-((numElements * 4) + 10): integers representing *end* offsets of byte serialized values bytes ((numElements * 4) + 10)-(numBytesUsed + 2): 4-byte integer representing length of value, followed by bytes for value. Length of value stored has no meaning, if next offset is strictly greater than the current offset, and if they are the same, -1 at this field means null, and 0 at this field means some object (potentially non-null - e. g. in the string case, that is serialized as an empty sequence of bytes).

V2 Storage Format Meta, header and value files are separate and header file stored in native endian byte order. Meta File: byte 1: version (0x2) byte 2 == 0x1 =>; allowReverseLookup bytes 3-6: numberOfElementsPerValueFile expressed as power of 2. That means all the value files contains same number of items except last value file and may have fewer elements. bytes 7-10 =>; numElements bytes 11-14 =>; columnNameLength bytes 15-columnNameLength =>; columnName

Header file name is identified as: StringUtils.format("%s_header", columnName) value files are identified as: StringUtils.format("%s_value_%d", columnName, fileNumber) number of value files == numElements/numberOfElementsPerValueFile

Field Summary

Fields

Modifier and Type	Field and Description
static ObjectStrategy<String>	STRING_STRATEGY

Method Summary

Modifier and Type	Method and Description
void	close()
static <T> GenericIndexed<T>	fromArray(T[] objects, ObjectStrategy<T> strategy)
static <T> GenericIndexed<T>	fromIterable(Iterable<T> objectsIterable, ObjectStrategy<T> strategy)
T	get(int index)
Class<? extends T>	getClazz()
long	getSerializedSize() Returns the number of bytes, that this Serializer will write to the output _channel_ (not smoosher) on a Serializer.writeTo(java.nio.channels.WritableByteChannel, org.apache.druid.java.util.common.io.smoosh.FileSmoosher) call.
int	indexOf(T value) Returns the index of "value" in this GenericIndexed object, or (-(insertion point) - 1) if the value is not present, in the manner of Arrays.binarySearch.
void	inspectRuntimeShape(RuntimeShapeInspector inspector) Implementations of this method should call inspector.visit() with all fields of this class, which meet two conditions: 1.
Iterator<T>	iterator()
static <T> GenericIndexed<T>	read(ByteBuffer buffer, ObjectStrategy<T> strategy)
static <T> GenericIndexed<T>	read(ByteBuffer buffer, ObjectStrategy<T> strategy, SmooshedFileMapper fileMapper)
org.apache.druid.segment.data.GenericIndexed.BufferIndexed	singleThreaded() Create a non-thread-safe Indexed, which may perform better than the underlying Indexed.
int	size()
String	toString()
void	writeTo(WritableByteChannel channel, FileSmoosher smoosher) Writes serialized form of this object to the given channel.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

STRING_STRATEGY

public static final ObjectStrategy<String> STRING_STRATEGY

Method Detail

read

public static <T> GenericIndexed<T> read(ByteBuffer buffer,
                                         ObjectStrategy<T> strategy)

read

public static <T> GenericIndexed<T> read(ByteBuffer buffer,
                                         ObjectStrategy<T> strategy,
                                         SmooshedFileMapper fileMapper)

fromArray

public static <T> GenericIndexed<T> fromArray(T[] objects,
                                              ObjectStrategy<T> strategy)

fromIterable

public static <T> GenericIndexed<T> fromIterable(Iterable<T> objectsIterable,
                                                 ObjectStrategy<T> strategy)

getClazz

public Class<? extends T> getClazz()

size

public int size()

Specified by:

size in interface Indexed<T>

get

public T get(int index)

Specified by:

get in interface Indexed<T>

indexOf

public int indexOf(@Nullable
                   T value)

Returns the index of "value" in this GenericIndexed object, or (-(insertion point) - 1) if the value is not present, in the manner of Arrays.binarySearch. This strengthens the contract of Indexed, which only guarantees that values-not-found will return some negative number.

Specified by:

indexOf in interface Indexed<T>

Parameters:

value - value to search for

Returns:

index of value, or negative number equal to (-(insertion point) - 1).

iterator

public Iterator<T> iterator()

Specified by:

iterator in interface Iterable<T>

getSerializedSize

public long getSerializedSize()

Description copied from interface: Serializer

Returns the number of bytes, that this Serializer will write to the output _channel_ (not smoosher) on a Serializer.writeTo(java.nio.channels.WritableByteChannel, org.apache.druid.java.util.common.io.smoosh.FileSmoosher) call.

Specified by:

getSerializedSize in interface Serializer

writeTo

public void writeTo(WritableByteChannel channel,
                    FileSmoosher smoosher)
             throws IOException

Description copied from interface: Serializer

Writes serialized form of this object to the given channel. If parallel data streams are needed, they could be created with the provided smoosher.

Specified by:

writeTo in interface Serializer

Throws:

IOException

singleThreaded

public org.apache.druid.segment.data.GenericIndexed.BufferIndexed singleThreaded()

Create a non-thread-safe Indexed, which may perform better than the underlying Indexed.

Returns:

a non-thread-safe Indexed

inspectRuntimeShape

public void inspectRuntimeShape(RuntimeShapeInspector inspector)

Description copied from interface: HotLoopCallee

Implementations of this method should call inspector.visit() with all fields of this class, which meet two conditions: 1. They are used in methods of this class, annotated with CalledFromHotLoop 2. They are either: a. Nullable objects b. Instances of HotLoopCallee c. Objects, which don't always have a specific class in runtime. For example, a field of type Set could be HashSet or TreeSet in runtime, depending on how this instance (the instance on which inspectRuntimeShape() is called) is configured. d. ByteBuffer or similar objects, where byte order matters e. boolean flags, affecting branch taking f. Arrays of objects, meeting any of conditions a-e.

Specified by:

inspectRuntimeShape in interface HotLoopCallee

toString

public String toString()

Overrides:

toString in class Object

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable