GenericIndexed (druid-processing 25.0.0 API)

java.lang.Object
- org.apache.druid.segment.data.GenericIndexed<T>

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<T>, HotLoopCallee, CloseableIndexed<T>, Indexed<T>, Serializer
```
public class GenericIndexed<T>
extends Object
implements CloseableIndexed<T>, Serializer
```
A generic, flat storage mechanism. Use static methods fromArray() or fromIterable() to construct. If input is sorted, supports binary search index lookups. If input is not sorted, only supports array-like index lookups.
V1 Storage Format:
byte 1: version (0x1) byte 2 == 0x1 =>; allowReverseLookup bytes 3-6 =>; numBytesUsed bytes 7-10 =>; numElements bytes 10-((numElements * 4) + 10): integers representing *end* offsets of byte serialized values bytes ((numElements * 4) + 10)-(numBytesUsed + 2): 4-byte integer representing length of value, followed by bytes for value. Length of value stored has no meaning, if next offset is strictly greater than the current offset, and if they are the same, -1 at this field means null, and 0 at this field means some object (potentially non-null - e. g. in the string case, that is serialized as an empty sequence of bytes).
V2 Storage Format Meta, header and value files are separate and header file stored in native endian byte order. Meta File: byte 1: version (0x2) byte 2 == 0x1 =>; allowReverseLookup bytes 3-6: numberOfElementsPerValueFile expressed as power of 2. That means all the value files contains same number of items except last value file and may have fewer elements. bytes 7-10 =>; numElements bytes 11-14 =>; columnNameLength bytes 15-columnNameLength =>; columnName
Header file name is identified as: StringUtils.format("%s_header", columnName) value files are identified as: StringUtils.format("%s_value_%d", columnName, fileNumber) number of value files == numElements/numberOfElementsPerValueFile The version EncodedStringDictionaryWriter.VERSION is reserved and must never be specified as the GenericIndexed version byte, else it will interfere with string column deserialization.

See Also:

GenericIndexedWriter

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

class GenericIndexed.BufferIndexed
Single-threaded view.

Nested Classes
Modifier and Type	Class and Description
`class`	`GenericIndexed.BufferIndexed` Single-threaded view.

Field Summary

Fields
Modifier and Type	Field and Description
`static ObjectStrategy<String>`	`STRING_STRATEGY`
`static ObjectStrategy<ByteBuffer>`	`UTF8_STRATEGY` An ObjectStrategy that returns a big-endian ByteBuffer pointing to original data.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`close()`
`static <T> GenericIndexed<T>`	`fromArray(T[] objects, ObjectStrategy<T> strategy)`
`static <T> GenericIndexed<T>`	`fromIterable(Iterable<T> objectsIterable, ObjectStrategy<T> strategy)`
`T`	`get(int index)` Get the value at specified position
`Class<? extends T>`	`getClazz()`
`long`	`getSerializedSize()` Returns the number of bytes, that this Serializer will write to the output _channel_ (not smoosher) on a `Serializer.writeTo(java.nio.channels.WritableByteChannel, org.apache.druid.java.util.common.io.smoosh.FileSmoosher)` call.
`int`	`indexOf(T value)` Returns the index of "value" in this GenericIndexed object, or (-(insertion point) - 1) if the value is not present, in the manner of Arrays.binarySearch.
`void`	`inspectRuntimeShape(RuntimeShapeInspector inspector)` Implementations of this method should call `inspector.visit()` with all fields of this class, which meet two conditions: 1.
`boolean`	`isSorted()` Indicates if this value set is sorted, the implication being that the contract of `Indexed.indexOf(T)` is strenthened to return a negative number equal to (-(insertion point) - 1) when the value is not present in the set.
`Iterator<T>`	`iterator()`
`static GenericIndexed<ResourceHolder<ByteBuffer>>`	`ofCompressedByteBuffers(Iterable<ByteBuffer> buffers, CompressionStrategy compression, int bufferSize, ByteOrder order, Closer closer)`
`static <T> GenericIndexed<T>`	`read(ByteBuffer buffer, ObjectStrategy<T> strategy)`
`static <T> GenericIndexed<T>`	`read(ByteBuffer buffer, ObjectStrategy<T> strategy, SmooshedFileMapper fileMapper)`
`GenericIndexed.BufferIndexed`	`singleThreaded()` Create a non-thread-safe Indexed, which may perform better than the underlying Indexed.
`int`	`size()` Number of elements in the value set
`String`	`toString()`
`void`	`writeTo(WritableByteChannel channel, FileSmoosher smoosher)` Writes serialized form of this object to the given channel.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.druid.segment.data.Indexed
checkIndex

Methods inherited from interface java.lang.Iterable
forEach, spliterator

- Field Detail
  - UTF8_STRATEGY
```
public static final ObjectStrategy<ByteBuffer> UTF8_STRATEGY
```
    An ObjectStrategy that returns a big-endian ByteBuffer pointing to original data. The returned ByteBuffer is a fresh read-only instance, so it is OK for callers to modify its position, limit, etc. However, it does point to the original data, so callers must take care not to use it if the original data may have been freed. The compare method of this instance uses StringUtils.compareUtf8UsingJavaStringOrdering(byte[], byte[]) so that behavior is consistent with STRING_STRATEGY.
  - STRING_STRATEGY
```
public static final ObjectStrategy<String> STRING_STRATEGY
```
- Method Detail
  - read
```
public static <T> GenericIndexed<T> read(ByteBuffer buffer,
                                         ObjectStrategy<T> strategy)
```
  - read
```
public static <T> GenericIndexed<T> read(ByteBuffer buffer,
                                         ObjectStrategy<T> strategy,
                                         SmooshedFileMapper fileMapper)
```
  - fromArray
```
public static <T> GenericIndexed<T> fromArray(T[] objects,
                                              ObjectStrategy<T> strategy)
```
  - ofCompressedByteBuffers
```
public static GenericIndexed<ResourceHolder<ByteBuffer>> ofCompressedByteBuffers(Iterable<ByteBuffer> buffers,
                                                                                 CompressionStrategy compression,
                                                                                 int bufferSize,
                                                                                 ByteOrder order,
                                                                                 Closer closer)
```
  - fromIterable
```
public static <T> GenericIndexed<T> fromIterable(Iterable<T> objectsIterable,
                                                 ObjectStrategy<T> strategy)
```
  - getClazz
```
public Class<? extends T> getClazz()
```
  - size
```
public int size()
```
    Description copied from interface: Indexed
    
    Number of elements in the value set
    
    Specified by:
    
    size in interface Indexed<T>
  - get
```
public T get(int index)
```
    Description copied from interface: Indexed
    
    Get the value at specified position
    
    Specified by:
    
    get in interface Indexed<T>
  - indexOf
```
public int indexOf(@Nullable
                   T value)
```
    Returns the index of "value" in this GenericIndexed object, or (-(insertion point) - 1) if the value is not present, in the manner of Arrays.binarySearch. This strengthens the contract of Indexed, which only guarantees that values-not-found will return some negative number.
    
    Specified by:
    
    indexOf in interface Indexed<T>
    
    Parameters:
    
    value - value to search for
    
    Returns:
    
    index of value, or negative number equal to (-(insertion point) - 1).
  - isSorted
```
public boolean isSorted()
```
    Description copied from interface: Indexed
    
    Indicates if this value set is sorted, the implication being that the contract of Indexed.indexOf(T) is strenthened to return a negative number equal to (-(insertion point) - 1) when the value is not present in the set.
    
    Specified by:
    
    isSorted in interface Indexed<T>
  - iterator
```
public Iterator<T> iterator()
```
    Specified by:
    
    iterator in interface Iterable<T>
  - getSerializedSize
```
public long getSerializedSize()
```
    Description copied from interface: Serializer
    
    Returns the number of bytes, that this Serializer will write to the output _channel_ (not smoosher) on a Serializer.writeTo(java.nio.channels.WritableByteChannel, org.apache.druid.java.util.common.io.smoosh.FileSmoosher) call.
    
    Specified by:
    
    getSerializedSize in interface Serializer
  - writeTo
```
public void writeTo(WritableByteChannel channel,
                    FileSmoosher smoosher)
             throws IOException
```
    Description copied from interface: Serializer
    
    Writes serialized form of this object to the given channel. If parallel data streams are needed, they could be created with the provided smoosher.
    
    Specified by:
    
    writeTo in interface Serializer
    
    Throws:
    
    IOException
  - singleThreaded
```
public GenericIndexed.BufferIndexed singleThreaded()
```
    Create a non-thread-safe Indexed, which may perform better than the underlying Indexed.
    
    Returns:
    
    a non-thread-safe Indexed
  - inspectRuntimeShape
```
public void inspectRuntimeShape(RuntimeShapeInspector inspector)
```
    Description copied from interface: HotLoopCallee
    
    Implementations of this method should call inspector.visit() with all fields of this class, which meet two conditions: 1. They are used in methods of this class, annotated with CalledFromHotLoop 2. They are either: a. Nullable objects b. Instances of HotLoopCallee c. Objects, which don't always have a specific class in runtime. For example, a field of type Set could be HashSet or TreeSet in runtime, depending on how this instance (the instance on which inspectRuntimeShape() is called) is configured. d. ByteBuffer or similar objects, where byte order matters e. boolean flags, affecting branch taking f. Arrays of objects, meeting any of conditions a-e.
    
    Specified by:
    
    inspectRuntimeShape in interface HotLoopCallee
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - close
```
public void close()
```
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable

Class GenericIndexed<T>

Nested Class Summary

Field Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.druid.segment.data.Indexed

Methods inherited from interface java.lang.Iterable

Field Detail

UTF8_STRATEGY

STRING_STRATEGY

Method Detail

read

read

fromArray

ofCompressedByteBuffers

fromIterable

getClazz

size

get

indexOf

isSorted

iterator

getSerializedSize

writeTo

singleThreaded

inspectRuntimeShape

toString

close