Package org.apache.pinot.segment.local.io.util

Class VarLengthValueWriter

  • All Implemented Interfaces:
    Closeable, AutoCloseable
    public class VarLengthValueWriter
extends Object
implements Closeable
    The value writer for var-length values (STRING and BYTES). The layout of the file is as follows:

    Header Section:

    • Magic bytes: Chose this to be ".vl;" to avoid conflicts with the fixed size value buffer implementations. By having special characters, this avoids conflicts with regular bytes/strings dictionaries.
    • Version number: This is an integer and can be used for the evolution of the store implementation by incrementing version for every incompatible change to the store/format.
    • Number of elements in the store.
    • The offset where the data section starts. Though the data section usually starts right after he header, having this explicitly will let the store to be evolved freely without any assumptions about where the data section starts.

    Data section:

    • Offsets Array: Integer offsets of the start position of the byte arrays. Example: [O(0), O(1),...O(n)] where O is the Offset function. Length of nth element is computed as: O(n+1) - O(n). Since the last element's length can't be computed using this formula, we store an extra offset at the end to be able to compute last element's length with the same formula, without depending on underlying buffer's size.
    • All byte arrays.
    See Also:
    FixedByteValueReaderWriter