Package org.apache.pinot.segment.local.io.writer.impl

Class VarByteChunkSVForwardIndexWriter

  • All Implemented Interfaces:
    Closeable, AutoCloseable, VarByteChunkWriter
    @NotThreadSafe
public class VarByteChunkSVForwardIndexWriter
extends BaseChunkSVForwardIndexWriter
implements VarByteChunkWriter
    Class to write out variable length bytes into a single column. The layout of the file is as follows:

    Header Section:

    • Integer: File format version.
    • Integer: Total number of chunks.
    • Integer: Number of docs per chunk.
    • Integer: Length of longest entry (in bytes).
    • Integer: Total number of docs (version 2 onwards).
    • Integer: Compression type enum value (version 2 onwards).
    • Integer: Start offset of data header (version 2 onwards).
    • Integer array: Integer offsets for all chunks in the data (upto version 2), Long array: Long offsets for all chunks in the data (version 3 onwards)

    Individual Chunks:

    • Integer offsets to start position of rows: For partial chunks, offset values are 0 for missing rows.
    • Data bytes.
    Only sequential writes are supported.

    • Field Detail

      • CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE

        public static final int CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE
        See Also:
        Constant Field Values

    • Constructor Detail

      • VarByteChunkSVForwardIndexWriter

        public VarByteChunkSVForwardIndexWriter​(File file,
                                        ChunkCompressionType compressionType,
                                        int totalDocs,
                                        int numDocsPerChunk,
                                        int lengthOfLongestEntry,
                                        int writerVersion)
                                 throws IOException
        Constructor for the class.
        Parameters:
        file - File to write to.
        compressionType - Type of compression to use.
        totalDocs - Total number of docs to write.
        numDocsPerChunk - Number of documents per chunk.
        lengthOfLongestEntry - Length of longest entry (in bytes)
        writerVersion - writer format version
        Throws:
        FileNotFoundException - Throws FileNotFoundException if the specified file is not found.
        IOException

    • Method Detail

      • putStrings

        public void putStrings​(String[] values)

      • putByteArrays

        public void putByteArrays​(byte[][] values)

      • writeChunk

        protected void writeChunk()
        Helper method to compress and write the current chunk.
        • Chunk header is of fixed size, so fills out any remaining offsets for partially filled chunks.
        • Compresses and writes the chunk to the data file.
        • Updates the header with the current chunks offset.
        • Clears up the buffers, so that they can be reused.
        Overrides:
        writeChunk in class BaseChunkSVForwardIndexWriter