Package org.apache.pinot.common.datablock

Class BaseDataBlock

  • All Implemented Interfaces:
    DataBlock
    Direct Known Subclasses:
    ColumnarDataBlock, MetadataBlock, RowDataBlock
    public abstract class BaseDataBlock
extends Object
implements DataBlock
    Base data block mostly replicating implementation of DataTableImplV3. +-----------------------------------------------+ | 13 integers of header: | | VERSION | | NUM_ROWS | | NUM_COLUMNS | | EXCEPTIONS SECTION START OFFSET | | EXCEPTIONS SECTION LENGTH | | DICTIONARY_MAP SECTION START OFFSET | | DICTIONARY_MAP SECTION LENGTH | | DATA_SCHEMA SECTION START OFFSET | | DATA_SCHEMA SECTION LENGTH | | FIXED_SIZE_DATA SECTION START OFFSET | | FIXED_SIZE_DATA SECTION LENGTH | | VARIABLE_SIZE_DATA SECTION START OFFSET | | VARIABLE_SIZE_DATA SECTION LENGTH | +-----------------------------------------------+ | EXCEPTIONS SECTION | +-----------------------------------------------+ | DICTIONARY_MAP SECTION | +-----------------------------------------------+ | DATA_SCHEMA SECTION | +-----------------------------------------------+ | FIXED_SIZE_DATA SECTION | +-----------------------------------------------+ | VARIABLE_SIZE_DATA SECTION | +-----------------------------------------------+ | METADATA LENGTH | | METADATA SECTION | +-----------------------------------------------+ To support both row and columnar data format. the size of the data payload will be exactly the same. the only difference is the data layout in FIXED_SIZE_DATA and VARIABLE_SIZE_DATA section, see each impl for details.

    • Field Detail

      • _errCodeToExceptionMap

        protected Map<Integer,​String> _errCodeToExceptionMap

      • _numRows

        protected int _numRows

      • _numColumns

        protected int _numColumns

      • _fixDataSize

        protected int _fixDataSize

      • _stringDictionary

        protected String[] _stringDictionary

      • _fixedSizeDataBytes

        protected byte[] _fixedSizeDataBytes

      • _fixedSizeData

        protected ByteBuffer _fixedSizeData

      • _variableSizeDataBytes

        protected byte[] _variableSizeDataBytes

      • _variableSizeData

        protected ByteBuffer _variableSizeData

    • Constructor Detail

      • BaseDataBlock

        public BaseDataBlock​(int numRows,
                     @Nullable
                     DataSchema dataSchema,
                     String[] stringDictionary,
                     byte[] fixedSizeDataBytes,
                     byte[] variableSizeDataBytes)
        construct a base data block.
        Parameters:
        numRows - num of rows in the block
        dataSchema - schema of the data in the block
        stringDictionary - dictionary encoding map
        fixedSizeDataBytes - byte[] for fix-sized columns.
        variableSizeDataBytes - byte[] for variable length columns (arrays).

      • BaseDataBlock

        public BaseDataBlock()
        Construct empty data table.

    • Method Detail

      • getDataBlockVersionType

        protected abstract int getDataBlockVersionType()
        Return the int serialized form of the data block version and type.
        Returns:

      • getOffsetInFixedBuffer

        protected abstract int getOffsetInFixedBuffer​(int rowId,
                                              int colId)
        return the offset in _fixedSizeDataBytes of the row/column ID.
        Parameters:
        rowId - row ID
        colId - column ID
        Returns:
        the offset in the fixed size buffer for the row/columnID.

      • positionOffsetInVariableBufferAndGetLength

        protected abstract int positionOffsetInVariableBufferAndGetLength​(int rowId,
                                                                  int colId)
        position the _variableSizeDataBytes to the corresponding row/column ID. and return the length of bytes to extract from the variable size buffer.
        Parameters:
        rowId - row ID
        colId - column ID
        Returns:
        the length to extract from variable size buffer.

      • getInt

        public int getInt​(int rowId,
                  int colId)
        Specified by:
        getInt in interface DataBlock

      • getLong

        public long getLong​(int rowId,
                    int colId)
        Specified by:
        getLong in interface DataBlock

      • getFloat

        public float getFloat​(int rowId,
                      int colId)
        Specified by:
        getFloat in interface DataBlock

      • getDouble

        public double getDouble​(int rowId,
                        int colId)
        Specified by:
        getDouble in interface DataBlock

      • getIntArray

        public int[] getIntArray​(int rowId,
                         int colId)
        Specified by:
        getIntArray in interface DataBlock

      • getLongArray

        public long[] getLongArray​(int rowId,
                           int colId)
        Specified by:
        getLongArray in interface DataBlock

      • getFloatArray

        public float[] getFloatArray​(int rowId,
                             int colId)
        Specified by:
        getFloatArray in interface DataBlock

      • getDoubleArray

        public double[] getDoubleArray​(int rowId,
                               int colId)
        Specified by:
        getDoubleArray in interface DataBlock

      • getNullRowIds

        @Nullable
public org.roaringbitmap.RoaringBitmap getNullRowIds​(int colId)
        Specified by:
        getNullRowIds in interface DataBlock

      • serializeStringDictionary

        protected byte[] serializeStringDictionary()
                                    throws IOException
        Helper method to serialize dictionary map.
        Throws:
        IOException