Package org.apache.pinot.segment.spi.memory

Class PinotDataBuffer

  • All Implemented Interfaces:
    Closeable, AutoCloseable
    Direct Known Subclasses:
    BasePinotLBuffer, PinotByteBuffer
    @ThreadSafe
public abstract class PinotDataBuffer
extends Object
implements Closeable
    The class PinotDataBuffer is the byte buffer for Pinot data that resides in off-heap memory.

    The byte buffer may be memory mapped (MMAP) or direct allocated (DIRECT).

    Supports buffers larger than 2GB.

    This class will also track the number and memory usage of the buffers.

    NOTE: All the accesses to the buffer are unchecked for performance reason. Behavior of accessing buffer with invalid index is undefined.

    Backward-compatible:

    • Index file (forward index, inverted index, dictionary) is always big-endian
    • Star-tree file is always little-endian
    • Temporary buffer should be allocated using native-order for performance

    • Field Detail

      • NATIVE_ORDER

        public static final ByteOrder NATIVE_ORDER

      • NON_NATIVE_ORDER

        public static final ByteOrder NON_NATIVE_ORDER

      • BULK_BYTES_PROCESSING_THRESHOLD

        public static final int BULK_BYTES_PROCESSING_THRESHOLD
        See Also:
        Constant Field Values

    • Constructor Detail

      • PinotDataBuffer

        protected PinotDataBuffer​(boolean closeable)

    • Method Detail

      • allocateDirect

        public static PinotDataBuffer allocateDirect​(long size,
                                             ByteOrder byteOrder,
                                             @Nullable
                                             String description)
        Allocates a buffer using direct memory.

        NOTE: The contents of the allocated buffer are not defined.

        Parameters:
        size - The size of the buffer
        byteOrder - The byte order of the buffer (big-endian or little-endian)
        description - The description of the buffer
        Returns:
        The buffer allocated

      • loadBigEndianFile

        public static PinotDataBuffer loadBigEndianFile​(File file)
                                         throws IOException
        Allocates a buffer using direct memory and loads a big-endian file into the buffer.
        Throws:
        IOException

      • mapFile

        public static PinotDataBuffer mapFile​(File file,
                                      boolean readOnly,
                                      long offset,
                                      long size,
                                      ByteOrder byteOrder,
                                      @Nullable
                                      String description)
                               throws IOException
        Memory maps a file into a buffer.

        NOTE: If the file gets extended, the contents of the extended portion of the file are not defined.

        Throws:
        IOException

      • getDirectBufferCount

        public static long getDirectBufferCount()

      • getDirectBufferUsage

        public static long getDirectBufferUsage()

      • getMmapBufferCount

        public static long getMmapBufferCount()

      • getMmapBufferUsage

        public static long getMmapBufferUsage()

      • getAllocationFailureCount

        public static long getAllocationFailureCount()

      • getBufferInfo

        public static List<String> getBufferInfo()

      • getByte

        public abstract byte getByte​(int offset)

      • getByte

        public abstract byte getByte​(long offset)

      • putByte

        public abstract void putByte​(int offset,
                             byte value)

      • putByte

        public abstract void putByte​(long offset,
                             byte value)

      • getChar

        public abstract char getChar​(int offset)

      • getChar

        public abstract char getChar​(long offset)

      • putChar

        public abstract void putChar​(int offset,
                             char value)

      • putChar

        public abstract void putChar​(long offset,
                             char value)

      • getShort

        public abstract short getShort​(int offset)

      • getShort

        public abstract short getShort​(long offset)

      • putShort

        public abstract void putShort​(int offset,
                              short value)

      • putShort

        public abstract void putShort​(long offset,
                              short value)

      • getInt

        public abstract int getInt​(int offset)

      • getInt

        public abstract int getInt​(long offset)

      • putInt

        public abstract void putInt​(int offset,
                            int value)

      • putInt

        public abstract void putInt​(long offset,
                            int value)

      • getLong

        public abstract long getLong​(int offset)

      • getLong

        public abstract long getLong​(long offset)

      • putLong

        public abstract void putLong​(int offset,
                             long value)

      • putLong

        public abstract void putLong​(long offset,
                             long value)

      • getFloat

        public abstract float getFloat​(int offset)

      • getFloat

        public abstract float getFloat​(long offset)

      • putFloat

        public abstract void putFloat​(int offset,
                              float value)

      • putFloat

        public abstract void putFloat​(long offset,
                              float value)

      • getDouble

        public abstract double getDouble​(int offset)

      • getDouble

        public abstract double getDouble​(long offset)

      • putDouble

        public abstract void putDouble​(int offset,
                               double value)

      • putDouble

        public abstract void putDouble​(long offset,
                               double value)

      • copyTo

        public abstract void copyTo​(long offset,
                            byte[] buffer,
                            int destOffset,
                            int size)

      • copyTo

        public void copyTo​(long offset,
                   byte[] buffer)

      • copyTo

        public abstract void copyTo​(long offset,
                            PinotDataBuffer buffer,
                            long destOffset,
                            long size)

      • readFrom

        public abstract void readFrom​(long offset,
                              byte[] buffer,
                              int srcOffset,
                              int size)

      • readFrom

        public void readFrom​(long offset,
                     byte[] buffer)

      • readFrom

        public abstract void readFrom​(long offset,
                              ByteBuffer buffer)

      • readFrom

        public abstract void readFrom​(long offset,
                              File file,
                              long srcOffset,
                              long size)
                       throws IOException
        Throws:
        IOException

      • size

        public abstract long size()

      • view

        public abstract PinotDataBuffer view​(long start,
                                     long end,
                                     ByteOrder byteOrder)
        Creates a view of the range [start, end) of this buffer with the given byte order. Calling flush() or close() has no effect on view.

      • view

        public PinotDataBuffer view​(long start,
                            long end)
        Creates a view of the range [start, end) of this buffer with the current byte order. Calling flush() or close() has no effect on view.

      • toDirectByteBuffer

        public abstract ByteBuffer toDirectByteBuffer​(long offset,
                                              int size,
                                              ByteOrder byteOrder)

      • toDirectByteBuffer

        public ByteBuffer toDirectByteBuffer​(long offset,
                                     int size)

      • flush

        public abstract void flush()