Package org.apache.pinot.segment.local.io.writer.impl

Class VarByteChunkForwardIndexWriter

java.lang.Object

org.apache.pinot.segment.local.io.writer.impl.BaseChunkForwardIndexWriter

org.apache.pinot.segment.local.io.writer.impl.VarByteChunkForwardIndexWriter

All Implemented Interfaces:

Closeable, AutoCloseable, VarByteChunkWriter

@NotThreadSafe
public class VarByteChunkForwardIndexWriter
extends BaseChunkForwardIndexWriter
implements VarByteChunkWriter

Chunk-based raw (non-dictionary-encoded) forward index writer where each chunk contains fixed number of docs, and the entries are variable length.

The layout of each chunk is as follows:

• Header Section: start offsets (stored as int) of the entry within the data section. For partial chunks, offset values are 0 for missing entries.
• Data Section

Field Summary

Fields

Modifier and Type	Field	Description
static int	CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE

Fields inherited from class org.apache.pinot.segment.local.io.writer.impl.BaseChunkForwardIndexWriter

_chunkBuffer, _chunkCompressor, _chunkSize, _compressedBuffer, _dataFile, _dataOffset, _header

Constructor Summary

Constructors

Constructor	Description
VarByteChunkForwardIndexWriter(File file, ChunkCompressionType compressionType, int totalDocs, int numDocsPerChunk, int lengthOfLongestEntry, int writerVersion)	Constructor for the class.

Method Summary

Modifier and Type	Method	Description
void	putBigDecimal(BigDecimal value)
void	putBytes(byte[] value)
void	putBytesMV(byte[][] values)
void	putString(String value)
void	putStringMV(String[] values)
protected void	writeChunk()	Helper method to compress and write the current chunk.

Methods inherited from class org.apache.pinot.segment.local.io.writer.impl.BaseChunkForwardIndexWriter

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.io.Closeable

close

Field Detail

CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE

public static final int CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE

See Also:

Constant Field Values

Constructor Detail

VarByteChunkForwardIndexWriter

public VarByteChunkForwardIndexWriter(File file,
                                      ChunkCompressionType compressionType,
                                      int totalDocs,
                                      int numDocsPerChunk,
                                      int lengthOfLongestEntry,
                                      int writerVersion)
                               throws IOException

Constructor for the class.

Parameters:

file - File to write to.

compressionType - Type of compression to use.

totalDocs - Total number of docs to write.

numDocsPerChunk - Number of documents per chunk.

lengthOfLongestEntry - Length of longest entry (in bytes)

writerVersion - writer format version

Throws:

FileNotFoundException - Throws FileNotFoundException if the specified file is not found.

IOException

Method Detail

putBigDecimal

public void putBigDecimal(BigDecimal value)

Specified by:

putBigDecimal in interface VarByteChunkWriter

putString

public void putString(String value)

Specified by:

putString in interface VarByteChunkWriter

putBytes

public void putBytes(byte[] value)

Specified by:

putBytes in interface VarByteChunkWriter

putStringMV

public void putStringMV(String[] values)

Specified by:

putStringMV in interface VarByteChunkWriter

putBytesMV

public void putBytesMV(byte[][] values)

Specified by:

putBytesMV in interface VarByteChunkWriter

writeChunk

protected void writeChunk()

Helper method to compress and write the current chunk.

• Chunk header is of fixed size, so fills out any remaining offsets for partially filled chunks.
• Compresses and writes the chunk to the data file.
• Updates the header with the current chunks offset.
• Clears up the buffers, so that they can be reused.

Overrides:

writeChunk in class BaseChunkForwardIndexWriter