org.apache.druid.segment.serde.cell

Class CellWriter

java.lang.Object

org.apache.druid.segment.serde.cell.CellWriter

All Implemented Interfaces:

Closeable, AutoCloseable, Serializer

public class CellWriter
extends Object
implements Serializer, Closeable

usage:

CellReader effectively stores a list of byte[] payloads that are retrievable randomly by index. The entirety of the data is block compressed. For reading, see CellReader. Example usage:

 StagedSerde<Fuu> fuuSerDe = new ...
 // note that cellWriter.close() *must* be called before writeTo() in order to finalize the index
 try (CellWriter cellWriter = new CellWriter.Builder(segmentWriteOutMedium).build()) {

    fuuList.stream().map(fuuSerDe:serialize).forEach(cellWriter::write);
  }
  // at this point cellWriter contains the index and compressed data


  // transfers the index and compressed data in the format specified below. This method is idempotent and copies
  // the data each time.
  cellWriter.writeTo(writableChannel, fileSmoosher); // 2nd argument currently unused, may be null

  

Note that for use with CellReader, the contents written to the writableChannel must be available as a ByteBuffer

Internal Storage Details

 serialized data is of the form:

    [cell index]
    [payload storage]

 each of these items is stored in compressed streams of blocks with a block index.

 A BlockCompressedPayloadWriter stores byte[] payloads. These may be accessed by creating a
 BlockCompressedPayloadReader over the produced ByteBuffer. Reads may be done by giving a location in the
 uncompressed stream and a size

 NOTE: BlockCompressedPayloadBuffer does not store nulls on write(). However, the cellIndex stores an entry
 with a size of 0 for nulls and CellReader will return null for any null written

  [blockIndexSize:int]
 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |      block index
 |      compressed block # -> block start in compressed stream position (relative to data start)
 |
 |      0: [block position: int]
 |      1: [block position: int]
 |      ...
 |      i: [block position: int]
 |      ...
 |      n: [block position: int]
 |      n+1: [total compressed size ] // stored to simplify invariant of n+1 - n = length(n)
 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 [dataSize:int]
 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 | [compressed payload block 1]
 | [compressed payload block 2]
 | ...
 | [compressed paylod block n]
 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 the CellIndexWriter stores an array of longs using the BlockCompressedPayloadWriter

 logically this an array of longs

 |    0: start_0 : long
 |    1: start_1 : long
 |    ...
 |    n: start_n : long
 |    n+1: start_n + length_n : long  //ie, next position that would have been written to
 |                                   //used again for invariant of length_i = row_i+1 - row_i
 |
 |    but this will be stored as block compressed. Reads are done by addressing it as a long array of bytes
 |
 |    [block index size]
 |    [block index>
 |
 |    [data stream size]
 |    [block compressed payload stream]

 resulting in

 |    [cell index size]
 | ----cell index------------------------
 |    [block index size]
 |    [block index]
 |    [data stream size]
 |    [block compressed payload stream]
 | -------------------------------------
 |    [data stream size]
 | ----data stream------------------------
 |    [block index size]
 |    [block index]
 |    [data stream size]
 |    [block compressed payload stream]
 | -------------------------------------
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CellWriter.Builder

Method Summary

Modifier and Type	Method and Description
static CellWriter.Builder	builder(SegmentWriteOutMedium segmentWriteOutMedium)
void	close()
long	getSerializedSize() Returns the number of bytes, that this Serializer will write to the output _channel_ (not smoosher) on a Serializer.writeTo(java.nio.channels.WritableByteChannel, org.apache.druid.java.util.common.io.smoosh.FileSmoosher) call.
void	write(byte[] cellBytes)
void	write(ByteBuffer cellByteBuffer)
void	writeTo(WritableByteChannel channel, FileSmoosher smoosher) Writes serialized form of this object to the given channel.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

write

public void write(byte[] cellBytes)
           throws IOException

Throws:

IOException

write

public void write(ByteBuffer cellByteBuffer)
           throws IOException

Throws:

IOException

writeTo

public void writeTo(WritableByteChannel channel,
                    @Nullable
                    FileSmoosher smoosher)
             throws IOException

Description copied from interface: Serializer

Writes serialized form of this object to the given channel. If parallel data streams are needed, they could be created with the provided smoosher.

Specified by:

writeTo in interface Serializer

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getSerializedSize

public long getSerializedSize()

Description copied from interface: Serializer

Returns the number of bytes, that this Serializer will write to the output _channel_ (not smoosher) on a Serializer.writeTo(java.nio.channels.WritableByteChannel, org.apache.druid.java.util.common.io.smoosh.FileSmoosher) call.

Specified by:

getSerializedSize in interface Serializer

builder

public static CellWriter.Builder builder(SegmentWriteOutMedium segmentWriteOutMedium)