parquet.bytes

Class CapacityByteArrayOutputStream

java.lang.Object

java.io.OutputStream

parquet.bytes.CapacityByteArrayOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable

public class CapacityByteArrayOutputStream
extends OutputStream

functionality of ByteArrayOutputStream without the memory and copy overhead It will linearly create a new slab of the initial size when needed (instead of creating a new buffer and copying the data). After 10 slabs their size will increase exponentially (similar to ByteArrayOutputStream behavior) by making the new slab size the size of the existing data. When reusing a buffer it will adjust the slab size based on the previous data size (reset())

Author:

Julien Le Dem

Constructor Summary

Constructors

Constructor and Description
CapacityByteArrayOutputStream(int initialSize)

Method Summary

Methods

Modifier and Type	Method and Description
int	getCapacity()
long	getCurrentIndex()
String	memUsageString(String prefix)
void	reset() When re-using an instance with reset, it will adjust slab size based on previous data size.
void	setByte(long index, byte value) Replace the byte stored at position index in this stream with value
long	size()
void	write(byte[] b, int off, int len)
void	write(int b)
void	writeTo(OutputStream out) Writes the complete contents of this buffer to the specified output stream argument.

Methods inherited from class java.io.OutputStream

close, flush, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CapacityByteArrayOutputStream

public CapacityByteArrayOutputStream(int initialSize)

Parameters:

initialSize - the initialSize of the buffer (also slab size)

Method Detail

write

public void write(int b)

Specified by:

write in class OutputStream

write

public void write(byte[] b,
         int off,
         int len)

Overrides:

write in class OutputStream

writeTo

public void writeTo(OutputStream out)
             throws IOException

Writes the complete contents of this buffer to the specified output stream argument. the output stream's write method out.write(slab, 0, slab.length)) will be called once per slab.

Parameters:

out - the output stream to which to write the data.

Throws:

IOException - if an I/O error occurs.

getCapacity

public int getCapacity()

Returns:

the size of the allocated buffer

reset

public void reset()

When re-using an instance with reset, it will adjust slab size based on previous data size. The intent is to reuse the same instance for the same type of data (for example, the same column). The assumption is that the size in the buffer will be consistent. Otherwise we fall back to exponentialy double the slab size.

• if we used less than half of the first slab (and it is above the minimum slab size), we will make the slab size smaller.
• if we used more than the slab count threshold (10), we will re-adjust the slab size.

if re-adjusting the slab size we will make it 1/5th of the previous used size in the buffer so that overhead of extra memory allocation is about 20% If we used less than the available slabs we free up the unused ones to reduce memory overhead.

size

public long size()

Returns:

the size of the buffered data

getCurrentIndex

public long getCurrentIndex()

Returns:

the index of the last value being written to this stream, which can be passed to setByte(long, byte) in order to change it

setByte

public void setByte(long index,
           byte value)

Replace the byte stored at position index in this stream with value

Parameters:

index - which byte to replace

value - the value to replace it with

memUsageString

public String memUsageString(String prefix)

Parameters:

prefix - a prefix to be used for every new line in the string

Returns:

a text representation of the memory usage of this structure