Package com.clickhouse.data

Class ClickHouseDataProcessor

java.lang.Object

com.clickhouse.data.ClickHouseDataProcessor

public abstract class ClickHouseDataProcessor
extends Object

This defines a data processor for dealing with serialization and deserialization of one or multiple ClickHouseFormat. Unlike ClickHouseDeserializer and ClickHouseSerializer, which is for specific column or data type, data processor is a combination of both, and it can handle more scenarios like separator between columns and rows.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static final class	ClickHouseDataProcessor.UseObjectConfig

Field Summary

Fields

Modifier and Type	Field	Description
protected final ClickHouseColumn[]	columns
protected final ClickHouseDataConfig	config
protected final ClickHouseRecord	currentRecord
static final List<ClickHouseColumn>	DEFAULT_COLUMNS
protected final ClickHouseDeserializer[]	deserializers
protected static final String	ERROR_FAILED_TO_READ
protected static final String	ERROR_FAILED_TO_WRITE
protected static final String	ERROR_REACHED_END_OF_STREAM
protected static final String	ERROR_UNKNOWN_DATA_TYPE
protected final ClickHouseInputStream	input
protected final ClickHouseOutputStream	output
protected int	readPosition	Column index shared by read(ClickHouseValue), records(), and values().
protected final Iterator<ClickHouseRecord>	records
protected final ClickHouseSerializer[]	serializers
protected final Map<String,Object>	settings
protected final ClickHouseValue[]	templates
protected final Iterator<ClickHouseValue>	values
protected int	writePosition	Column index shared by write(ClickHouseValue).

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	ClickHouseDataProcessor(ClickHouseDataConfig config, ClickHouseInputStream input, ClickHouseOutputStream output, List<ClickHouseColumn> columns, Map<String,Serializable> settings)	Default constructor.

Method Summary

Modifier and Type	Method	Description
protected ClickHouseDeserializer[]	buildDeserializeSteps(ClickHouseColumn column)	Builds list of steps to deserialize value for the given column.
protected ClickHouseSerializer[]	buildSerializeSteps(ClickHouseColumn column)	Builds list of steps to serialize value for the given column.
protected abstract ClickHouseRecord	createRecord()	Factory method to create a record.
final List<ClickHouseColumn>	getColumns()	Gets list of columns to process.
abstract ClickHouseDeserializer	getDeserializer(ClickHouseDataConfig config, ClickHouseColumn column)
final ClickHouseDeserializer[]	getDeserializers(ClickHouseDataConfig config, List<ClickHouseColumn> columns)
abstract ClickHouseSerializer	getSerializer(ClickHouseDataConfig config, ClickHouseColumn column)
final ClickHouseSerializer[]	getSerializers(ClickHouseDataConfig config, List<ClickHouseColumn> columns)
protected boolean	hasMoreToRead()	Checks whether there's more to read from input stream.
protected Iterator<ClickHouseRecord>	initRecords()	Initializes iterator of ClickHouseRecord for reading values record by record.
protected Iterator<ClickHouseValue>	initValues()	Initializes iterator of ClickHouseValue for reading values one by one.
ClickHouseValue	read(ClickHouseValue value)	Reads deserialized value of next column(at readPosition) directly from input stream.
protected void	readAndFill(ClickHouseRecord r)	Reads columns(starting from readPosition) from input stream and fill deserialized data into the given record.
protected void	readAndFill(ClickHouseValue value)	Reads next column(at readPosition from input stream and fill deserialized data into the given value object.
protected abstract List<ClickHouseColumn>	readColumns()	Reads columns from input stream.
final Iterable<ClickHouseRecord>	records()	Returns an iterable collection of records which can be walked through in a foreach-loop.
final Iterable<ClickHouseValue>	values()	Returns an iterable collection of values which can be walked through in a foreach-loop.
void	write(ClickHouseValue value)	Writes serialized value of next column(at readPosition) to output stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_COLUMNS

public static final List<ClickHouseColumn> DEFAULT_COLUMNS

ERROR_FAILED_TO_READ

protected static final String ERROR_FAILED_TO_READ

See Also:

• Constant Field Values

ERROR_FAILED_TO_WRITE

protected static final String ERROR_FAILED_TO_WRITE

See Also:

• Constant Field Values

ERROR_REACHED_END_OF_STREAM

protected static final String ERROR_REACHED_END_OF_STREAM

See Also:

• Constant Field Values

ERROR_UNKNOWN_DATA_TYPE

protected static final String ERROR_UNKNOWN_DATA_TYPE

See Also:

• Constant Field Values

config

protected final ClickHouseDataConfig config

input

protected final ClickHouseInputStream input

output

protected final ClickHouseOutputStream output

columns

protected final ClickHouseColumn[] columns

currentRecord

protected final ClickHouseRecord currentRecord

templates

protected final ClickHouseValue[] templates

settings

protected final Map<String,Object> settings

records

protected final Iterator<ClickHouseRecord> records

values

protected final Iterator<ClickHouseValue> values

deserializers

protected final ClickHouseDeserializer[] deserializers

serializers

protected final ClickHouseSerializer[] serializers

readPosition

protected int readPosition

Column index shared by read(ClickHouseValue), records(), and values().

writePosition

protected int writePosition

Column index shared by write(ClickHouseValue).

Constructor Details

ClickHouseDataProcessor

protected ClickHouseDataProcessor(ClickHouseDataConfig config,
 ClickHouseInputStream input,
 ClickHouseOutputStream output,
 List<ClickHouseColumn> columns,
 Map<String,Serializable> settings)
                           throws IOException

Default constructor.

Parameters:

config - non-null confinguration contains information like format

input - input stream for deserialization, can be null when output is available

output - outut stream for serialization, can be null when input is available

columns - nullable columns

settings - nullable settings

Throws:

IOException - when failed to read columns from input stream

Method Details

hasMoreToRead

protected boolean hasMoreToRead()
                         throws UncheckedIOException

Checks whether there's more to read from input stream.

Returns:

true if there's more; false otherwise

Throws:

UncheckedIOException - when failed to read data from input stream

buildDeserializeSteps

protected ClickHouseDeserializer[] buildDeserializeSteps(ClickHouseColumn column)

Builds list of steps to deserialize value for the given column.

Parameters:

column - non-null column

Returns:

non-null list of steps for deserialization

buildSerializeSteps

protected ClickHouseSerializer[] buildSerializeSteps(ClickHouseColumn column)

Builds list of steps to serialize value for the given column.

Parameters:

column - non-null column

Returns:

non-null list of steps for serialization

createRecord

protected abstract ClickHouseRecord createRecord()

Factory method to create a record.

Returns:

new record

initRecords

protected Iterator<ClickHouseRecord> initRecords()

Initializes iterator of ClickHouseRecord for reading values record by record. Usually this should be only called once during instantiation.

Returns:

non-null iterator of ClickHouseRecord

initValues

protected Iterator<ClickHouseValue> initValues()

Initializes iterator of ClickHouseValue for reading values one by one. Usually this should be only called once during instantiation.

Returns:

non-null iterator of ClickHouseValue

readAndFill

protected void readAndFill(ClickHouseRecord r)
                    throws IOException

Reads columns(starting from readPosition) from input stream and fill deserialized data into the given record. This method is only used when iterating through records().

Parameters:

r - non-null record to fill

Throws:

IOException - when failed to read columns from input stream

readAndFill

protected void readAndFill(ClickHouseValue value)
                    throws IOException

Reads next column(at readPosition from input stream and fill deserialized data into the given value object. This method is mainly used when iterating through values(). In default implementation, it's also used in readAndFill(ClickHouseRecord) for simplicity.

Parameters:

value - non-null value object to fill

Throws:

IOException - when failed to read column from input stream

readColumns

protected abstract List<ClickHouseColumn> readColumns()
                                               throws IOException

Reads columns from input stream. Usually this will be only called once during instantiation.

Returns:

non-null list of columns

Throws:

IOException - when failed to read columns from input stream

getDeserializer

public abstract ClickHouseDeserializer getDeserializer(ClickHouseDataConfig config,
 ClickHouseColumn column)

getDeserializers

public final ClickHouseDeserializer[] getDeserializers(ClickHouseDataConfig config,
 List<ClickHouseColumn> columns)

getSerializer

public abstract ClickHouseSerializer getSerializer(ClickHouseDataConfig config,
 ClickHouseColumn column)

getSerializers

public final ClickHouseSerializer[] getSerializers(ClickHouseDataConfig config,
 List<ClickHouseColumn> columns)

getColumns

public final List<ClickHouseColumn> getColumns()

Gets list of columns to process.

Returns:

list of columns to process

records

public final Iterable<ClickHouseRecord> records()

Returns an iterable collection of records which can be walked through in a foreach-loop. Please pay attention that: 1) UncheckedIOException might be thrown when iterating through the collection; and 2) it's not supposed to be called for more than once because the input stream will be closed at the end of reading.

Returns:

non-null iterable records

Throws:

UncheckedIOException - when failed to access the input stream

values

public final Iterable<ClickHouseValue> values()

Returns an iterable collection of values which can be walked through in a foreach-loop. In general, this is slower than records(), because the latter reads data in bulk. However, it's particular useful when you're reading large values with limited memory - e.g. a binary field with a few GB bytes. Similarly, the input stream will be closed at the end of reading.

Returns:

non-null iterable values

Throws:

UncheckedIOException - when failed to access the input stream

read

public ClickHouseValue read(ClickHouseValue value)
                     throws IOException

Reads deserialized value of next column(at readPosition) directly from input stream. Unlike records(), which reads multiple values at a time, this method will only read one for each call.

Parameters:

value - value to update, could be null

Returns:

updated value or a new ClickHouseValue when it is null

Throws:

IOException - when failed to read data from input stream

write

public void write(ClickHouseValue value)
           throws IOException

Writes serialized value of next column(at readPosition) to output stream.

Parameters:

value - non-null value to be serialized

Throws:

IOException - when failed to write data to output stream