Package com.clickhouse.data

Class ClickHouseInputStream

java.lang.Object
java.io.InputStream
com.clickhouse.data.ClickHouseInputStream
All Implemented Interfaces:
Closeable, AutoCloseable, Iterable<ClickHouseByteBuffer>
public abstract class ClickHouseInputStream extends InputStream implements Iterable<ClickHouseByteBuffer>
Extended input stream for read optimization. Methods like readBuffer(int), readByte(), readBytes(int), and readCustom(ClickHouseDataUpdater) are added to reduce object creation as well as closing the stream when it reaches end of stream. This class is also responsible for creating various input stream as needed.

  • Field Details

  • Constructor Details

  • Method Details

    • wrap

      public static ClickHouseInputStream wrap(ClickHousePassThruStream stream, InputStream input, int bufferSize, ClickHouseCompression compression, int compressionLevel, Runnable postCloseAction)
      Wraps the given input stream.
      Parameters:
      stream - pass-thru stream, could be null
      input - non-null input stream
      bufferSize - buffer size
      compression - compression algorithm
      compressionLevel - compression level
      postCloseAction - custom action will be performed right after closing the wrapped input stream
      Returns:
      non-null wrapped input stream

    • wrap

      public static ClickHouseInputStream wrap(InputStream input, int bufferSize, long length, Runnable postCloseAction)
      Wraps the given input stream with length limitation. Please pay attention that calling close() method of the wrapper will never close the inner input stream.
      Parameters:
      input - non-null input stream
      bufferSize - buffer size
      length - maximum bytes can be read from the input
      postCloseAction - custom action will be performed right after closing the wrapped input stream
      Returns:
      non-null wrapped input stream

    • empty

      public static ClickHouseInputStream empty()
      Gets an empty input stream that produces nothing and cannot be closed.
      Returns:
      empty input stream

    • of

      public static ClickHouseInputStream of(BlockingQueue<ByteBuffer> queue, int timeout)
      Wraps the given blocking queue.
      Parameters:
      queue - non-null blocking queue
      timeout - read timeout in milliseconds
      Returns:
      wrapped input

    • of

      public static ClickHouseInputStream of(BlockingQueue<ByteBuffer> queue, int timeout, Runnable postCloseAction)
      Wraps the given blocking queue.
      Parameters:
      queue - non-null blocking queue
      timeout - read timeout in milliseconds
      postCloseAction - custom action will be performed right after closing the input stream
      Returns:
      wrapped input

    • of

      public static ClickHouseInputStream of(ClickHouseDeferredValue<InputStream> deferredInput, int bufferSize, Runnable postCloseAction)
      Wraps the deferred input stream.
      Parameters:
      deferredInput - non-null deferred input stream
      bufferSize - buffer size which is always greater than zero(usually 4096 or larger)
      postCloseAction - custom action will be performed right after closing the input stream
      Returns:
      wrapped input

    • of

      public static ClickHouseInputStream of(ClickHousePassThruStream stream, int bufferSize, Runnable postCloseAction)
      Wraps the given pass-thru stream as input stream.
      Parameters:
      stream - non-null pass-thru stream
      bufferSize - buffer size which is always greater than zero(usually 4096 or larger)
      postCloseAction - custom action will be performed right after closing the input stream
      Returns:
      wrapped input

    • of

      public static ClickHouseInputStream of(ClickHouseWriter writer)
      Creates an input stream using the given customer writer. Behind the scene, a piped stream will be created, writer will be called in a separate worker thread for writing.
      Parameters:
      writer - non-null customer writer
      Returns:
      wrapped input

    • of

      public static ClickHouseInputStream of(ClickHouseDataConfig config, ClickHouseWriter writer)
      Creates an input stream using the given customer writer. Behind the scene, a piped stream will be created, writer will be called in a separate worker thread for writing.
      Parameters:
      config - configuration, could be null
      writer - non-null customer writer
      Returns:
      wrapped input

    • of

      public static ClickHouseInputStream of(InputStream input)
      Wraps the given input stream.
      Parameters:
      input - input stream
      Returns:
      wrapped input, or the same input if it's instance of ClickHouseInputStream

    • of

      public static ClickHouseInputStream of(InputStream input, int bufferSize)
      Wraps the given input stream.
      Parameters:
      input - input stream
      bufferSize - buffer size which is always greater than zero(usually 4096 or larger)
      Returns:
      wrapped input, or the same input if it's instance of ClickHouseInputStream

    • of

      public static ClickHouseInputStream of(InputStream input, int bufferSize, Runnable postCloseAction)
      Wraps the given input stream.
      Parameters:
      input - input stream
      bufferSize - buffer size which is always greater than zero(usually 4096 or larger)
      postCloseAction - custom action will be performed right after closing the input stream
      Returns:
      wrapped input, or the same input if it's instance of ClickHouseInputStream

    • of

      public static ClickHouseInputStream of(InputStream input, ClickHouseCompression compression)
      Wraps the given input stream.
      Parameters:
      input - input stream
      compression - compression algorithm, null or ClickHouseCompression.NONE means no compression
      Returns:
      wrapped input, or the same input if it's instance of ClickHouseInputStream

    • of

      public static ClickHouseInputStream of(InputStream input, int bufferSize, ClickHouseCompression compression, int compressionLevel, Runnable postCloseAction)
      Wraps the given input stream.
      Parameters:
      input - input stream
      bufferSize - buffer size which is always greater than zero(usually 4096 or larger)
      compression - compression algorithm, null or ClickHouseCompression.NONE means no compression
      compressionLevel - compression level
      postCloseAction - custom action will be performed right after closing the input stream
      Returns:
      wrapped input, or the same input if it's instance of ClickHouseInputStream

    • of

      public static ClickHouseInputStream of(byte[]... bytes)
      Wraps the given byte arrays.
      Parameters:
      bytes - array of byte array
      Returns:
      non-null input stream
      See Also:

    • of

      public static ClickHouseInputStream of(ByteBuffer... buffers)
      Wraps the given byte buffers.
      Parameters:
      buffers - array of byte buffer
      Returns:
      non-null input stream
      See Also:

    • of

      public static ClickHouseInputStream of(File... files)
      Wraps the given files.
      Parameters:
      files - array of file
      Returns:
      non-null input stream
      See Also:

    • of

      public static ClickHouseInputStream of(InputStream... inputs)
      Wraps the given input streams.
      Parameters:
      inputs - array of input stream
      Returns:
      non-null input stream
      See Also:

    • of

      public static ClickHouseInputStream of(String... strings)
      Wraps the given (UTF-8)strings.
      Parameters:
      strings - array of string
      Returns:
      non-null input stream
      See Also:

    • of

      public static ClickHouseInputStream of(URL... urls)
      Wraps the given URLs.
      Parameters:
      urls - array of URL
      Returns:
      non-null input stream
      See Also:

    • of

      public static <T> ClickHouseInputStream of(T[] source, Class<T> clazz, Function<T,byte[]> converter, Runnable postCloseAction)
      Wraps the given array of object as byte array based binary input stream.
      Type Parameters:
      T - type of the object
      Parameters:
      source - array of object
      clazz - class of the object
      converter - optional transformer to convert each object into byte array
      postCloseAction - custom action will be performed right after closing the input stream
      Returns:
      non-null input stream
      See Also:

    • of

      public static <T> ClickHouseInputStream of(Iterable<T> source, Class<T> clazz, Function<T,byte[]> converter, Runnable postCloseAction)
      Wraps the given iterable objects as byte array based binary input stream. byte[], ByteBuffer, InputStream, File, String, and URL are all supported by default.
      Type Parameters:
      T - type of the object
      Parameters:
      source - iterable objects(e.g. byte[], ByteBuffer, and String etc.)
      clazz - class of the object
      converter - optional transformer to convert each object into byte array
      postCloseAction - custom action will be performed right after closing the input stream
      Returns:
      non-null input stream

    • of

      public static <T> ClickHouseInputStream of(Iterable<T> source, Class<T> clazz, Function<T,byte[]> converter, int bufferSize, Runnable postCloseAction)
      Wraps the given iterable objects as byte array based binary input stream. byte[], ByteBuffer, InputStream, File, String, and URL are all supported by default.
      Type Parameters:
      T - type of the object
      Parameters:
      source - iterable objects(e.g. byte[], ByteBuffer, and String etc.)
      clazz - class of the object
      converter - optional transformer to convert each object into byte array
      bufferSize - read buffer size
      postCloseAction - custom action will be performed right after closing the input stream
      Returns:
      non-null input stream

    • pipe

      public static long pipe(InputStream input, OutputStream output, int bufferSize) throws IOException
      Transfers data from input stream to output stream. Input stream will be closed but output stream will remain open. Please pay attention that you need to explictly call output.flush() before closing output stream.
      Parameters:
      input - non-null input stream, which will be closed
      output - non-null output stream, which will remain open
      bufferSize - buffer size, zero or negative number will be treated as ClickHouseDataConfig.DEFAULT_BUFFER_SIZE
      Returns:
      written bytes
      Throws:
      IOException - when error occured reading from input stream or writing data to output stream

    • pipe

      public static long pipe(InputStream input, OutputStream output, byte[] buffer) throws IOException
      Transfers data from input stream to output stream. Input stream will be closed but output stream will remain open. Please pay attention that you need to explictly call output.flush() before closing output stream.
      Parameters:
      input - non-null input stream, which will be closed
      output - non-null output stream, which will remain open
      buffer - non-empty buffer
      Returns:
      written bytes
      Throws:
      IOException - when error occured reading from input stream or writing data to output stream

    • save

      public static File save(InputStream input)
      Saves data from the given input stream to a temporary file, which will be deleted after JVM exited.
      Parameters:
      input - non-null input stream
      Returns:
      non-null temporary file

    • save

      public static File save(InputStream input, File file)
      Saves data from the given input stream to the specified file.
      Parameters:
      input - non-null input stream
      file - target file, could be null
      Returns:
      non-null file

    • save

      public static File save(ClickHouseDataConfig config, InputStream input)
      Saves data from the given input stream to a temporary file, which will be deleted after JVM exited.
      Parameters:
      config - config, could be null
      input - non-null input stream
      Returns:
      non-null temporary file

    • save

      public static File save(ClickHouseDataConfig config, InputStream input, File file)
      Saves data from the given input stream to the specified file.
      Parameters:
      config - config, could be null
      input - non-null input stream
      file - target file, could be null
      Returns:
      non-null file

    • save

      @Deprecated public static File save(InputStream in, int bufferSize, int timeout)
      Deprecated.
      will be dropped in 0.5, please use save(InputStream) instead
      Saves data from the given input stream to a temporary file.
      Parameters:
      in - non-null input stream
      bufferSize - buffer size
      timeout - timeout in milliseconds
      Returns:
      non-null temporary file

    • save

      @Deprecated public static File save(File file, InputStream in, int bufferSize, int timeout, boolean deleteOnExit)
      Deprecated.
      will be dropped in 0.5, please use save(InputStream, File) instead
      Saves data from the given input stream to the specified file.
      Parameters:
      file - target file, could be null
      in - non-null input stream
      bufferSize - buffer size
      timeout - timeout in milliseconds
      deleteOnExit - whether the file should be deleted after JVM exit
      Returns:
      non-null file

    • closeQuietly

      protected void closeQuietly()
      Closes the input stream quietly.

    • ensureOpen

      protected void ensureOpen() throws IOException
      Checks whether the input stream has been closed and throw an exception if it is.
      Throws:
      IOException - when the input stream has been closed

    • getBuffer

      protected abstract ClickHouseByteBuffer getBuffer()
      Gets reference to current byte buffer.
      Returns:
      non-null byte buffer

    • reusableBuffer

      protected boolean reusableBuffer()
      Checks whether the internal buffer is reused among multiple reads.
      Returns:
      true if the internal buffer is reused among multiple reads; false otherwise

    • nextBuffer

      protected abstract ClickHouseByteBuffer nextBuffer() throws IOException
      Gets reference to next byte buffer available for read. An empty byte buffer will be returned (nextBuffer().isEmpty() == true), when it reaches end of the input stream.
      Returns:
      non-null byte buffer
      Throws:
      IOException

    • getUnderlyingFile

      public ClickHouseFile getUnderlyingFile()
      Gets underlying file. Same as ClickHouseFile.of(getUnderlyingStream()).
      Returns:
      non-null underlying file

    • getUnderlyingStream

      public ClickHousePassThruStream getUnderlyingStream()
      Gets underlying stream.
      Returns:
      non-null underlying stream

    • hasUnderlyingStream

      public boolean hasUnderlyingStream()
      Checks if there's underlying input stream. Same as getUnderlyingStream().hasInput().
      Returns:
      true if there's underlying input stream; false otherwise

    • getUserData

      public final Object getUserData(String key)
      Gets user data associated with this input stream.
      Parameters:
      key - key
      Returns:
      value, could be null

    • getUserData

      public final <T> T getUserData(String key, T defaultValue)
      Gets user data associated with this input stream.
      Type Parameters:
      T - type of the value
      Parameters:
      key - key
      defaultValue - default value
      Returns:
      value, could be null

    • removeUserData

      public final Object removeUserData(String key)
      Removes user data.
      Parameters:
      key - key
      Returns:
      removed user data, could be null

    • setUserData

      public final <T> T setUserData(String key, T value)
      Sets user data.
      Parameters:
      key - key
      value - value
      Returns:
      overidded value, could be null

    • peek

      public abstract int peek() throws IOException
      Peeks one byte. It's similar as InputStream.read() except it never changes cursor.
      Returns:
      the next byte of data, or -1 if the end of the stream is reached
      Throws:
      IOException - when failed to read value from input stream or reached end of the stream

    • pipe

      public long pipe(ClickHouseOutputStream output) throws IOException
      Reads all remaining bytes and write into given output stream. Current input stream will be closed automatically at the end of writing, but output will remain open.
      Parameters:
      output - non-null output stream
      Returns:
      bytes being written into output stream
      Throws:
      IOException - when failed to read value from input stream or reached end of the stream

    • readUnsignedByte

      public short readUnsignedByte() throws IOException
      Reads an unsigned byte from the input stream. Unlike InputStream.read(), it will throw IOException if the input stream has been closed.
      Returns:
      unsigned byte
      Throws:
      IOException - when failed to read value from input stream or reached end of the stream

    • readBuffer

      public ClickHouseByteBuffer readBuffer(int length) throws IOException
      Reads byte buffer from the input stream.
      Parameters:
      length - byte length
      Returns:
      non-null byte buffer
      Throws:
      IOException - when failed to read bytes from input stream, not able to retrieve all bytes, or reached end of the stream

    • readBufferUntil

      public abstract ClickHouseByteBuffer readBufferUntil(byte[] separator) throws IOException
      Reads byte buffer from the input stream until the first match of the separator.
      Parameters:
      separator - non-empty separator
      Returns:
      non-null byte buffer
      Throws:
      IOException - when failed to read bytes from input stream or not able to retrieve all bytes

    • readBoolean

      public boolean readBoolean() throws IOException
      Reads a byte as boolean. The byte value can be either 0 (false) or 1 (true).
      Returns:
      boolean value
      Throws:
      IOException - when failed to read boolean value from input stream or reached end of the stream

    • readByte

      public abstract byte readByte() throws IOException
      Reads one single byte from the input stream. Unlike InputStream.read(), it will throw IOException if the input stream has been closed. In general, this method should be faster than InputStream.read(), especially when it's an input stream backed by byte[] or ByteBuffer.
      Returns:
      byte value if present
      Throws:
      IOException - when failed to read value from input stream or reached end of the stream

    • readBytes

      public byte[] readBytes(int length) throws IOException
      Reads length bytes from the input stream. It behaves in the same way as DataInput.readFully(byte[]), except it will throw IOException when the input stream has been closed.
      Parameters:
      length - number of bytes to read
      Returns:
      byte array and its length should be length
      Throws:
      IOException - when failed to read value from input stream, not able to retrieve all bytes, or reached end of the stream

    • readCustom

      public abstract ClickHouseByteBuffer readCustom(ClickHouseDataUpdater reader) throws IOException
      Reads bytes using custom reader. Stream will be closed automatically when it reached end of stream. However, unlike readBuffer(int), this method will never throw EOFException.
      Parameters:
      reader - non-null data reader
      Returns:
      non-null byte buffer
      Throws:
      IOException - when failed to read bytes from input stream, not able to retrieve all bytes, or reached end of the stream

    • readString

      public String readString(Charset charset) throws IOException
      Reads binary string from the input stream. readVarInt() will be first called automatically to understand byte length of the string.
      Parameters:
      charset - charset, null is treated as StandardCharsets.UTF_8
      Returns:
      non-null string
      Throws:
      IOException - when failed to read value from input stream, not able to retrieve all bytes, or reached end of the stream

    • readString

      public String readString(int byteLength, Charset charset) throws IOException
      Reads binary string from the input stream. When byteLength is zero or negative number, this method will always return empty string.
      Parameters:
      byteLength - length in byte
      charset - charset, null is treated as StandardCharsets.UTF_8
      Returns:
      non-null string
      Throws:
      IOException - when failed to read value from input stream, not able to retrieve all bytes, or reached end of the stream

    • readAsciiString

      public String readAsciiString() throws IOException
      Reads ascii string from input stream. readVarInt() will be first called automatically to understand byte length of the string.
      Returns:
      non-null ascii string
      Throws:
      IOException - when failed to read value from input stream, not able to retrieve all bytes, or reached end of the stream

    • readAsciiString

      public String readAsciiString(int byteLength) throws IOException
      Reads ascii string from input stream. Similar as readString(byteLength, StandardCharsets.US_ASCII).
      Parameters:
      byteLength - length in byte
      Returns:
      non-null ascii string
      Throws:
      IOException - when failed to read value from input stream, not able to retrieve all bytes, or reached end of the stream

    • readUnicodeString

      public String readUnicodeString() throws IOException
      Reads unicode string from input stream.
      Returns:
      non-null unicode string
      Throws:
      IOException - when failed to read value from input stream, not able to retrieve all bytes, or reached end of the stream

    • readUnicodeString

      public String readUnicodeString(int byteLength) throws IOException
      Reads unicode string from input stream. Similar as readString(byteLength, null).
      Parameters:
      byteLength - length in byte
      Returns:
      non-null unicode string
      Throws:
      IOException - when failed to read value from input stream, not able to retrieve all bytes, or reached end of the stream

    • readVarInt

      public int readVarInt() throws IOException
      Reads a varint from input stream.
      Returns:
      varint
      Throws:
      IOException - when failed to read value from input stream or reached end of the stream

    • readVarLong

      public long readVarLong() throws IOException
      Reads 64-bit varint as long from input stream.
      Returns:
      64-bit varint
      Throws:
      IOException - when failed to read value from input stream or reached end of the stream

    • setCopyToTarget

      public final void setCopyToTarget(OutputStream out) throws IOException
      Sets target output stream to copy bytes to. This is mainly used for testing, for example: dump input into a file while reading.
      Parameters:
      out - the output stream to write bytes to
      Throws:
      IOException - when failed to flush previous target or not able to write remaining bytes in buffer to the given output stream

    • isClosed

      public boolean isClosed()
      Checks if the input stream has been closed or not.
      Returns:
      true if the input stream has been closed; false otherwise

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Overrides:
      close in class InputStream
      Throws:
      IOException

    • iterator

      public Iterator<ClickHouseByteBuffer> iterator()
      Specified by:
      iterator in interface Iterable<ClickHouseByteBuffer>