Packages

c

org.apache.spark.sql.execution.datasources.parquet

VectorizedRleValuesReader

final class VectorizedRleValuesReader extends ValuesReader with VectorizedValuesReader

A values reader for Parquet's run-length encoded data. This is based off of the version in parquet-mr with these changes:

  • Supports the vectorized interface.
  • Works on byte arrays(byte[]) instead of making byte streams.

This encoding is used in multiple places:

  • Definition/Repetition levels
  • Dictionary ids.
  • Boolean type values of Parquet DataPageV2
Linear Supertypes
VectorizedValuesReader, ValuesReader, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. VectorizedRleValuesReader
  2. VectorizedValuesReader
  3. ValuesReader
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new VectorizedRleValuesReader(bitWidth: Int, readLength: Boolean)
  2. new VectorizedRleValuesReader(bitWidth: Int)
  3. new VectorizedRleValuesReader()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def initFromPage(valueCount: Int, in: ByteBufferInputStream): Unit
    Definition Classes
    VectorizedRleValuesReader → ValuesReader
    Annotations
    @Override()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. def readBatch(state: ParquetReadState, values: WritableColumnVector, defLevels: WritableColumnVector, valueReader: VectorizedValuesReader, updater: ParquetVectorUpdater): Unit

    Reads a batch of definition levels and values into vector 'defLevels' and 'values' respectively.

    Reads a batch of definition levels and values into vector 'defLevels' and 'values' respectively. The values are read using 'valueReader'.

    The related states such as row index, offset, number of values left in the batch and page, are tracked by 'state'. The type-specific 'updater' is used to update or skip values.

    This reader reads the definition levels and then will read from 'valueReader' for the non-null values. If the value is null, 'values' will be populated with null value.

  17. def readBatchRepeated(state: ParquetReadState, repLevels: WritableColumnVector, defLevelsReader: VectorizedRleValuesReader, defLevels: WritableColumnVector, values: WritableColumnVector, valueReader: VectorizedValuesReader, updater: ParquetVectorUpdater): Unit

    Reads a batch of repetition levels, definition levels and values into 'repLevels', 'defLevels' and 'values' respectively.

    Reads a batch of repetition levels, definition levels and values into 'repLevels', 'defLevels' and 'values' respectively. The definition levels and values are read via 'defLevelsReader' and 'valueReader' respectively.

    The related states such as row index, offset, number of rows left in the batch and page, are tracked by 'state'. The type-specific 'updater' is used to update or skip values.

  18. def readBatchRepeatedInternal(state: ParquetReadState, repLevels: WritableColumnVector, defLevelsReader: VectorizedRleValuesReader, defLevels: WritableColumnVector, values: WritableColumnVector, nulls: WritableColumnVector, valuesReused: Boolean, valueReader: VectorizedValuesReader, updater: ParquetVectorUpdater): Unit

    Keep reading repetition level values from the page until either: 1) we've read enough top-level rows to fill the current batch, or 2) we've drained the data page completely.

    Keep reading repetition level values from the page until either: 1) we've read enough top-level rows to fill the current batch, or 2) we've drained the data page completely.

    valuesReused

    whether 'values' vector is reused for 'nulls'

  19. def readBinary(len: Int): Binary
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  20. def readBinary(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  21. def readBoolean(): Boolean
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader → ValuesReader
    Annotations
    @Override()
  22. def readBooleans(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  23. def readByte(): Byte
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  24. def readBytes(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  25. def readBytes(): Binary
    Definition Classes
    ValuesReader
  26. def readDouble(): Double
    Definition Classes
    ValuesReader
  27. def readDoubles(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  28. def readFloat(): Float
    Definition Classes
    ValuesReader
  29. def readFloats(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  30. def readInteger(): Int
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader → ValuesReader
    Annotations
    @Override()
  31. def readIntegers(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  32. def readIntegers(state: ParquetReadState, values: WritableColumnVector, nulls: WritableColumnVector, defLevels: WritableColumnVector, valueReader: VectorizedValuesReader): Unit

    Decoding for dictionary ids.

    Decoding for dictionary ids. The IDs are populated into 'values' and the nullability is populated into 'nulls'.

  33. def readIntegersRepeated(state: ParquetReadState, repLevels: WritableColumnVector, defLevelsReader: VectorizedRleValuesReader, defLevels: WritableColumnVector, values: WritableColumnVector, nulls: WritableColumnVector, valueReader: VectorizedValuesReader): Unit

    Reads a batch of repetition levels, definition levels and integer values into 'repLevels', 'defLevels', 'values' and 'nulls' respectively.

    Reads a batch of repetition levels, definition levels and integer values into 'repLevels', 'defLevels', 'values' and 'nulls' respectively. The definition levels and values are read via 'defLevelsReader' and 'valueReader' respectively.

    The 'values' vector is used to hold non-null values, while 'nulls' vector is used to hold null values.

    The related states such as row index, offset, number of rows left in the batch and page, are tracked by 'state'.

    Unlike 'readBatchRepeated', this is used to decode dictionary indices in dictionary encoding.

  34. def readIntegersWithRebase(total: Int, c: WritableColumnVector, rowId: Int, failIfRebase: Boolean): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  35. def readLong(): Long
    Definition Classes
    ValuesReader
  36. def readLongs(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  37. def readLongsWithRebase(total: Int, c: WritableColumnVector, rowId: Int, failIfRebase: Boolean, timeZone: String): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  38. def readShort(): Short
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  39. def readShorts(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  40. def readUnsignedIntegers(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  41. def readUnsignedLongs(total: Int, c: WritableColumnVector, rowId: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  42. def readValueDictionaryId(): Int
    Definition Classes
    VectorizedRleValuesReader → ValuesReader
    Annotations
    @Override()
  43. def skip(): Unit
    Definition Classes
    VectorizedRleValuesReader → ValuesReader
    Annotations
    @Override()
  44. def skip(arg0: Int): Unit
    Definition Classes
    ValuesReader
  45. def skipBinary(total: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  46. def skipBooleans(total: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  47. def skipBytes(total: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  48. def skipDoubles(total: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  49. def skipFixedLenByteArray(total: Int, len: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  50. def skipFloats(total: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  51. def skipIntegers(total: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  52. def skipLongs(total: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  53. def skipShorts(total: Int): Unit
    Definition Classes
    VectorizedRleValuesReader → VectorizedValuesReader
    Annotations
    @Override()
  54. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  55. def toString(): String
    Definition Classes
    AnyRef → Any
  56. def updateNextOffset(arg0: Int): Unit
    Attributes
    protected[values]
    Definition Classes
    ValuesReader
  57. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  58. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  59. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Deprecated Value Members

  1. def getNextOffset(): Int
    Definition Classes
    ValuesReader
    Annotations
    @Deprecated
    Deprecated
  2. def initFromPage(arg0: Int, arg1: Array[Byte], arg2: Int): Unit
    Definition Classes
    ValuesReader
    Annotations
    @throws( classOf[java.io.IOException] ) @Deprecated
    Deprecated
  3. def initFromPage(arg0: Int, arg1: ByteBuffer, arg2: Int): Unit
    Definition Classes
    ValuesReader
    Annotations
    @throws( classOf[java.io.IOException] ) @Deprecated
    Deprecated

Inherited from VectorizedValuesReader

Inherited from ValuesReader

Inherited from AnyRef

Inherited from Any

Ungrouped