Packages

c

org.apache.spark.sql.execution.vectorized

WritableColumnVector

abstract class WritableColumnVector extends ColumnVector

This class adds write APIs to ColumnVector. It supports all the types and contains put APIs as well as their batched versions. The batched versions are preferable whenever possible.

Capacity: The data stored is dense but the arrays are not fixed capacity. It is the responsibility of the caller to call reserve() to ensure there is enough room before adding elements. This means that the put() APIs do not check as in common cases (i.e. flat schemas), the lengths are known up front.

A WritableColumnVector should be considered immutable once originally created. In other words, it is not valid to call put APIs after reads until reset() is called.

WritableColumnVector are intended to be reused.

Linear Supertypes
ColumnVector, AutoCloseable, AnyRef, Any
Known Subclasses
OffHeapColumnVector, OnHeapColumnVector
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WritableColumnVector
  2. ColumnVector
  3. AutoCloseable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new WritableColumnVector(capacity: Int, type: DataType)
    Attributes
    protected[vectorized]

Abstract Value Members

  1. abstract def getArrayLength(rowId: Int): Int
  2. abstract def getArrayOffset(rowId: Int): Int
  3. abstract def getBoolean(arg0: Int): Boolean
    Definition Classes
    ColumnVector
  4. abstract def getByte(arg0: Int): Byte
    Definition Classes
    ColumnVector
  5. abstract def getByteBuffer(rowId: Int, count: Int): ByteBuffer

    Gets the values of bytes from [rowId, rowId + count), as a ByteBuffer.

    Gets the values of bytes from [rowId, rowId + count), as a ByteBuffer. This method is similar to int), but avoids making a copy.

  6. abstract def getBytesAsUTF8String(rowId: Int, count: Int): UTF8String

    Gets the values of bytes from [rowId, rowId + count), as a UTF8String.

    Gets the values of bytes from [rowId, rowId + count), as a UTF8String. This method is similar to int), but can save data copy as UTF8String is used as a pointer.

    Attributes
    protected[vectorized]
  7. abstract def getDictId(rowId: Int): Int

    Returns the dictionary Id for rowId.

    Returns the dictionary Id for rowId.

    This should only be called when this WritableColumnVector represents dictionaryIds. We have this separate method for dictionaryIds as per SPARK-16928.

  8. abstract def getDouble(arg0: Int): Double
    Definition Classes
    ColumnVector
  9. abstract def getFloat(arg0: Int): Float
    Definition Classes
    ColumnVector
  10. abstract def getInt(arg0: Int): Int
    Definition Classes
    ColumnVector
  11. abstract def getLong(arg0: Int): Long
    Definition Classes
    ColumnVector
  12. abstract def getShort(arg0: Int): Short
    Definition Classes
    ColumnVector
  13. abstract def isNullAt(arg0: Int): Boolean
    Definition Classes
    ColumnVector
  14. abstract def putArray(rowId: Int, offset: Int, length: Int): Unit

    Puts a byte array that already exists in this column.

  15. abstract def putBoolean(rowId: Int, value: Boolean): Unit

    Sets value to the value at rowId.

  16. abstract def putBooleans(rowId: Int, src: Byte): Unit

    Sets bits from [src[0], src[7]] to [rowId, rowId + 7] src must contain bit-packed 8 booleans in the byte.

  17. abstract def putBooleans(rowId: Int, count: Int, value: Boolean): Unit

    Sets value to [rowId, rowId + count).

  18. abstract def putByte(rowId: Int, value: Byte): Unit

    Sets value to the value at rowId.

  19. abstract def putByteArray(rowId: Int, value: Array[Byte], offset: Int, count: Int): Int

    Sets values from [value + offset, value + offset + count) to the values at rowId.

  20. abstract def putBytes(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count]) to [rowId, rowId + count)

  21. abstract def putBytes(rowId: Int, count: Int, value: Byte): Unit

    Sets value to [rowId, rowId + count).

  22. abstract def putDouble(rowId: Int, value: Double): Unit

    Sets value to the value at rowId.

  23. abstract def putDoubles(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count * 8]) to [rowId, rowId + count) The data in src must be ieee formatted doubles in platform native endian.

  24. abstract def putDoubles(rowId: Int, count: Int, src: Array[Double], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count]) to [rowId, rowId + count)

  25. abstract def putDoubles(rowId: Int, count: Int, value: Double): Unit

    Sets value to [rowId, rowId + count).

  26. abstract def putDoublesLittleEndian(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count * 8]) to [rowId, rowId + count) The data in src must be ieee formatted doubles in little endian.

  27. abstract def putFloat(rowId: Int, value: Float): Unit

    Sets value to the value at rowId.

  28. abstract def putFloats(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count * 4]) to [rowId, rowId + count) The data in src must be ieee formatted floats in platform native endian.

  29. abstract def putFloats(rowId: Int, count: Int, src: Array[Float], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count]) to [rowId, rowId + count)

  30. abstract def putFloats(rowId: Int, count: Int, value: Float): Unit

    Sets value to [rowId, rowId + count).

  31. abstract def putFloatsLittleEndian(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count * 4]) to [rowId, rowId + count) The data in src must be ieee formatted floats in little endian.

  32. abstract def putInt(rowId: Int, value: Int): Unit

    Sets value to the value at rowId.

  33. abstract def putInts(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count * 4]) to [rowId, rowId + count) The data in src must be 4-byte platform native endian ints.

  34. abstract def putInts(rowId: Int, count: Int, src: Array[Int], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count]) to [rowId, rowId + count)

  35. abstract def putInts(rowId: Int, count: Int, value: Int): Unit

    Sets value to [rowId, rowId + count).

  36. abstract def putIntsLittleEndian(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count * 4]) to [rowId, rowId + count) The data in src must be 4-byte little endian ints.

  37. abstract def putLong(rowId: Int, value: Long): Unit

    Sets value to the value at rowId.

  38. abstract def putLongs(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count * 8]) to [rowId, rowId + count) The data in src must be 8-byte platform native endian longs.

  39. abstract def putLongs(rowId: Int, count: Int, src: Array[Long], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count]) to [rowId, rowId + count)

  40. abstract def putLongs(rowId: Int, count: Int, value: Long): Unit

    Sets value to [rowId, rowId + count).

  41. abstract def putLongsLittleEndian(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src + srcIndex, src + srcIndex + count * 8) to [rowId, rowId + count) The data in src must be 8-byte little endian longs.

  42. abstract def putNotNull(rowId: Int): Unit

    Sets null/not null to the value at rowId.

  43. abstract def putNotNulls(rowId: Int, count: Int): Unit
  44. abstract def putNull(rowId: Int): Unit
  45. abstract def putNulls(rowId: Int, count: Int): Unit

    Sets null/not null to the values at [rowId, rowId + count).

  46. abstract def putShort(rowId: Int, value: Short): Unit

    Sets value to the value at rowId.

  47. abstract def putShorts(rowId: Int, count: Int, src: Array[Byte], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count * 2]) to [rowId, rowId + count) The data in src must be 2-byte platform native endian shorts.

  48. abstract def putShorts(rowId: Int, count: Int, src: Array[Short], srcIndex: Int): Unit

    Sets values from [src[srcIndex], src[srcIndex + count]) to [rowId, rowId + count)

  49. abstract def putShorts(rowId: Int, count: Int, value: Short): Unit

    Sets value to [rowId, rowId + count).

  50. abstract def reserveInternal(capacity: Int): Unit

    Ensures that there is enough storage to store capacity elements.

    Ensures that there is enough storage to store capacity elements. That is, the put() APIs must work for all rowIds < capacity.

    Attributes
    protected[vectorized]
  51. abstract def reserveNewColumn(capacity: Int, type: DataType): WritableColumnVector

    Reserve a new column.

    Reserve a new column.

    Attributes
    protected[vectorized]

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def addElementsAppended(num: Int): Unit

    Increment number of elements appended by 'num'.

    Increment number of elements appended by 'num'.

    This is useful when one wants to use the 'putXXX' API to add new elements to the vector, but still want to keep count of how many elements have been added (since the 'putXXX' APIs don't increment count).

  5. final def appendArray(length: Int): Int
  6. final def appendBoolean(v: Boolean): Int
  7. final def appendBooleans(count: Int, src: Byte, offset: Int): Int

    Append bits from [src[offset], src[offset + count]) src must contain bit-packed 8 booleans in the byte.

  8. final def appendBooleans(count: Int, v: Boolean): Int
  9. final def appendByte(v: Byte): Int
  10. final def appendByteArray(value: Array[Byte], offset: Int, length: Int): Int
  11. final def appendBytes(length: Int, src: Array[Byte], offset: Int): Int
  12. final def appendBytes(count: Int, v: Byte): Int
  13. final def appendDouble(v: Double): Int
  14. final def appendDoubles(length: Int, src: Array[Double], offset: Int): Int
  15. final def appendDoubles(count: Int, v: Double): Int
  16. final def appendFloat(v: Float): Int
  17. final def appendFloats(length: Int, src: Array[Float], offset: Int): Int
  18. final def appendFloats(count: Int, v: Float): Int
  19. final def appendInt(v: Int): Int
  20. final def appendInts(length: Int, src: Array[Int], offset: Int): Int
  21. final def appendInts(count: Int, v: Int): Int
  22. final def appendLong(v: Long): Int
  23. final def appendLongs(length: Int, src: Array[Long], offset: Int): Int
  24. final def appendLongs(count: Int, v: Long): Int
  25. final def appendNotNull(): Int
  26. final def appendNotNulls(count: Int): Int
  27. final def appendNull(): Int

    Append APIs.

    Append APIs. These APIs all behave similarly and will append data to the current vector. It is not valid to mix the put and append APIs. The append APIs are slower and should only be used if the sizes are not known up front. In all these cases, the return value is the rowId for the first appended element.

  28. final def appendNulls(count: Int): Int
  29. final def appendShort(v: Short): Int
  30. final def appendShorts(length: Int, src: Array[Short], offset: Int): Int
  31. final def appendShorts(count: Int, v: Short): Int
  32. final def appendStruct(isNull: Boolean): Int

    Appends a NULL struct.

    Appends a NULL struct. This *has* to be used for structs instead of appendNull() as this recursively appends a NULL to its children. We don't have this logic as the general appendNull implementation to optimize the more common non-struct case.

  33. def arrayData(): WritableColumnVector
  34. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  35. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  36. def close(): Unit
    Definition Classes
    WritableColumnVector → ColumnVector → AutoCloseable
    Annotations
    @Override()
  37. final def dataType(): DataType
    Definition Classes
    ColumnVector
  38. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  39. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  40. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  41. final def getArray(rowId: Int): ColumnarArray
    Definition Classes
    WritableColumnVector → ColumnVector
    Annotations
    @Override()
  42. def getBinary(rowId: Int): Array[Byte]
    Definition Classes
    WritableColumnVector → ColumnVector
    Annotations
    @Override()
  43. def getBooleans(arg0: Int, arg1: Int): Array[Boolean]
    Definition Classes
    ColumnVector
  44. def getBytes(arg0: Int, arg1: Int): Array[Byte]
    Definition Classes
    ColumnVector
  45. def getChild(ordinal: Int): WritableColumnVector
    Definition Classes
    WritableColumnVector → ColumnVector
    Annotations
    @Override()
  46. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  47. def getDecimal(rowId: Int, precision: Int, scale: Int): Decimal
    Definition Classes
    WritableColumnVector → ColumnVector
    Annotations
    @Override()
  48. def getDictionaryIds(): WritableColumnVector

    Returns the underlying integer column for ids of dictionary.

  49. def getDoubles(arg0: Int, arg1: Int): Array[Double]
    Definition Classes
    ColumnVector
  50. final def getElementsAppended(): Int

    Returns the elements appended.

    Returns the elements appended. This is useful

  51. def getFloats(arg0: Int, arg1: Int): Array[Float]
    Definition Classes
    ColumnVector
  52. final def getInterval(arg0: Int): CalendarInterval
    Definition Classes
    ColumnVector
  53. def getInts(arg0: Int, arg1: Int): Array[Int]
    Definition Classes
    ColumnVector
  54. def getLongs(arg0: Int, arg1: Int): Array[Long]
    Definition Classes
    ColumnVector
  55. final def getMap(rowId: Int): ColumnarMap
    Definition Classes
    WritableColumnVector → ColumnVector
    Annotations
    @Override()
  56. def getNumChildren(): Int

    Returns the number of child vectors.

  57. def getShorts(arg0: Int, arg1: Int): Array[Short]
    Definition Classes
    ColumnVector
  58. final def getStruct(arg0: Int): ColumnarRow
    Definition Classes
    ColumnVector
  59. def getUTF8String(rowId: Int): UTF8String
    Definition Classes
    WritableColumnVector → ColumnVector
    Annotations
    @Override()
  60. def hasDictionary(): Boolean

    Returns true if this column has a dictionary.

  61. def hasNull(): Boolean
    Definition Classes
    WritableColumnVector → ColumnVector
    Annotations
    @Override()
  62. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  63. final def isAllNull(): Boolean

    Whether this column only contains null values.

  64. def isArray(): Boolean
    Attributes
    protected[vectorized]
  65. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  66. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  67. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  68. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  69. def numNulls(): Int
    Definition Classes
    WritableColumnVector → ColumnVector
    Annotations
    @Override()
  70. def putBooleans(rowId: Int, count: Int, src: Byte, srcIndex: Int): Unit

    Sets bits from [src[srcIndex], src[srcIndex + count]) to [rowId, rowId + count) src must contain bit-packed 8 booleans in the byte.

  71. final def putByteArray(rowId: Int, value: Array[Byte]): Int
  72. def putDecimal(rowId: Int, value: Decimal, precision: Int): Unit
  73. def putInterval(rowId: Int, value: CalendarInterval): Unit
  74. def reserve(requiredCapacity: Int): Unit
  75. def reserveAdditional(additionalCapacity: Int): Unit
  76. def reserveDictionaryIds(capacity: Int): WritableColumnVector

    Reserve a integer column for ids of dictionary.

  77. def reset(): Unit

    Resets this column for writing.

    Resets this column for writing. The currently stored values are no longer accessible.

  78. final def setAllNull(): Unit

    Marks this column only contains null values.

  79. def setDictionary(dictionary: Dictionary): Unit

    Update the dictionary.

  80. final def setIsConstant(): Unit

    Marks this column as being constant.

  81. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  82. def toString(): String
    Definition Classes
    AnyRef → Any
  83. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  84. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  85. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from ColumnVector

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped