Packages

class RapidsTable extends RapidsBufferBase with RapidsBufferChannelWritable

A RapidsTable is the spill store holder of a cuDF Table.

The table is not contiguous in GPU memory. Instead, this RapidsBuffer instance allows us to use the cuDF chunked_pack API to make the table contiguous as the spill is happening.

This class owns the cuDF table and will close it when close is called.

Linear Supertypes
RapidsBufferChannelWritable, RapidsDeviceMemoryStore.RapidsBufferBase, RapidsBuffer, AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RapidsTable
  2. RapidsBufferChannelWritable
  3. RapidsBufferBase
  4. RapidsBuffer
  5. AutoCloseable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new RapidsTable(id: RapidsBufferId, table: Table, spillPriority: Long)

    id

    the RapidsBufferId this table is associated with

    table

    the cuDF table that we are managing

    spillPriority

    a starting spill priority

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addReference(): Boolean

    Try to add a reference to this buffer to acquire it.

    Try to add a reference to this buffer to acquire it.

    returns

    true if the reference was added or false if this buffer is no longer valid

    Definition Classes
    RapidsBufferBase → RapidsBuffer
    Note

    The close method must be called for every successfully obtained reference.

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. def close(): Unit

    close() is called by client code to decrease the ref count of this RapidsBufferBase.

    close() is called by client code to decrease the ref count of this RapidsBufferBase. In the off chance that by the time close is invoked, the buffer was freed (not valid) then this close call winds up freeing the resources of the rapids buffer.

    Definition Classes
    RapidsBufferBase → AutoCloseable
  8. def columnarBatchFromDeviceBuffer(devBuffer: DeviceMemoryBuffer, sparkTypes: Array[DataType]): ColumnarBatch
    Attributes
    protected
    Definition Classes
    RapidsBufferBase
  9. def copyToMemoryBuffer(srcOffset: Long, dst: MemoryBuffer, dstOffset: Long, length: Long, stream: Stream): Unit

    Copy the content of this buffer into the specified memory buffer, starting from the given offset.

    Copy the content of this buffer into the specified memory buffer, starting from the given offset.

    srcOffset

    offset to start copying from.

    dst

    the memory buffer to copy into.

    dstOffset

    offset to copy into.

    length

    number of bytes to copy.

    stream

    CUDA stream to use

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  12. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. def free(): Unit

    Mark the buffer as freed and no longer valid.

    Mark the buffer as freed and no longer valid. This is called by the store when removing a buffer (it is no longer tracked).

    Definition Classes
    RapidsTable → RapidsBufferBase → RapidsBuffer
    Note

    The resources may not be immediately released if the buffer has outstanding references. In that case the resources will be released when the reference count reaches zero.

  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def getColumnarBatch(sparkTypes: Array[DataType]): ColumnarBatch

    Produce a ColumnarBatch from our table, and in the process make ourselves not spillable.

    Produce a ColumnarBatch from our table, and in the process make ourselves not spillable.

    sparkTypes

    the spark data types the batch should have

    Definition Classes
    RapidsTable → RapidsBufferBase → RapidsBuffer
  16. def getCopyIterator: RapidsBufferCopyIterator

    At spill time, obtain an iterator used to copy this buffer to a different tier.

    At spill time, obtain an iterator used to copy this buffer to a different tier.

    Definition Classes
    RapidsBuffer
  17. def getDeviceMemoryBuffer: DeviceMemoryBuffer

    TODO: we want to remove this method from the buffer, instead we want the catalog to be responsible for producing the DeviceMemoryBuffer by asking the buffer.

    TODO: we want to remove this method from the buffer, instead we want the catalog to be responsible for producing the DeviceMemoryBuffer by asking the buffer. This hides the RapidsBuffer from clients and simplifies locking.

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  18. def getHostColumnarBatch(sparkTypes: Array[DataType]): ColumnarBatch

    Get the host-backed columnar batch from this buffer.

    Get the host-backed columnar batch from this buffer. The caller must have successfully acquired the buffer beforehand.

    If this RapidsBuffer was added originally to the device tier, or if this is a just a buffer (not a batch), this function will throw.

    sparkTypes

    the spark data types the batch should have

    Definition Classes
    RapidsBuffer
    Note

    It is the responsibility of the caller to close the batch.

    See also

    addReference

  19. def getHostMemoryBuffer: HostMemoryBuffer

    Get the host memory buffer from the underlying storage.

    Get the host memory buffer from the underlying storage. If the buffer currently resides outside of host memory, a new HostMemoryBuffer is created with the data copied over. The caller must have successfully acquired the buffer beforehand.

    Definition Classes
    RapidsBufferBase → RapidsBuffer
    Note

    It is the responsibility of the caller to close the buffer.

    See also

    addReference

  20. def getMemoryBuffer: MemoryBuffer

    Get the underlying memory buffer.

    Get the underlying memory buffer. This may be either a HostMemoryBuffer or a DeviceMemoryBuffer depending on where the buffer currently resides. The caller must have successfully acquired the buffer beforehand.

    Definition Classes
    RapidsTable → RapidsBuffer
    Note

    It is the responsibility of the caller to close the buffer.

    See also

    addReference

  21. def getPackedSizeBytes: Long

    The size of this buffer if it has already gone through contiguous_split.

    The size of this buffer if it has already gone through contiguous_split.

    Definition Classes
    RapidsTable → RapidsBuffer
    Note

    Use this function when allocating a target buffer for spill or shuffle purposes.

  22. def getSpillPriority: Long

    Get the spill priority value for this buffer.

    Get the spill priority value for this buffer. Lower values are higher priority for spilling, meaning buffers with lower values will be preferred for spilling over buffers with a higher value.

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  23. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  24. val id: RapidsBufferId

    The buffer identifier for this buffer.

    The buffer identifier for this buffer.

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  25. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  26. var isValid: Boolean
    Attributes
    protected[this]
    Definition Classes
    RapidsBufferBase
  27. def makeChunkedPacker: ChunkedPacker

    Makes a new chunked packer.

    Makes a new chunked packer. It is the responsibility of the caller to close this.

    Definition Classes
    RapidsTable → RapidsBuffer
  28. def materializeMemoryBuffer: MemoryBuffer

    Materialize the memory buffer from the underlying storage.

    Materialize the memory buffer from the underlying storage.

    If the buffer resides in device or host memory, only reference count is incremented. If the buffer resides in secondary storage, a new host or device memory buffer is created, with the data copied to the new buffer. The caller must have successfully acquired the buffer beforehand.

    Attributes
    protected
    Definition Classes
    RapidsBufferBase
    Note

    It is the responsibility of the caller to close the buffer.

    ,

    This is an internal API only used by Rapids buffer stores.

    See also

    addReference

  29. val memoryUsedBytes: Long

    The size of this buffer in bytes in its _current_ store.

    The size of this buffer in bytes in its _current_ store. As the buffer goes through contiguous split (either added as a contiguous table already, or spilled to host), its size changes because contiguous_split adds its own alignment padding.

    Definition Classes
    RapidsTable → RapidsBuffer
    Note

    Do not use this size to allocate a target buffer to copy, always use getPackedSize.

  30. def meta: TableMeta

    Descriptor for how the memory buffer is formatted

    Descriptor for how the memory buffer is formatted

    Definition Classes
    RapidsTable → RapidsBufferBase → RapidsBuffer
  31. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  32. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  33. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  34. def onColumnSpillable(column: ColumnVector): Unit

    Mark a column as spillable

    Mark a column as spillable

    column

    the ColumnVector to mark as spillable

  35. var refcount: Int
    Attributes
    protected[this]
    Definition Classes
    RapidsBufferBase
  36. def releaseResources(): Unit

    Release the underlying resources for this buffer.

    Release the underlying resources for this buffer.

    Attributes
    protected
    Definition Classes
    RapidsTable → RapidsBufferBase
  37. def setSpillPriority(priority: Long): Unit

    Set the spill priority for this buffer.

    Set the spill priority for this buffer. Lower values are higher priority for spilling, meaning buffers with lower values will be preferred for spilling over buffers with a higher value.

    priority

    new priority value for this buffer

    Definition Classes
    RapidsBufferBase → RapidsBuffer
    Note

    should only be called from the buffer catalog

  38. val storageTier: StorageTier

    The storage tier for this buffer

    The storage tier for this buffer

    Definition Classes
    RapidsTable → RapidsBuffer
  39. val supportsChunkedPacker: Boolean
    Definition Classes
    RapidsTable → RapidsBuffer
  40. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  41. def toString(): String
    Definition Classes
    RapidsBufferBase → AnyRef → Any
  42. def updateSpillability(): Unit

    Update the spillability state of this RapidsTable.

    Update the spillability state of this RapidsTable. This is invoked from two places:

    - from the onColumnSpillable callback, which is invoked from a ColumnVector.EventHandler.onClosed callback.

    - after adding a table to the store to mark the table as spillable if all columns are spillable.

    Definition Classes
    RapidsTable → RapidsBuffer
  43. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  44. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  45. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  46. def withMemoryBufferReadLock[K](body: (MemoryBuffer) ⇒ K): K

    Obtains a read lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

    Obtains a read lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

    K

    any return type specified by body

    body

    function that takes a MemoryBuffer and produces K

    returns

    the result of body(memoryBuffer)

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  47. def withMemoryBufferWriteLock[K](body: (MemoryBuffer) ⇒ K): K

    Obtains a write lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

    Obtains a write lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

    K

    any return type specified by body

    body

    function that takes a MemoryBuffer and produces K

    returns

    the result of body(memoryBuffer)

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  48. def writeToChannel(outputChannel: WritableByteChannel, stream: Stream): Long

    At spill time, write this buffer to an nio WritableByteChannel.

    At spill time, write this buffer to an nio WritableByteChannel.

    stream

    the Cuda.Stream for the spilling thread. If the RapidsBuffer that implements this method is on the device, synchronization may be needed for staged copies.

    returns

    the amount of bytes written to the channel

    Definition Classes
    RapidsTable → RapidsBufferChannelWritable

Inherited from RapidsBufferChannelWritable

Inherited from RapidsDeviceMemoryStore.RapidsBufferBase

Inherited from RapidsBuffer

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped