trait RapidsShuffleTransport extends AutoCloseable

This defines what a "transport" should support. The intention is to allow for various transport implementations to exist, for different communication frameworks.

It is an AutoCloseable and so the caller should close when the transport is no longer needed.

Linear Supertypes
AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RapidsShuffleTransport
  2. AutoCloseable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def cancelPending(handler: RapidsShuffleFetchHandler): Unit

    Cancel requests that are waiting in the queue (not in-flight) for a specific handler (if any), and unregister the handler.

  2. abstract def close(): Unit
    Definition Classes
    AutoCloseable
    Annotations
    @throws( classOf[java.lang.Exception] )
  3. abstract def connect(peerBlockManagerId: BlockManagerId): ClientConnection

    Connect to peer given a BlockManagerId

    Connect to peer given a BlockManagerId

    This function is used during transport early start, if enabled.

    peerBlockManagerId

    the peer's block manager id

    returns

    a client connection for the peer

  4. abstract def doneBytesInFlight(bytesCompleted: Long): Unit

    (throttle) Signals that bytesCompleted are done, allowing more requests through the throttle.

    (throttle) Signals that bytesCompleted are done, allowing more requests through the throttle.

    bytesCompleted

    amount of bytes handled

  5. abstract def getDirectByteBuffer(size: Long): RefCountedDirectByteBuffer

    Returns a wrapped buffer of size Long.

    Returns a wrapped buffer of size Long. The buffer may or may not be pooled.

    The caller should call .close() on the returned RefCountedDirectByteBuffer when done.

    size

    size of buffer required

    returns

    the ref counted buffer

  6. abstract def makeClient(blockManagerId: BlockManagerId): RapidsShuffleClient

    This function will connect (if not connected already) to a peer described by blockManagerId.

    This function will connect (if not connected already) to a peer described by blockManagerId. Connections are cached.

    blockManagerId

    the peer's block manager id

    returns

    RapidsShuffleClient instance that can be used to interact with the peer

  7. abstract def makeServer(requestHandler: RapidsShuffleRequestHandler): RapidsShuffleServer

    This function should only be needed once.

    This function should only be needed once. The caller creates *a* server and it is used for the duration of the process.

    requestHandler

    used to get metadata info, and acquire tables used in the shuffle.

    returns

    the server instance

  8. abstract def queuePending(reqs: Seq[PendingTransferRequest]): Unit

    (throttle) Adds a set of requests to be throttled as limits allowed.

    (throttle) Adds a set of requests to be throttled as limits allowed.

    reqs

    requests to add to the throttle queue

  9. abstract def tryGetReceiveBounceBuffers(remaining: Long, totalRequired: Int): Seq[BounceBuffer]

    Get receive bounce buffers needed for a receive, limited by the amount of bytes to be received, and a hard limit on the number of buffers set by the caller using totalRequired.

    Get receive bounce buffers needed for a receive, limited by the amount of bytes to be received, and a hard limit on the number of buffers set by the caller using totalRequired.

    This function is non blocking. If it can't satisfy the bounce buffer request, an empty sequence is returned.

    remaining

    amount of bytes remaining in the receive

    totalRequired

    maximum no. of buffers that should be returned

    returns

    a sequence of bounce buffers, or empty if the request can't be satisfied

  10. abstract def tryGetSendBounceBuffers(remaining: Long, totalRequired: Int): Seq[SendBounceBuffers]

    Get send bounce buffers needed for a receive, limited by the amount of bytes to be sent, and a hard limit on the number of buffers set by the caller using totalRequired.

    Get send bounce buffers needed for a receive, limited by the amount of bytes to be sent, and a hard limit on the number of buffers set by the caller using totalRequired.

    This function is non blocking. If it can't satisfy the bounce buffer request, an empty sequence is returned.

    remaining

    amount of bytes remaining in the receive

    totalRequired

    maximum no. of buffers that should be returned

    returns

    a sequence of send bounce buffers, or empty if the request can't be satisfied

    Note

    the send bounce buffer object most likely includes both a device buffer and a host memory buffer, since sends can come from the device or the host.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  16. def toString(): String
    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped