Packages

o

org.apache.spark.storage

RapidsShuffleBlockFetcherIterator

object RapidsShuffleBlockFetcherIterator

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RapidsShuffleBlockFetcherIterator
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class FetchRequest(address: BlockManagerId, blocks: Seq[FetchBlockInfo]) extends Product with Serializable

    A request to fetch blocks from a remote BlockManager.

    A request to fetch blocks from a remote BlockManager.

    address

    remote BlockManager to fetch from.

    blocks

    Sequence of the information for blocks to fetch from the same address.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. def makeIterator(context: TaskContext, blockManager: BlockManager, sparkEnv: SparkEnv, blocksByAddress: Iterator[(BlockManagerId, Seq[(BlockId, Long, Int)])], serializerManager: SerializerManager, readMetrics: ShuffleReadMetricsReporter, fetchContinuousBlocksInBatch: Boolean): RapidsShuffleBlockFetcherIterator
  13. def mergeContinuousShuffleBlockIdsIfNeeded(blocks: Seq[FetchBlockInfo], doBatchFetch: Boolean): Seq[FetchBlockInfo]

    This function is used to merged blocks when doBatchFetch is true.

    This function is used to merged blocks when doBatchFetch is true. Blocks which have the same mapId can be merged into one block batch. The block batch is specified by a range of reduceId, which implies the continuous shuffle blocks that we can fetch in a batch. For example, input blocks like (shuffle_0_0_0, shuffle_0_0_1, shuffle_0_1_0) can be merged into (shuffle_0_0_0_2, shuffle_0_1_0_1), and input blocks like (shuffle_0_0_0_2, shuffle_0_0_2, shuffle_0_0_3) can be merged into (shuffle_0_0_0_4).

    blocks

    blocks to be merged if possible. May contains already merged blocks.

    doBatchFetch

    whether to merge blocks.

    returns

    the input blocks if doBatchFetch=false, or the merged blocks if doBatchFetch=true.

  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped