Packages

o

org.apache.spark.storage

RapidsShuffleBlockFetcherIterator

object RapidsShuffleBlockFetcherIterator

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RapidsShuffleBlockFetcherIterator
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class FetchRequest(address: BlockManagerId, blocks: Seq[FetchBlockInfo], forMergedMetas: Boolean = false) extends Product with Serializable

    A request to fetch blocks from a remote BlockManager.

    A request to fetch blocks from a remote BlockManager.

    address

    remote BlockManager to fetch from.

    blocks

    Sequence of the information for blocks to fetch from the same address.

    forMergedMetas

    true if this request is for requesting push-merged meta information; false if it is for regular or shuffle chunks.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. val isNettyOOMOnShuffle: AtomicBoolean

    A flag which indicates whether the Netty OOM error has raised during shuffle.

    A flag which indicates whether the Netty OOM error has raised during shuffle. If true, unless there's no in-flight fetch requests, all the pending shuffle fetch requests will be deferred until the flag is unset (whenever there's a complete fetch request).

  13. def makeIterator(context: TaskContext, blockManager: BlockManager, sparkEnv: SparkEnv, blocksByAddress: Iterator[(BlockManagerId, Seq[(BlockId, Long, Int)])], serializerManager: SerializerManager, readMetrics: ShuffleReadMetricsReporter, fetchContinuousBlocksInBatch: Boolean): RapidsShuffleBlockFetcherIterator
  14. def mergeContinuousShuffleBlockIdsIfNeeded(blocks: Seq[FetchBlockInfo], doBatchFetch: Boolean): Seq[FetchBlockInfo]

    This function is used to merged blocks when doBatchFetch is true.

    This function is used to merged blocks when doBatchFetch is true. Blocks which have the same mapId can be merged into one block batch. The block batch is specified by a range of reduceId, which implies the continuous shuffle blocks that we can fetch in a batch. For example, input blocks like (shuffle_0_0_0, shuffle_0_0_1, shuffle_0_1_0) can be merged into (shuffle_0_0_0_2, shuffle_0_1_0_1), and input blocks like (shuffle_0_0_0_2, shuffle_0_0_2, shuffle_0_0_3) can be merged into (shuffle_0_0_0_4).

    blocks

    blocks to be merged if possible. May contains already merged blocks.

    doBatchFetch

    whether to merge blocks.

    returns

    the input blocks if doBatchFetch=false, or the merged blocks if doBatchFetch=true.

  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. def resetNettyOOMFlagIfPossible(freeMemoryLowerBound: Long): Unit
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped