Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##: Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def canCreatePartitioning: Boolean

   Whether this shuffle spec can be used to create partitionings for the other children.

   Whether this shuffle spec can be used to create partitionings for the other children.

   Definition Classes

   HashShuffleSpec → ShuffleSpec

6.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws(classOf[java.lang.CloneNotSupportedException]) @native()

7.  def createPartitioning(clustering: Seq[Expression]): Partitioning

   Creates a partitioning that can be used to re-partition the other side with the given clustering expressions.

   Creates a partitioning that can be used to re-partition the other side with the given clustering expressions.

   This will only be called when:

   • isCompatibleWith returns false on the side where the clustering is from.

   Definition Classes

   HashShuffleSpec → ShuffleSpec

8.  val distribution: ClusteredDistribution
9.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

10.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.Throwable])

11.  final def getClass(): Class[_ <: AnyRef]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

12.  lazy val hashKeyPositions: Seq[BitSet]

    A sequence where each element is a set of positions of the hash partition key to the cluster keys.

    A sequence where each element is a set of positions of the hash partition key to the cluster keys. For instance, if cluster keys are [a, b, b] and hash partition keys are [a, b], the result will be [(0), (1, 2)].

    This is useful to check compatibility between two HashShuffleSpecs. If the cluster keys are [a, b, b] and [x, y, z] for the two join children, and the hash partition keys are [a, b] and [x, z], they are compatible. With the positions, we can do the compatibility check by looking at if the positions of hash partition keys from two sides have overlapping.

13.  def isCompatibleWith(other: ShuffleSpec): Boolean

    Returns true iff this spec is compatible with the provided shuffle spec.

    Returns true iff this spec is compatible with the provided shuffle spec.

    A true return value means that the data partitioning from this spec can be seen as co-partitioned with the other, and therefore no shuffle is required when joining the two sides.

    Note that Spark assumes this to be reflexive, symmetric and transitive.

    Definition Classes

    HashShuffleSpec → ShuffleSpec

14.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

15.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

16.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

17.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

18.  def numPartitions: Int

    Returns the number of partitions of this shuffle spec

    Returns the number of partitions of this shuffle spec

    Definition Classes

    HashShuffleSpec → ShuffleSpec

19.  val partitioning: HashPartitioning
20.  def productElementNames: Iterator[String]

    Definition Classes

    Product

21.  final def synchronized[T0](arg0: => T0): T0

    Definition Classes

    AnyRef

22.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

23.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

24.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException]) @native()