trait ShuffleSpec extends AnyRef
This is used in the scenario where an operator has multiple children (e.g., join) and one or more of which have their own requirement regarding whether its data can be considered as co-partitioned from others. This offers APIs for:
- Comparing with specs from other children of the operator and check if they are compatible. When two specs are compatible, we can say their data are co-partitioned, and Spark will potentially be able to eliminate shuffle if necessary.
- Creating a partitioning that can be used to re-partition another child, so that to make it having a compatible partitioning as this node.
- Alphabetic
- By Inheritance
- ShuffleSpec
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def canCreatePartitioning: Boolean
Whether this shuffle spec can be used to create partitionings for the other children.
- abstract def isCompatibleWith(other: ShuffleSpec): Boolean
Returns true iff this spec is compatible with the provided shuffle spec.
Returns true iff this spec is compatible with the provided shuffle spec.
A true return value means that the data partitioning from this spec can be seen as co-partitioned with the
other, and therefore no shuffle is required when joining the two sides.Note that Spark assumes this to be reflexive, symmetric and transitive.
- abstract def numPartitions: Int
Returns the number of partitions of this shuffle spec
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def createPartitioning(clustering: Seq[Expression]): Partitioning
Creates a partitioning that can be used to re-partition the other side with the given clustering expressions.
Creates a partitioning that can be used to re-partition the other side with the given clustering expressions.
This will only be called when:
- isCompatibleWith returns false on the side where the
clusteringis from.
- isCompatibleWith returns false on the side where the
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()