trait Partitioning extends AnyRef
Describes how an operator's output is split across partitions. It has 2 major properties:
- number of partitions. 2. if it can satisfy a given distribution.
- Alphabetic
- By Inheritance
- Partitioning
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract val numPartitions: Int
Returns the number of partitions that the data is split across
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def createShuffleSpec(distribution: ClusteredDistribution): ShuffleSpec
Creates a shuffle spec for this partitioning and its required distribution.
Creates a shuffle spec for this partitioning and its required distribution. The spec is used in the scenario where an operator has multiple children (e.g., join), and is used to decide whether this child is co-partitioned with others, therefore whether extra shuffle shall be introduced.
- distribution
the required clustered distribution for this partitioning
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def satisfies(required: Distribution): Boolean
Returns true iff the guarantees made by this Partitioning are sufficient to satisfy the partitioning scheme mandated by the
requiredDistribution, i.e.Returns true iff the guarantees made by this Partitioning are sufficient to satisfy the partitioning scheme mandated by the
requiredDistribution, i.e. the current dataset does not need to be re-partitioned for therequiredDistribution (it is possible that tuples within a partition need to be reorganized).A Partitioning can never satisfy a Distribution if its
numPartitionsdoesn't match Distribution.requiredNumPartitions. - def satisfies0(required: Distribution): Boolean
The actual method that defines whether this Partitioning can satisfy the given Distribution, after the
numPartitionscheck.The actual method that defines whether this Partitioning can satisfy the given Distribution, after the
numPartitionscheck.By default a Partitioning can satisfy UnspecifiedDistribution, and AllTuples if the Partitioning only have one partition. Implementations can also overwrite this method with special logic.
- Attributes
- protected
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()