Packages

case class KeyGroupedPartitioning(expressions: Seq[Expression], numPartitions: Int, partitionValues: Seq[InternalRow] = Seq.empty) extends Partitioning with Product with Serializable

Represents a partitioning where rows are split across partitions based on transforms defined by expressions. partitionValuesOpt, if defined, should contain value of partition key(s) in ascending order, after evaluated by the transforms in expressions, for each input partition. In addition, its length must be the same as the number of input partitions (and thus is a 1-1 mapping). The partitionValues may contain duplicated partition values.

For example, if expressions is [years(ts_col)], then a valid value of partitionValuesOpt is [0, 1, 2], which represents 3 input partitions with distinct partition values. All rows in each partition have the same value for column ts_col (which is of timestamp type), after being applied by the years transform.

On the other hand, [0, 0, 1] is not a valid value for partitionValuesOpt since 0 is duplicated twice.

expressions

partition expressions for the partitioning.

numPartitions

the number of partitions

partitionValues

the values for the cluster keys of the distribution, must be in ascending order.

Linear Supertypes
Serializable, Serializable, Product, Equals, Partitioning, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KeyGroupedPartitioning
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. Partitioning
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new KeyGroupedPartitioning(expressions: Seq[Expression], numPartitions: Int, partitionValues: Seq[InternalRow] = Seq.empty)

    expressions

    partition expressions for the partitioning.

    numPartitions

    the number of partitions

    partitionValues

    the values for the cluster keys of the distribution, must be in ascending order.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def createShuffleSpec(distribution: ClusteredDistribution): ShuffleSpec

    Creates a shuffle spec for this partitioning and its required distribution.

    Creates a shuffle spec for this partitioning and its required distribution. The spec is used in the scenario where an operator has multiple children (e.g., join), and is used to decide whether this child is co-partitioned with others, therefore whether extra shuffle shall be introduced.

    distribution

    the required clustered distribution for this partitioning

    Definition Classes
    KeyGroupedPartitioning → Partitioning
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. val expressions: Seq[Expression]
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. val numPartitions: Int

    Returns the number of partitions that the data is split across

    Returns the number of partitions that the data is split across

    Definition Classes
    KeyGroupedPartitioning → Partitioning
  16. val partitionValues: Seq[InternalRow]
  17. final def satisfies(required: Distribution): Boolean

    Returns true iff the guarantees made by this Partitioning are sufficient to satisfy the partitioning scheme mandated by the required Distribution, i.e.

    Returns true iff the guarantees made by this Partitioning are sufficient to satisfy the partitioning scheme mandated by the required Distribution, i.e. the current dataset does not need to be re-partitioned for the required Distribution (it is possible that tuples within a partition need to be reorganized).

    A Partitioning can never satisfy a Distribution if its numPartitions doesn't match Distribution.requiredNumPartitions.

    Definition Classes
    Partitioning
  18. def satisfies0(required: Distribution): Boolean

    The actual method that defines whether this Partitioning can satisfy the given Distribution, after the numPartitions check.

    The actual method that defines whether this Partitioning can satisfy the given Distribution, after the numPartitions check.

    By default a Partitioning can satisfy UnspecifiedDistribution, and AllTuples if the Partitioning only have one partition. Implementations can also overwrite this method with special logic.

    Definition Classes
    KeyGroupedPartitioning → Partitioning
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. lazy val uniquePartitionValues: Seq[InternalRow]
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Partitioning

Inherited from AnyRef

Inherited from Any

Ungrouped