org.apache.spark.sql.execution.aggregate

TungstenAggregationIterator

class TungstenAggregationIterator extends Iterator[UnsafeRow] with Logging

An iterator used to evaluate aggregate functions. It operates on UnsafeRows.

This iterator first uses hash-based aggregation to process input rows. It uses a hash map to store groups and their corresponding aggregation buffers. If we this map cannot allocate memory from org.apache.spark.shuffle.ShuffleMemoryManager, it switches to sort-based aggregation. The process of the switch has the following step:

The code of this class is organized as follows:

Linear Supertypes
Logging, Iterator[UnsafeRow], TraversableOnce[UnsafeRow], GenTraversableOnce[UnsafeRow], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. TungstenAggregationIterator
  2. Logging
  3. Iterator
  4. TraversableOnce
  5. GenTraversableOnce
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new TungstenAggregationIterator(groupingExpressions: Seq[NamedExpression], nonCompleteAggregateExpressions: Seq[AggregateExpression2], completeAggregateExpressions: Seq[AggregateExpression2], initialInputBufferOffset: Int, resultExpressions: Seq[NamedExpression], newMutableProjection: (Seq[Expression], Seq[Attribute]) ⇒ () ⇒ MutableProjection, originalInputAttributes: Seq[Attribute], testFallbackStartsAt: Option[Int], numInputRows: LongSQLMetric, numOutputRows: LongSQLMetric)

    groupingExpressions

    expressions for grouping keys

    nonCompleteAggregateExpressions

    AggregateExpression2 containing AggregateFunction2s with mode Partial, PartialMerge, or Final.

    completeAggregateExpressions

    AggregateExpression2 containing AggregateFunction2s with mode Complete.

    initialInputBufferOffset

    If this iterator is used to handle functions with mode PartialMerge or Final. The input rows have the format of grouping keys + aggregation buffer. This offset indicates the starting position of aggregation buffer in a input row.

    resultExpressions

    expressions for generating output rows.

    newMutableProjection

    the function used to create mutable projections.

    originalInputAttributes

    attributes of representing input rows from inputIter.

Type Members

  1. class GroupedIterator[B >: A] extends AbstractIterator[Seq[B]] with Iterator[Seq[B]]

    Definition Classes
    Iterator

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. def ++[B >: UnsafeRow](that: ⇒ GenTraversableOnce[B]): Iterator[B]

    Definition Classes
    Iterator

  5. def /:[B](z: B)(op: (B, UnsafeRow) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  6. def :\[B](z: B)(op: (UnsafeRow, B) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  7. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  8. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  9. def addString(b: StringBuilder): StringBuilder

    Definition Classes
    TraversableOnce

  10. def addString(b: StringBuilder, sep: String): StringBuilder

    Definition Classes
    TraversableOnce

  11. def addString(b: StringBuilder, start: String, sep: String, end: String): StringBuilder

    Definition Classes
    TraversableOnce

  12. def aggregate[B](z: B)(seqop: (B, UnsafeRow) ⇒ B, combop: (B, B) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  13. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  14. def buffered: BufferedIterator[UnsafeRow]

    Definition Classes
    Iterator

  15. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  16. def collect[B](pf: PartialFunction[UnsafeRow, B]): Iterator[B]

    Definition Classes
    Iterator
    Annotations
    @migration
    Migration

    (Changed in version 2.8.0) collect has changed. The previous behavior can be reproduced with toSeq.

  17. def collectFirst[B](pf: PartialFunction[UnsafeRow, B]): Option[B]

    Definition Classes
    TraversableOnce

  18. def contains(elem: Any): Boolean

    Definition Classes
    Iterator

  19. def copyToArray[B >: UnsafeRow](xs: Array[B], start: Int, len: Int): Unit

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  20. def copyToArray[B >: UnsafeRow](xs: Array[B]): Unit

    Definition Classes
    TraversableOnce → GenTraversableOnce

  21. def copyToArray[B >: UnsafeRow](xs: Array[B], start: Int): Unit

    Definition Classes
    TraversableOnce → GenTraversableOnce

  22. def copyToBuffer[B >: UnsafeRow](dest: Buffer[B]): Unit

    Definition Classes
    TraversableOnce

  23. def corresponds[B](that: GenTraversableOnce[B])(p: (UnsafeRow, B) ⇒ Boolean): Boolean

    Definition Classes
    Iterator

  24. def count(p: (UnsafeRow) ⇒ Boolean): Int

    Definition Classes
    TraversableOnce → GenTraversableOnce

  25. def drop(n: Int): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  26. def dropWhile(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  27. def duplicate: (Iterator[UnsafeRow], Iterator[UnsafeRow])

    Definition Classes
    Iterator

  28. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  29. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  30. def exists(p: (UnsafeRow) ⇒ Boolean): Boolean

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  31. def filter(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  32. def filterNot(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  33. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  34. def find(p: (UnsafeRow) ⇒ Boolean): Option[UnsafeRow]

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  35. def flatMap[B](f: (UnsafeRow) ⇒ GenTraversableOnce[B]): Iterator[B]

    Definition Classes
    Iterator

  36. def fold[A1 >: UnsafeRow](z: A1)(op: (A1, A1) ⇒ A1): A1

    Definition Classes
    TraversableOnce → GenTraversableOnce

  37. def foldLeft[B](z: B)(op: (B, UnsafeRow) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  38. def foldRight[B](z: B)(op: (UnsafeRow, B) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  39. def forall(p: (UnsafeRow) ⇒ Boolean): Boolean

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  40. def foreach[U](f: (UnsafeRow) ⇒ U): Unit

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  41. def free(): Unit

    Free memory used in the underlying map.

  42. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  43. def grouped[B >: UnsafeRow](size: Int): GroupedIterator[B]

    Definition Classes
    Iterator

  44. def hasDefiniteSize: Boolean

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  45. final def hasNext: Boolean

    Definition Classes
    TungstenAggregationIterator → Iterator

  46. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  47. def indexOf[B >: UnsafeRow](elem: B): Int

    Definition Classes
    Iterator

  48. def indexWhere(p: (UnsafeRow) ⇒ Boolean): Int

    Definition Classes
    Iterator

  49. def isEmpty: Boolean

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  50. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  51. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  52. def isTraversableAgain: Boolean

    Definition Classes
    Iterator → GenTraversableOnce

  53. def length: Int

    Definition Classes
    Iterator

  54. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  55. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  56. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  57. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  58. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  59. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  60. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  61. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  62. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  63. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  64. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  65. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  66. def map[B](f: (UnsafeRow) ⇒ B): Iterator[B]

    Definition Classes
    Iterator

  67. def max[B >: UnsafeRow](implicit cmp: Ordering[B]): UnsafeRow

    Definition Classes
    TraversableOnce → GenTraversableOnce

  68. def maxBy[B](f: (UnsafeRow) ⇒ B)(implicit cmp: Ordering[B]): UnsafeRow

    Definition Classes
    TraversableOnce → GenTraversableOnce

  69. def min[B >: UnsafeRow](implicit cmp: Ordering[B]): UnsafeRow

    Definition Classes
    TraversableOnce → GenTraversableOnce

  70. def minBy[B](f: (UnsafeRow) ⇒ B)(implicit cmp: Ordering[B]): UnsafeRow

    Definition Classes
    TraversableOnce → GenTraversableOnce

  71. def mkString: String

    Definition Classes
    TraversableOnce → GenTraversableOnce

  72. def mkString(sep: String): String

    Definition Classes
    TraversableOnce → GenTraversableOnce

  73. def mkString(start: String, sep: String, end: String): String

    Definition Classes
    TraversableOnce → GenTraversableOnce

  74. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  75. final def next(): UnsafeRow

    Definition Classes
    TungstenAggregationIterator → Iterator

  76. def nonEmpty: Boolean

    Definition Classes
    TraversableOnce → GenTraversableOnce

  77. final def notify(): Unit

    Definition Classes
    AnyRef

  78. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  79. def outputForEmptyGroupingKeyWithoutInput(): UnsafeRow

    Generate a output row when there is no input and there is no grouping expression.

  80. def padTo[A1 >: UnsafeRow](len: Int, elem: A1): Iterator[A1]

    Definition Classes
    Iterator

  81. def partition(p: (UnsafeRow) ⇒ Boolean): (Iterator[UnsafeRow], Iterator[UnsafeRow])

    Definition Classes
    Iterator

  82. def patch[B >: UnsafeRow](from: Int, patchElems: Iterator[B], replaced: Int): Iterator[B]

    Definition Classes
    Iterator

  83. def product[B >: UnsafeRow](implicit num: Numeric[B]): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  84. def reduce[A1 >: UnsafeRow](op: (A1, A1) ⇒ A1): A1

    Definition Classes
    TraversableOnce → GenTraversableOnce

  85. def reduceLeft[B >: UnsafeRow](op: (B, UnsafeRow) ⇒ B): B

    Definition Classes
    TraversableOnce

  86. def reduceLeftOption[B >: UnsafeRow](op: (B, UnsafeRow) ⇒ B): Option[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  87. def reduceOption[A1 >: UnsafeRow](op: (A1, A1) ⇒ A1): Option[A1]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  88. def reduceRight[B >: UnsafeRow](op: (UnsafeRow, B) ⇒ B): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  89. def reduceRightOption[B >: UnsafeRow](op: (UnsafeRow, B) ⇒ B): Option[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  90. def reversed: List[UnsafeRow]

    Attributes
    protected[this]
    Definition Classes
    TraversableOnce

  91. def sameElements(that: Iterator[_]): Boolean

    Definition Classes
    Iterator

  92. def scanLeft[B](z: B)(op: (B, UnsafeRow) ⇒ B): Iterator[B]

    Definition Classes
    Iterator

  93. def scanRight[B](z: B)(op: (UnsafeRow, B) ⇒ B): Iterator[B]

    Definition Classes
    Iterator

  94. def seq: Iterator[UnsafeRow]

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  95. def size: Int

    Definition Classes
    TraversableOnce → GenTraversableOnce

  96. def slice(from: Int, until: Int): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  97. def sliding[B >: UnsafeRow](size: Int, step: Int): GroupedIterator[B]

    Definition Classes
    Iterator

  98. def span(p: (UnsafeRow) ⇒ Boolean): (Iterator[UnsafeRow], Iterator[UnsafeRow])

    Definition Classes
    Iterator

  99. def start(parentIter: Iterator[InternalRow]): Unit

    Start processing input rows.

    Start processing input rows. Only after this method is called will this iterator be non-empty.

  100. def sum[B >: UnsafeRow](implicit num: Numeric[B]): B

    Definition Classes
    TraversableOnce → GenTraversableOnce

  101. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  102. def take(n: Int): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  103. def takeWhile(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  104. def to[Col[_]](implicit cbf: CanBuildFrom[Nothing, UnsafeRow, Col[UnsafeRow]]): Col[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  105. def toArray[B >: UnsafeRow](implicit arg0: ClassTag[B]): Array[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  106. def toBuffer[B >: UnsafeRow]: Buffer[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  107. def toIndexedSeq: IndexedSeq[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  108. def toIterable: Iterable[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  109. def toIterator: Iterator[UnsafeRow]

    Definition Classes
    Iterator → GenTraversableOnce

  110. def toList: List[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  111. def toMap[T, U](implicit ev: <:<[UnsafeRow, (T, U)]): Map[T, U]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  112. def toSeq: Seq[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  113. def toSet[B >: UnsafeRow]: Set[B]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  114. def toStream: Stream[UnsafeRow]

    Definition Classes
    Iterator → GenTraversableOnce

  115. def toString(): String

    Definition Classes
    Iterator → AnyRef → Any

  116. def toTraversable: Traversable[UnsafeRow]

    Definition Classes
    Iterator → TraversableOnce → GenTraversableOnce

  117. def toVector: Vector[UnsafeRow]

    Definition Classes
    TraversableOnce → GenTraversableOnce

  118. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  119. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  120. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  121. def withFilter(p: (UnsafeRow) ⇒ Boolean): Iterator[UnsafeRow]

    Definition Classes
    Iterator

  122. def zip[B](that: Iterator[B]): Iterator[(UnsafeRow, B)]

    Definition Classes
    Iterator

  123. def zipAll[B, A1 >: UnsafeRow, B1 >: B](that: Iterator[B], thisElem: A1, thatElem: B1): Iterator[(A1, B1)]

    Definition Classes
    Iterator

  124. def zipWithIndex: Iterator[(UnsafeRow, Int)]

    Definition Classes
    Iterator

Deprecated Value Members

  1. def /:\[A1 >: UnsafeRow](z: A1)(op: (A1, A1) ⇒ A1): A1

    Definition Classes
    GenTraversableOnce
    Annotations
    @deprecated
    Deprecated

    (Since version 2.10.0) use fold instead

Inherited from Logging

Inherited from Iterator[UnsafeRow]

Inherited from TraversableOnce[UnsafeRow]

Inherited from GenTraversableOnce[UnsafeRow]

Inherited from AnyRef

Inherited from Any

Ungrouped