Type Members

1.  class ArrayBasedMapBuilder extends Serializable

   A builder of ArrayBasedMapData, which fails if a null map key is detected, and removes duplicated map keys w.r.t.

   A builder of ArrayBasedMapData, which fails if a null map key is detected, and removes duplicated map keys w.r.t. the last wins policy.

2.  class ArrayBasedMapData extends MapData

   A simple MapData implementation which is backed by 2 arrays.

   A simple MapData implementation which is backed by 2 arrays.

   Note that, user is responsible to guarantee that the key array does not have duplicated elements, otherwise the behavior is undefined.

3.  abstract class ArrayData extends SpecializedGetters with Serializable
4.  class ArrayDataIndexedSeq[T] extends IndexedSeq[T]

   Implements an IndexedSeq interface for ArrayData.

   Implements an IndexedSeq interface for ArrayData. Notice that if the original ArrayData is a primitive array and contains null elements, it is better to ask for IndexedSeq[Any], instead of IndexedSeq[Int], in order to keep the null elements.

5.  case class BadRecordException(record: () ⇒ UTF8String, partialResults: () ⇒ Array[InternalRow] = () => Array.empty[InternalRow], cause: Throwable) extends Exception with Product with Serializable

   Exception thrown when the underlying parser meet a bad record and can't parse it.

   Exception thrown when the underlying parser meet a bad record and can't parse it.

   record

   a function to return the record that cause the parser to fail

   partialResults

   a function that returns an row array, which is the partial results of parsing this bad record.

   cause

   the actual exception about why the record is bad and can't be parsed.

6.  case class CannotParseJSONFieldException(fieldName: String, fieldValue: String, jsonType: JsonToken, dataType: DataType) extends RuntimeException with Product with Serializable

   No-stacktrace equivalent of QueryExecutionErrors.cannotParseJSONFieldError.

   No-stacktrace equivalent of QueryExecutionErrors.cannotParseJSONFieldError. Used for code control flow in the parser without overhead of creating a full exception.

7.  class CaseInsensitiveMap[T] extends Map[String, T] with Serializable

   Builds a map in which keys are case insensitive.

   Builds a map in which keys are case insensitive. Input map can be accessed for cases where case-sensitive information is required. The primary constructor is marked private to avoid nested case-insensitive map creation, otherwise the keys in the original map will become case-insensitive in this scenario. Note: CaseInsensitiveMap is serializable. However, after transformation, e.g. filterKeys(), it may become not serializable.

8.  class CharVarcharCodegenUtils extends AnyRef
9.  sealed trait DateFormatter extends Serializable
10.  trait DateTimeFormatterHelper extends AnyRef
11.  class DefaultDateFormatter extends Iso8601DateFormatter

    The formatter for dates which doesn't require users to specify a pattern.

    The formatter for dates which doesn't require users to specify a pattern. While formatting, it uses the default pattern DateFormatter.defaultPattern. In parsing, it follows the CAST logic in conversion of strings to Catalyst's DateType.

12.  class DefaultTimestampFormatter extends Iso8601TimestampFormatter

    The formatter for timestamps which doesn't require users to specify a pattern.

    The formatter for timestamps which doesn't require users to specify a pattern. While formatting, it uses the default pattern TimestampFormatter.defaultPattern(). In parsing, it follows the CAST logic in conversion of strings to Catalyst's TimestampType.

13.  case class EmptyJsonFieldValueException(dataType: DataType) extends RuntimeException with Product with Serializable

    No-stacktrace equivalent of QueryExecutionErrors.emptyJsonFieldValueError.

    No-stacktrace equivalent of QueryExecutionErrors.emptyJsonFieldValueError. Used for code control flow in the parser without overhead of creating a full exception.

14.  class FailureSafeParser[IN] extends AnyRef
15.  class FractionTimestampFormatter extends Iso8601TimestampFormatter

    The formatter parses/formats timestamps according to the pattern yyyy-MM-dd HH:mm:ss.[..fff..] where [..fff..] is a fraction of second up to microsecond resolution.

    The formatter parses/formats timestamps according to the pattern yyyy-MM-dd HH:mm:ss.[..fff..] where [..fff..] is a fraction of second up to microsecond resolution. The formatter does not output trailing zeros in the fraction. For example, the timestamp 2019-03-05 15:00:01.123400 is formatted as the string 2019-03-05 15:00:01.1234.

16.  class GenericArrayData extends ArrayData
17.  class HyperLogLogPlusPlusHelper extends Serializable
18.  class InternalRowComparableWrapper extends AnyRef

    Wraps the InternalRow with the corresponding DataType to make it comparable with the values in InternalRow.

    Wraps the InternalRow with the corresponding DataType to make it comparable with the values in InternalRow. It uses Spark's internal murmur hash to compute hash code from an row, and uses RowOrdering to perform equality checks.

19.  class Iso8601DateFormatter extends DateFormatter with DateTimeFormatterHelper
20.  class Iso8601TimestampFormatter extends TimestampFormatter with DateTimeFormatterHelper
21.  case class JsonArraysAsStructsException() extends RuntimeException with Product with Serializable

    Exception thrown when the underlying parser parses a JSON array as a struct.

22.  trait LegacyDateFormatter extends DateFormatter
23.  class LegacyFastDateFormatter extends LegacyDateFormatter

    The legacy formatter is based on Apache Commons FastDateFormat.

    The legacy formatter is based on Apache Commons FastDateFormat. The formatter uses the default JVM time zone intentionally for compatibility with Spark 2.4 and earlier versions.

    Note: Using of the default JVM time zone makes the formatter compatible with the legacy DateTimeUtils methods toJavaDate and fromJavaDate that are based on the default JVM time zone too.

24.  class LegacyFastTimestampFormatter extends TimestampFormatter
25.  class LegacySimpleDateFormatter extends LegacyDateFormatter

    The legacy formatter is based on java.text.SimpleDateFormat.

    The legacy formatter is based on java.text.SimpleDateFormat. The formatter uses the default JVM time zone intentionally for compatibility with Spark 2.4 and earlier versions.

    Note: Using of the default JVM time zone makes the formatter compatible with the legacy DateTimeUtils methods toJavaDate and fromJavaDate that are based on the default JVM time zone too.

26.  class LegacySimpleTimestampFormatter extends TimestampFormatter
27.  abstract class MapData extends Serializable

    This is an internal data representation for map type in Spark SQL.

    This is an internal data representation for map type in Spark SQL. This should not implement equals and hashCode because the type cannot be used as join keys, grouping keys, or in equality tests. See SPARK-9415 and PR#13847 for the discussions.

28.  implicit class MetadataColumnHelper extends AnyRef
29.  class MicrosCalendar extends GregorianCalendar

    The custom sub-class of GregorianCalendar is needed to get access to protected fields immediately after parsing.

    The custom sub-class of GregorianCalendar is needed to get access to protected fields immediately after parsing. We cannot use the get() method because it performs normalization of the fraction part. Accordingly, the MILLISECOND field doesn't contain original value.

    Also this class allows to set raw value to the MILLISECOND field directly before formatting.

30.  sealed trait ParseMode extends AnyRef
31.  case class PartialArrayDataResultException(partialResult: ArrayData, cause: Throwable) extends PartialValueException with Product with Serializable

    Exception thrown when the underlying parser returns a partial array result.

    Exception thrown when the underlying parser returns a partial array result.

    partialResult

    the partial array result.

    cause

    the actual exception about why the parser cannot return full result.

32.  case class PartialMapDataResultException(partialResult: MapData, cause: Throwable) extends PartialValueException with Product with Serializable

    Exception thrown when the underlying parser returns a partial map result.

    Exception thrown when the underlying parser returns a partial map result.

    partialResult

    the partial map result.

    cause

    the actual exception about why the parser cannot return full result.

33.  case class PartialResultArrayException(partialResults: Array[InternalRow], cause: Throwable) extends Exception with Product with Serializable

    Exception thrown when the underlying parser returns partial result list of parsing.

    Exception thrown when the underlying parser returns partial result list of parsing.

    partialResults

    the partial result list of parsing bad records.

    cause

    the actual exception about why the parser cannot return full result.

34.  case class PartialResultException(partialResult: InternalRow, cause: Throwable) extends PartialValueException with Product with Serializable

    Exception thrown when the underlying parser returns a partial result of parsing an object/row.

    Exception thrown when the underlying parser returns a partial result of parsing an object/row.

    partialResult

    the partial result of parsing a bad record.

    cause

    the actual exception about why the parser cannot return full result.

35.  abstract class PartialValueException extends Exception
36.  class QuantileSummaries extends Serializable

    Helper class to compute approximate quantile summary.

    Helper class to compute approximate quantile summary. This implementation is based on the algorithm proposed in the paper: "Space-efficient Online Computation of Quantile Summaries" by Greenwald, Michael and Khanna, Sanjeev. (https://doi.org/10.1145/375663.375670)

    In order to optimize for speed, it maintains an internal buffer of the last seen samples, and only inserts them after crossing a certain size threshold. This guarantees a near-constant runtime complexity compared to the original algorithm.

37.  case class RandomIndicesGenerator(randomSeed: Long) extends Product with Serializable

    This class is used to generate a random indices of given length.

    This class is used to generate a random indices of given length.

    This implementation uses the "inside-out" version of Fisher-Yates algorithm. Reference: https://en.wikipedia.org/wiki/Fisher%E2%80%93Yates_shuffle#The_%22inside-out%22_algorithm

38.  case class RandomUUIDGenerator(randomSeed: Long) extends Product with Serializable

    This class is used to generate a UUID from Pseudo-Random Numbers.

    This class is used to generate a UUID from Pseudo-Random Numbers.

    For the algorithm, see RFC 4122: A Universally Unique IDentifier (UUID) URN Namespace, section 4.4 "Algorithms for Creating a UUID from Truly Random or Pseudo-Random Numbers".

39.  case class StringAsDataTypeException(fieldName: String, fieldValue: String, dataType: DataType) extends RuntimeException with Product with Serializable

    Exception thrown when the underlying parser can not parses a String as a datatype.

40.  class StringKeyHashMap[T] extends AnyRef
41.  sealed trait TimestampFormatter extends Serializable
42.  class ToNumberParser extends Serializable

    This class represents a parser to implement the to_number or try_to_number SQL functions.

    This class represents a parser to implement the to_number or try_to_number SQL functions.

    It works by consuming an input string and a format string. This class accepts the format string as a field, and proceeds to iterate through the format string to generate a sequence of tokens (or throw an exception if the format string is invalid). Then when the function is called with an input string, this class steps through the sequence of tokens and compares them against the input string, returning a Spark Decimal object if they match (or throwing an exception otherwise).

43.  case class WriteDeltaProjections(rowProjection: Option[ProjectingInternalRow], rowIdProjection: ProjectingInternalRow, metadataProjection: Option[ProjectingInternalRow]) extends Product with Serializable