package util
- Alphabetic
- By Inheritance
- util
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- class ArrayBasedMapBuilder extends Serializable
A builder of ArrayBasedMapData, which fails if a null map key is detected, and removes duplicated map keys w.r.t.
A builder of ArrayBasedMapData, which fails if a null map key is detected, and removes duplicated map keys w.r.t. the last wins policy.
- class ArrayBasedMapData extends MapData
A simple
MapDataimplementation which is backed by 2 arrays.A simple
MapDataimplementation which is backed by 2 arrays.Note that, user is responsible to guarantee that the key array does not have duplicated elements, otherwise the behavior is undefined.
- abstract class ArrayData extends SpecializedGetters with Serializable
- class ArrayDataIndexedSeq[T] extends IndexedSeq[T]
Implements an
IndexedSeqinterface forArrayData.Implements an
IndexedSeqinterface forArrayData. Notice that if the originalArrayDatais a primitive array and contains null elements, it is better to ask forIndexedSeq[Any], instead ofIndexedSeq[Int], in order to keep the null elements. - case class BadRecordException(record: () => UTF8String, partialResults: () => Array[InternalRow] = () => Array.empty[InternalRow], cause: Throwable) extends Exception with Product with Serializable
Exception thrown when the underlying parser meet a bad record and can't parse it.
Exception thrown when the underlying parser meet a bad record and can't parse it.
- record
a function to return the record that cause the parser to fail
- partialResults
a function that returns an row array, which is the partial results of parsing this bad record.
- cause
the actual exception about why the record is bad and can't be parsed.
- case class CannotParseJSONFieldException(fieldName: String, fieldValue: String, jsonType: JsonToken, dataType: DataType) extends RuntimeException with Product with Serializable
No-stacktrace equivalent of
QueryExecutionErrors.cannotParseJSONFieldError.No-stacktrace equivalent of
QueryExecutionErrors.cannotParseJSONFieldError. Used for code control flow in the parser without overhead of creating a full exception. - class CharVarcharCodegenUtils extends AnyRef
- case class EmptyJsonFieldValueException(dataType: DataType) extends RuntimeException with Product with Serializable
No-stacktrace equivalent of
QueryExecutionErrors.emptyJsonFieldValueError.No-stacktrace equivalent of
QueryExecutionErrors.emptyJsonFieldValueError. Used for code control flow in the parser without overhead of creating a full exception. - class FailureSafeParser[IN] extends AnyRef
- class GenericArrayData extends ArrayData
- class HyperLogLogPlusPlusHelper extends Serializable
- class InternalRowComparableWrapper extends AnyRef
Wraps the InternalRow with the corresponding DataType to make it comparable with the values in InternalRow.
Wraps the InternalRow with the corresponding DataType to make it comparable with the values in InternalRow. It uses Spark's internal murmur hash to compute hash code from an row, and uses RowOrdering to perform equality checks.
- case class JsonArraysAsStructsException() extends RuntimeException with Product with Serializable
Exception thrown when the underlying parser parses a JSON array as a struct.
- abstract class MapData extends Serializable
This is an internal data representation for map type in Spark SQL.
This is an internal data representation for map type in Spark SQL. This should not implement
equalsandhashCodebecause the type cannot be used as join keys, grouping keys, or in equality tests. See SPARK-9415 and PR#13847 for the discussions. - implicit class MetadataColumnHelper extends AnyRef
- sealed trait ParseMode extends AnyRef
- case class PartialArrayDataResultException(partialResult: ArrayData, cause: Throwable) extends PartialValueException with Product with Serializable
Exception thrown when the underlying parser returns a partial array result.
Exception thrown when the underlying parser returns a partial array result.
- partialResult
the partial array result.
- cause
the actual exception about why the parser cannot return full result.
- case class PartialMapDataResultException(partialResult: MapData, cause: Throwable) extends PartialValueException with Product with Serializable
Exception thrown when the underlying parser returns a partial map result.
Exception thrown when the underlying parser returns a partial map result.
- partialResult
the partial map result.
- cause
the actual exception about why the parser cannot return full result.
- case class PartialResultArrayException(partialResults: Array[InternalRow], cause: Throwable) extends Exception with Product with Serializable
Exception thrown when the underlying parser returns partial result list of parsing.
Exception thrown when the underlying parser returns partial result list of parsing.
- partialResults
the partial result list of parsing bad records.
- cause
the actual exception about why the parser cannot return full result.
- case class PartialResultException(partialResult: InternalRow, cause: Throwable) extends PartialValueException with Product with Serializable
Exception thrown when the underlying parser returns a partial result of parsing an object/row.
Exception thrown when the underlying parser returns a partial result of parsing an object/row.
- partialResult
the partial result of parsing a bad record.
- cause
the actual exception about why the parser cannot return full result.
- abstract class PartialValueException extends Exception
- class QuantileSummaries extends Serializable
Helper class to compute approximate quantile summary.
Helper class to compute approximate quantile summary. This implementation is based on the algorithm proposed in the paper: "Space-efficient Online Computation of Quantile Summaries" by Greenwald, Michael and Khanna, Sanjeev. (https://doi.org/10.1145/375663.375670)
In order to optimize for speed, it maintains an internal buffer of the last seen samples, and only inserts them after crossing a certain size threshold. This guarantees a near-constant runtime complexity compared to the original algorithm.
- case class RandomIndicesGenerator(randomSeed: Long) extends Product with Serializable
This class is used to generate a random indices of given length.
This class is used to generate a random indices of given length.
This implementation uses the "inside-out" version of Fisher-Yates algorithm. Reference: https://en.wikipedia.org/wiki/Fisher%E2%80%93Yates_shuffle#The_%22inside-out%22_algorithm
- case class RandomUUIDGenerator(randomSeed: Long) extends Product with Serializable
This class is used to generate a UUID from Pseudo-Random Numbers.
This class is used to generate a UUID from Pseudo-Random Numbers.
For the algorithm, see RFC 4122: A Universally Unique IDentifier (UUID) URN Namespace, section 4.4 "Algorithms for Creating a UUID from Truly Random or Pseudo-Random Numbers".
- case class StringAsDataTypeException(fieldName: String, fieldValue: String, dataType: DataType) extends RuntimeException with Product with Serializable
Exception thrown when the underlying parser can not parses a String as a datatype.
- class StringKeyHashMap[T] extends AnyRef
- class ToNumberParser extends Serializable
This class represents a parser to implement the to_number or try_to_number SQL functions.
This class represents a parser to implement the to_number or try_to_number SQL functions.
It works by consuming an input string and a format string. This class accepts the format string as a field, and proceeds to iterate through the format string to generate a sequence of tokens (or throw an exception if the format string is invalid). Then when the function is called with an input string, this class steps through the sequence of tokens and compares them against the input string, returning a Spark Decimal object if they match (or throwing an exception otherwise).
- case class WriteDeltaProjections(rowProjection: Option[ProjectingInternalRow], rowIdProjection: ProjectingInternalRow, metadataProjection: Option[ProjectingInternalRow]) extends Product with Serializable
Value Members
- val AUTO_GENERATED_ALIAS: String
- val INTERNAL_METADATA_KEYS: Seq[String]
- val METADATA_COL_ATTR_KEY: String
- val QUALIFIED_ACCESS_ONLY: String
If set, this metadata column can only be accessed with qualifiers, e.g.
If set, this metadata column can only be accessed with qualifiers, e.g.
qualifiers.colorqualifiers.*. If not set, metadata columns cannot be accessed via star. - def escapeSingleQuotedString(str: String): String
- def fileToString(file: File, encoding: Charset = UTF_8): String
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def quietly[A](f: => A): A
Silences output to stderr or stdout for the duration of f
- def quoteIdentifier(name: String): String
- def quoteIfNeeded(part: String): String
- def quoteNameParts(name: Seq[String]): String
- def removeInternalMetadata(schema: StructType): StructType
- def resourceToBytes(resource: String, classLoader: ClassLoader = Utils.getSparkClassLoader): Array[Byte]
- def resourceToString(resource: String, encoding: String = UTF_8.name(), classLoader: ClassLoader = Utils.getSparkClassLoader): String
- def sideBySide(left: Seq[String], right: Seq[String]): Seq[String]
- def sideBySide(left: String, right: String): Seq[String]
- def stackTraceToString(t: Throwable): String
- def stringToFile(file: File, str: String): File
- def toPrettySQL(e: Expression): String
- def truncatedString[T](seq: Seq[T], sep: String, maxFields: Int): String
Shorthand for calling truncatedString() without start or end strings.
- def truncatedString[T](seq: Seq[T], start: String, sep: String, end: String, maxFields: Int): String
Format a sequence with semantics similar to calling .mkString().
Format a sequence with semantics similar to calling .mkString(). Any elements beyond maxNumToStringFields will be dropped and replaced by a "... N more fields" placeholder.
- returns
the trimmed and formatted string.
- def usePrettyExpression(e: Expression): Expression
- object ArrayBasedMapData extends Serializable
- object ArrayData extends Serializable
- object CharVarcharUtils extends Logging with SparkCharVarcharUtils
- object CompressionCodecs
- object DateTimeUtils extends SparkDateTimeUtils
Helper functions for converting between internal and external date and time representations.
Helper functions for converting between internal and external date and time representations. Dates are exposed externally as java.sql.Date and are represented internally as the number of dates since the Unix epoch (1970-01-01). Timestamps are exposed externally as java.sql.Timestamp and are stored internally as longs, which are capable of storing timestamps with microsecond precision.
- case object DropMalformedMode extends ParseMode with Product with Serializable
This mode ignores the whole corrupted records.
- case object FailFastMode extends ParseMode with Product with Serializable
This mode throws an exception when it meets corrupted records.
- object GeneratedColumn
This object contains utility methods and values for Generated Columns
- object GeneratedColumnAnalyzer extends Analyzer
Analyzer for processing generated column expressions using built-in functions only.
- object HyperLogLogPlusPlusHelper extends Serializable
Constants used in the implementation of the HyperLogLogPlusPlus aggregate function.
Constants used in the implementation of the HyperLogLogPlusPlus aggregate function.
See the Appendix to HyperLogLog in Practice: Algorithmic Engineering of a State of the Art Cardinality (https://docs.google.com/document/d/1gyjfMHy43U9OWBXxfaeG-3MjGzejW1dlpyMwEYAAWEI/view?fullscreen) for more information.
- object InternalRowComparableWrapper
- object IntervalMathUtils
Helper functions for interval arithmetic operations with overflow.
- object IntervalUtils extends SparkIntervalUtils
- object NumberConverter
- object ParseMode extends Logging
- case object PermissiveMode extends ParseMode with Product with Serializable
This mode permissively parses the records.
- object QuantileSummaries extends Serializable
- object ResolveDefaultColumns extends QueryErrorsBase with ResolveDefaultColumnsUtils
This object contains fields to help process DEFAULT columns.
- object RowDeltaUtils
A utility that holds constants for handling deltas of rows.
- object SQLOrderingUtil
- object StringKeyHashMap
Build a map with String type of key, and it also supports either key case sensitive or insensitive.
- object StringUtils extends Logging
- object ToNumberParser extends Serializable
- object TypeUtils extends QueryErrorsBase
Functions to help with checking for valid data types and value comparison of various types.
- object UTF8StringUtils
Helper functions for casting string to numeric values.
- object UnsafeRowUtils