Packages

case class ExpressionEncoder[T](objSerializer: Expression, objDeserializer: Expression, clsTag: ClassTag[T]) extends Encoder[T] with Product with Serializable

A generic encoder for JVM objects that uses Catalyst Expressions for a serializer and a deserializer.

objSerializer

An expression that can be used to encode a raw object to corresponding Spark SQL representation that can be a primitive column, array, map or a struct. This represents how Spark SQL generally serializes an object of type T.

objDeserializer

An expression that will construct an object given a Spark SQL representation. This represents how Spark SQL generally deserializes a serialized value in Spark SQL representation back to an object of type T.

clsTag

A classtag for T.

Linear Supertypes
Product, Equals, Encoder[T], Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ExpressionEncoder
  2. Product
  3. Equals
  4. Encoder
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new ExpressionEncoder(objSerializer: Expression, objDeserializer: Expression, clsTag: ClassTag[T])

    objSerializer

    An expression that can be used to encode a raw object to corresponding Spark SQL representation that can be a primitive column, array, map or a struct. This represents how Spark SQL generally serializes an object of type T.

    objDeserializer

    An expression that will construct an object given a Spark SQL representation. This represents how Spark SQL generally deserializes a serialized value in Spark SQL representation back to an object of type T.

    clsTag

    A classtag for T.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def assertUnresolved(): Unit

    The process of resolution to a given schema throws away information about where a given field is being bound by ordinal instead of by name.

    The process of resolution to a given schema throws away information about where a given field is being bound by ordinal instead of by name. This method checks to make sure this process has not been done already in places where we plan to do later composition of encoders.

  6. val attrs: Seq[String]
    Attributes
    protected
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  8. val clsTag: ClassTag[T]
    Definition Classes
    ExpressionEncoder → Encoder
  9. def createDeserializer(): Deserializer[T]

    Create a deserializer that can convert a Spark SQL Row into an object of type T.

    Create a deserializer that can convert a Spark SQL Row into an object of type T.

    Note that you must resolveAndBind an encoder to a specific schema before you can create a deserializer.

  10. def createSerializer(): Serializer[T]

    Create a serializer that can convert an object of type T to a Spark SQL Row.

    Create a serializer that can convert an object of type T to a Spark SQL Row.

    Note that the returned Serializer is not thread safe. Multiple calls to serializer.apply(..) are allowed to return the same actual InternalRow object. Thus, the caller should copy the result before making another call if required.

  11. val deserializer: Expression

    Returns an expression that can be used to deserialize an input row to an object of type T with a compatible schema.

    Returns an expression that can be used to deserialize an input row to an object of type T with a compatible schema. Fields of the row will be extracted using UnresolvedAttribute. of the same name as the constructor arguments.

    For complex objects that are encoded to structs, Fields of the struct will be extracted using GetColumnByOrdinal with corresponding ordinal.

  12. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  14. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. def isSerializedAsStruct: Boolean

    Returns true if the type T is serialized as a struct by objSerializer.

  17. def isSerializedAsStructForTopLevel: Boolean

    If the type T is serialized as a struct, when it is encoded to a Spark SQL row, fields in the struct are naturally mapped to top-level columns in a row.

    If the type T is serialized as a struct, when it is encoded to a Spark SQL row, fields in the struct are naturally mapped to top-level columns in a row. In other words, the serialized struct is flattened to row. But in case of the T is also an Option type, it can't be flattened to top-level row, because in Spark SQL top-level row can't be null. This method returns true if T is serialized as struct and is not Option type.

  18. def namedExpressions: Seq[NamedExpression]

    Returns a new set (with unique ids) of NamedExpression that represent the serialized form of this object.

  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. val objDeserializer: Expression
  23. val objSerializer: Expression
  24. def productElementNames: Iterator[String]
    Definition Classes
    Product
  25. def resolveAndBind(attrs: Seq[Attribute] = DataTypeUtils.toAttributes(schema), analyzer: Analyzer = SimpleAnalyzer): ExpressionEncoder[T]

    Returns a new copy of this encoder, where the deserializer is resolved and bound to the given schema.

    Returns a new copy of this encoder, where the deserializer is resolved and bound to the given schema.

    Note that, ideally encoder is used as a container of serde expressions, the resolution and binding stuff should happen inside query framework. However, in some cases we need to use encoder as a function to do serialization directly(e.g. Dataset.collect), then we can use this method to do resolution and binding outside of query framework.

  26. val schema: StructType
    Definition Classes
    ExpressionEncoder → Encoder
  27. val schemaString: String
    Attributes
    protected
  28. val serializer: Seq[NamedExpression]

    A sequence of expressions, one for each top-level field that can be used to extract the values from a raw object into an InternalRow: 1.

    A sequence of expressions, one for each top-level field that can be used to extract the values from a raw object into an InternalRow: 1. If serializer encodes a raw object to a struct, strip the outer If-IsNull and get the CreateNamedStruct. 2. For other cases, wrap the single serializer with CreateNamedStruct.

  29. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  30. def toString(): String
    Definition Classes
    ExpressionEncoder → AnyRef → Any
  31. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  32. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  33. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from Product

Inherited from Equals

Inherited from Encoder[T]

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped