types

Type Members

1. case class ArrayType(elementType: DataType, containsNull: Boolean) extends DataType with Product with Serializable

   

   The data type for collections of multiple values.

   The data type for collections of multiple values. Internally these are represented as columns that contain a scala.collection.Seq.

   elementType

   The data type of values.

   containsNull

   Indicates if values have null values

2. abstract class DataType extends AnyRef

   

3. case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType with Product with Serializable

   

   A Decimal that might have fixed precision and scale, or unlimited values for these

4. abstract class FractionalType extends NumericType

   

5. abstract class IntegralType extends NumericType

   

6. case class MapType(keyType: DataType, valueType: DataType, valueContainsNull: Boolean) extends DataType with Product with Serializable

   

   The data type for Maps.

   The data type for Maps. Keys in a map are not allowed to have null values.

   keyType

   The data type of map keys.

   valueType

   The data type of map values.

   valueContainsNull

   Indicates if map values have null values.

7. abstract class NativeType extends DataType

   

8. abstract class NumericType extends NativeType with PrimitiveType

   

9. case class PrecisionInfo(precision: Int, scale: Int) extends Product with Serializable

   

   Precision parameters for a Decimal

10. trait PrimitiveType extends DataType

    

11. case class StructField(name: String, dataType: DataType, nullable: Boolean = true, metadata: Metadata = Metadata.empty) extends Product with Serializable

    

    A field inside a StructType.

    A field inside a StructType.

    name

    The name of this field.

    dataType

    The data type of this field.

    nullable

    Indicates if values of this field can be null values.

    metadata

    The metadata of this field. The metadata should be preserved during transformation if the content of the column is not modified, e.g, in selection.

12. case class StructType(fields: Seq[StructField]) extends DataType with Product with Serializable

    

13. abstract class UserDefinedType[UserType] extends DataType with Serializable

    

    ::DeveloperApi:: The data type for User Defined Types (UDTs).

    ::DeveloperApi:: The data type for User Defined Types (UDTs).

    This interface allows a user to make their own classes more interoperable with SparkSQL; e.g., by creating a UserDefinedType for a class X, it becomes possible to create a SchemaRDD which has class X in the schema.

    For SparkSQL to recognize UDTs, the UDT must be annotated with org.apache.spark.sql.catalyst.annotation.SQLUserDefinedType.

    The conversion via serialize occurs when instantiating a SchemaRDD from another RDD. The conversion via deserialize occurs when reading from a SchemaRDD.

    Annotations

    @DeveloperApi()