case classSchemaType(typeName: String, nullable: Boolean) extends Product with Serializable
Schema Converter for getting schema in json format into a spark Structure
The given schema for spark has almost no validity checks, so it will make sense
to combine this with the schema-validator. For loading data with schema, data is converted
to the type given in the schema. If this is not possible the whole row will be null (!).
A field can be null if its type is a 2-element array, one of which is "null". The converted
schema doesn't check for 'enum' fields, i.e. fields which are limited to a given set.
It also doesn't check for required fields or if additional properties are set to true
or false. If a field is specified in the schema, than you can select it and it will
be null if missing. If a field is not in the schema, it cannot be selected even if
given in the dataset.
Linear Supertypes
Serializable, Serializable, Product, Equals, AnyRef, Any
Schema Converter for getting schema in json format into a spark Structure
The given schema for spark has almost no validity checks, so it will make sense to combine this with the schema-validator. For loading data with schema, data is converted to the type given in the schema. If this is not possible the whole row will be null (!). A field can be null if its type is a 2-element array, one of which is "null". The converted schema doesn't check for 'enum' fields, i.e. fields which are limited to a given set. It also doesn't check for required fields or if additional properties are set to true or false. If a field is specified in the schema, than you can select it and it will be null if missing. If a field is not in the schema, it cannot be selected even if given in the dataset.