Packages

o

org.apache.spark.sql.delta

AllowColumnDefaultsTableFeature

object AllowColumnDefaultsTableFeature extends WriterFeature

This table feature represents support for column DEFAULT values for Delta Lake. With this feature, it is possible to assign default values to columns either at table creation time or later by using commands of the form: ALTER TABLE t ALTER COLUMN c SET DEFAULT v. Thereafter, queries from the table will return the specified default value instead of NULL when the corresponding field is not present in storage.

We create this as a writer-only feature rather than a reader/writer feature in order to simplify the query execution implementation for scanning Delta tables. This means that commands of the following form are not allowed: ALTER TABLE t ADD COLUMN c DEFAULT v. The reason is that when commands of that form execute (such as for other data sources like CSV or JSON), then the data source scan implementation must take responsibility to return the supplied default value for all rows, including those previously present in the table before the command executed. We choose to avoid this complexity for Delta table scans, so we make this a writer-only feature instead. Therefore, the analyzer can take care of the entire job when processing commands that introduce new rows into the table by injecting the column default value (if present) into the corresponding query plan. This comes at the expense of preventing ourselves from easily adding a default value to an existing non-empty table, because all data files would need to be rewritten to include the new column value in an expensive backfill.

Linear Supertypes
WriterFeature, TableFeature, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AllowColumnDefaultsTableFeature
  2. WriterFeature
  3. TableFeature
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. def isLegacyFeature: Boolean

    Determine if this feature is a legacy feature.

    Determine if this feature is a legacy feature. See the documentation of TableFeature for more information.

    Definition Classes
    TableFeature
  13. def isReaderWriterFeature: Boolean

    Determine if this feature applies to both readers and writers.

    Determine if this feature applies to both readers and writers.

    Definition Classes
    TableFeature
  14. def isRemovable: Boolean

    True if this feature can be removed.

    True if this feature can be removed.

    Definition Classes
    TableFeature
  15. def minProtocolVersion: Protocol

    Get a Protocol object stating the minimum reader and writer versions this feature requires.

    Get a Protocol object stating the minimum reader and writer versions this feature requires. For a feature that can only be explicitly supported, this method returns a protocol version that supports table features, either (0,7) or (3,7) depending on the feature is writer-only or reader-writer. For a legacy feature that can be implicitly supported, this method returns the first protocol version which introduced the said feature.

    For all features, if the table's protocol version does not support table features, then the minimum protocol version is enough. However, if the protocol version supports table features for the feature type (writer-only or reader-writer), then the minimum protocol version is not enough to support a feature. In this case the feature must also be explicitly listed in the appropriate feature sets in the Protocol.

    Definition Classes
    TableFeature
  16. val minReaderVersion: Int
    Definition Classes
    TableFeature
  17. val minWriterVersion: Int
    Definition Classes
    TableFeature
  18. val name: String
    Definition Classes
    TableFeature
  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. def requiredFeatures: Set[TableFeature]

    Set of table features that this table feature depends on.

    Set of table features that this table feature depends on. I.e. the set of features that need to be enabled if this table feature is enabled.

    Definition Classes
    TableFeature
  23. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  24. def toString(): String
    Definition Classes
    AnyRef → Any
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from WriterFeature

Inherited from TableFeature

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped