c

io.delta.flink.internal.table

DeltaDynamicTableSink

class DeltaDynamicTableSink extends DynamicTableSink with SupportsPartitioning

Sink of a dynamic Flink table to a Delta lake table.

It utilizes new Flink Sink API (available for Flink >= 1.12) and interfaces (available for Flink >= 1.13) provided for interoperability between this new Sink API and Table API. It also supports static partitioning.

For regular batch scenarios, the sink can solely accept insert-only rows and write out bounded streams.

For regular streaming scenarios, the sink can solely accept insert-only rows and can write out unbounded streams.

Linear Supertypes
SupportsPartitioning, DynamicTableSink, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaDynamicTableSink
  2. SupportsPartitioning
  3. DynamicTableSink
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaDynamicTableSink(basePath: Path, hadoopConf: Configuration, rowType: RowType, catalogTable: CatalogTable)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def applyStaticPartition(partition: Map[String, String]): Unit

    Static values for partitions that should set explicitly instead of being derived from the content of the records.

    Static values for partitions that should set explicitly instead of being derived from the content of the records.

    If all partition keys get a value assigned in the PARTITION clause, the operation is considered an "insertion into a static partition". In the below example, the query result should be written into the static partition region='europe', month='2020-01' which will be passed by the planner into applyStaticPartition(Map).

    INSERT INTO t PARTITION (region='europe', month='2020-01') SELECT a, b, c FROM my_view;

    If only a subset of all partition keys get a static value assigned in the PARTITION clause or with a constant part in a SELECT clause, the operation is considered an "insertion into a dynamic partition". In the below example, the static partition part is region='europe' which will be passed by the planner into #applyStaticPartition(Map). The remaining values for partition keys should be obtained from each individual record by the sink during runtime.

    INSERT INTO t PARTITION (region='europe') SELECT a, b, c, month FROM another_view;

    partition

    map of static partitions and their values.

    Definition Classes
    DeltaDynamicTableSink → SupportsPartitioning
    Annotations
    @Override()
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def asSummaryString(): String
    Definition Classes
    DeltaDynamicTableSink → DynamicTableSink
    Annotations
    @Override()
  7. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. def copy(): DynamicTableSink
    Definition Classes
    DeltaDynamicTableSink → DynamicTableSink
    Annotations
    @Override()
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. def getChangelogMode(requestedMode: ChangelogMode): ChangelogMode

    Returns the set of changes that the sink accepts during runtime.

    Returns the set of changes that the sink accepts during runtime.

    requestedMode

    expected set of changes by the current plan

    returns

    ChangelogMode only allowing for inserts to the Delta table

    Definition Classes
    DeltaDynamicTableSink → DynamicTableSink
    Annotations
    @Override()
  13. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def getSinkRuntimeProvider(context: Context): SinkRuntimeProvider

    Utility method for transition from Flink's DataStream to Table API.

    Utility method for transition from Flink's DataStream to Table API.

    context

    Context for creating runtime implementation via a SinkRuntimeProvider.

    returns

    provider representing DeltaSink implementation for writing the data to a Delta table.

    Definition Classes
    DeltaDynamicTableSink → DynamicTableSink
    Annotations
    @Override()
  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. def requiresPartitionGrouping(arg0: Boolean): Boolean
    Definition Classes
    SupportsPartitioning
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from SupportsPartitioning

Inherited from DynamicTableSink

Inherited from AnyRef

Inherited from Any

Ungrouped