Packages

c

io.delta.flink.source

RowDataContinuousDeltaSourceBuilder

class RowDataContinuousDeltaSourceBuilder extends ContinuousDeltaSourceBuilder[RowData, RowDataContinuousDeltaSourceBuilder]

A builder class for DeltaSource for a stream of RowData where the created source instance will operate in Continuous mode.

In Continuous mode, the DeltaSource will, by default, load the full state of the latest table version, and then start monitoring for changes. If you use either the RowDataContinuousDeltaSourceBuilder#startingVersion or RowDataContinuousDeltaSourceBuilder#startingTimestamp APIs, then the DeltaSource will start monitoring for changes from that historical version. It will not load the full table state at that historical table version.

For most common use cases use DeltaSource#forContinuousRowData utility method to instantiate the source. After instantiation of this builder you can either call RowDataBoundedDeltaSourceBuilder#build() method to get the instance of a DeltaSource or configure additional options using builder's API.

Linear Supertypes
ContinuousDeltaSourceBuilder[RowData, RowDataContinuousDeltaSourceBuilder], DeltaSourceBuilderBase[RowData, RowDataContinuousDeltaSourceBuilder], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RowDataContinuousDeltaSourceBuilder
  2. ContinuousDeltaSourceBuilder
  3. DeltaSourceBuilderBase
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def build[V <: DeltaSource[RowData]](): V
    Definition Classes
    DeltaSourceBuilderBase

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def build(): DeltaSource[RowData]

    Creates an instance of DeltaSource for a stream of RowData.

    Creates an instance of DeltaSource for a stream of RowData. Created source will work in Continuous mode, actively monitoring Delta table for new changes.

    This method can throw DeltaSourceValidationException in case of invalid arguments passed to Delta source builder.

    returns

    New DeltaSource instance.

    Annotations
    @Override() @SuppressWarnings()
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. def columnNames(columnNames: <repeated...>[String]): RowDataContinuousDeltaSourceBuilder

    Specifies an array of column names that should be read from Delta table.

    Specifies an array of column names that should be read from Delta table. If this method is not used, Source will read all columns from Delta table.

    If provided List is null or contains null, empty or blank elements it will throw a DeltaSourceValidationException by builder after calling build() method.

    columnNames

    column names that should be read.

  8. def columnNames(columnNames: List[String]): RowDataContinuousDeltaSourceBuilder

    Specifies a List of column names that should be read from Delta table.

    Specifies a List of column names that should be read from Delta table. If this method is not used, Source will read all columns from Delta table.

    If provided List is null or contains null, empty or blank elements it will throw a DeltaSourceValidationException by builder after calling build() method.

    columnNames

    column names that should be read.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → DeltaSourceBuilderBase
    Annotations
    @Override()
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. def getApplicableOptions(): Collection[String]
    Attributes
    protected[builder]
    Definition Classes
    ContinuousDeltaSourceBuilder → DeltaSourceBuilderBase
    Annotations
    @Override()
  13. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def getSourceConfiguration(): DeltaConnectorConfiguration

    returns

    A copy of DeltaConnectorConfiguration used by builder. The changes made on returned copy do not change the state of builder's configuration.

    Definition Classes
    DeltaSourceBuilderBase
  15. def getSourceSchema(): SourceSchema

    Extracts Delta table schema from DeltaLog io.delta.standalone.actions.Metadata including column names and column types converted to org.apache.flink.table.types.logical.LogicalType.

    Extracts Delta table schema from DeltaLog io.delta.standalone.actions.Metadata including column names and column types converted to org.apache.flink.table.types.logical.LogicalType.

    If #userColumnNames were defined, only those columns will be included in extracted schema.

    returns

    A SourceSchema including Delta table column names with their types that should be read from Delta table.

    Attributes
    protected[builder]
    Definition Classes
    DeltaSourceBuilderBase
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def ignoreChanges(ignoreChanges: Boolean): RowDataContinuousDeltaSourceBuilder

    Sets the "ignoreChanges" option.

    Sets the "ignoreChanges" option. When set to true, this option allows processing Delta table versions where data is changed (i.e. updated) or deleted.

    Note that setting this option to true can lead to duplicate processing of data, as, in the case of updates, existing rows may be rewritten in new files, and those new files will be treated as new data and be fully reprocessed.

    This option subsumes #ignoreDeletes option. Therefore, if you set "ignoreChanges" to true, your stream will not be disrupted by either deletions or updates to the source table.

    The default value for this option is false.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → ContinuousDeltaSourceBuilder
    Annotations
    @Override()
  18. def ignoreDeletes(ignoreDeletes: Boolean): RowDataContinuousDeltaSourceBuilder

    Sets the "ignoreDeletes" option.

    Sets the "ignoreDeletes" option. When set to true, this option allows processing Delta table versions where data is deleted.

    The default value for this option is false.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → ContinuousDeltaSourceBuilder
    Annotations
    @Override()
  19. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  20. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  23. def option(optionName: String, optionValue: Long): RowDataContinuousDeltaSourceBuilder

    Sets a configuration option.

    Sets a configuration option.

    optionName

    Option name to set.

    optionValue

    Option long value to set.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → DeltaSourceBuilderBase
    Annotations
    @Override()
  24. def option(optionName: String, optionValue: Int): RowDataContinuousDeltaSourceBuilder

    Sets a configuration option.

    Sets a configuration option.

    optionName

    Option name to set.

    optionValue

    Option int value to set.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → DeltaSourceBuilderBase
    Annotations
    @Override()
  25. def option(optionName: String, optionValue: Boolean): RowDataContinuousDeltaSourceBuilder

    Sets a configuration option.

    Sets a configuration option.

    optionName

    Option name to set.

    optionValue

    Option boolean value to set.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → DeltaSourceBuilderBase
    Annotations
    @Override()
  26. def option(optionName: String, optionValue: String): RowDataContinuousDeltaSourceBuilder

    Sets a configuration option.

    Sets a configuration option.

    optionName

    Option name to set.

    optionValue

    Option String value to set.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → DeltaSourceBuilderBase
    Annotations
    @Override()
  27. def prepareInapplicableOptionMessage(usedOptions: Collection[String], applicableOptions: Collection[String]): String
    Attributes
    protected[builder]
    Definition Classes
    DeltaSourceBuilderBase
  28. def prepareOptionExclusionMessage(mutualExclusiveOptions: <repeated...>[String]): String
    Attributes
    protected[builder]
    Definition Classes
    DeltaSourceBuilderBase
  29. def self(): RowDataContinuousDeltaSourceBuilder
    Attributes
    protected[builder]
    Definition Classes
    DeltaSourceBuilderBase
    Annotations
    @SuppressWarnings()
  30. def startingTimestamp(startingTimestamp: String): RowDataContinuousDeltaSourceBuilder

    Sets value of "startingTimestamp" option.

    Sets value of "startingTimestamp" option. This option is used to read only changes starting from the table version that was generated at or after the given timestamp.

    This option is mutually exclusive with #startingVersion(String) and #startingVersion(long) option.

    startingTimestamp

    The timestamp of the table from which we start reading changes. Supported formats are:

    • 2022-02-24
    • 2022-02-24 04:55:00
    • 2022-02-24 04:55:00.001
    • 2022-02-24T04:55:00
    • 2022-02-24T04:55:00.001
    • 2022-02-24T04:55:00.001Z
    Definition Classes
    RowDataContinuousDeltaSourceBuilder → ContinuousDeltaSourceBuilder
    Annotations
    @Override()
  31. def startingVersion(startingVersion: Long): RowDataContinuousDeltaSourceBuilder

    Sets value of "startingVersion" option.

    Sets value of "startingVersion" option. This option specifies the starting table version from which we want to start reading changes.

    This option is mutually exclusive with #startingTimestamp(String) option.

    startingVersion

    Delta table version to start reading changes from.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → ContinuousDeltaSourceBuilder
    Annotations
    @Override()
  32. def startingVersion(startingVersion: String): RowDataContinuousDeltaSourceBuilder

    Sets value of "startingVersion" option.

    Sets value of "startingVersion" option. This option specifies the starting table version from which we want to start reading changes.

    This option is mutually exclusive with #startingTimestamp(String) option.

    startingVersion

    Delta table version to start reading changes from. The values can be string numbers like "1", "10" etc. or keyword "latest", where in that case, changes from the latest Delta table version will be read.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → ContinuousDeltaSourceBuilder
    Annotations
    @Override()
  33. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  34. def toString(): String
    Definition Classes
    AnyRef → Any
  35. def updateCheckIntervalMillis(updateCheckInterval: Long): RowDataContinuousDeltaSourceBuilder

    Sets the value for "updateCheckIntervalMillis" option.

    Sets the value for "updateCheckIntervalMillis" option. This option is used to specify the check interval (in milliseconds) used for periodic Delta table changes checks.

    The default value for this option is 5000 ms.

    updateCheckInterval

    The update check internal in milliseconds.

    Definition Classes
    RowDataContinuousDeltaSourceBuilder → ContinuousDeltaSourceBuilder
    Annotations
    @Override()
  36. def validate(): Unit

    Validate definition of Delta source builder including mandatory and optional options.

    Validate definition of Delta source builder including mandatory and optional options.

    Attributes
    protected[builder]
    Definition Classes
    DeltaSourceBuilderBase
  37. def validateInapplicableOptions(): Validator

    Validated builder options that were used but they might be not applicable for given builder type, for example using options from bounded mode like "versionAsOf" for continuous mode builder.

    Validated builder options that were used but they might be not applicable for given builder type, for example using options from bounded mode like "versionAsOf" for continuous mode builder.

    returns

    The Validator object with all (if any) validation error messages.

    Attributes
    protected[builder]
    Definition Classes
    DeltaSourceBuilderBase
  38. def validateMandatoryOptions(): Validator
    Attributes
    protected[builder]
    Definition Classes
    DeltaSourceBuilderBase
  39. def validateOptionExclusions(): Validator

    This method should implement any logic for validation of mutually exclusive options.

    This method should implement any logic for validation of mutually exclusive options.

    returns

    Validator instance with validation error message.

    Attributes
    protected[builder]
    Definition Classes
    ContinuousDeltaSourceBuilder → DeltaSourceBuilderBase
    Annotations
    @Override()
  40. def validateOptionalParameters(): Validator
    Attributes
    protected[builder]
    Definition Classes
    DeltaSourceBuilderBase
  41. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  42. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  43. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from ContinuousDeltaSourceBuilder[RowData, RowDataContinuousDeltaSourceBuilder]

Inherited from DeltaSourceBuilderBase[RowData, RowDataContinuousDeltaSourceBuilder]

Inherited from AnyRef

Inherited from Any

Ungrouped