Type Members

1.  abstract class BaseCatalog extends AbstractCatalog

   Base implementation of Flink catalog.

   Base implementation of Flink catalog. This class handles Catalog operations for Delta tables that do not require interaction with _delta_log, for example view, database operations etc.

2.  final class CatalogExceptionHelper extends AnyRef
3.  trait CatalogLoader extends Serializable

   Creates a concrete catalog instance that will be used as decorated catalog by DeltaCatalog.

4.  class CatalogProxy extends BaseCatalog

   A proxy class that redirects calls to Delta Catalog or decorated catalog depending on table type.

5.  class DeltaCatalog extends AnyRef

   Delta Catalog implementation.

   Delta Catalog implementation. This class executes calls to _delta_log for catalog operations such as createTable, getTable etc. This class also prepares, persists and uses data store in metastore using decorated catalog implementation.

   Catalog operations that are not in scope of Delta Table or do not require _delta_log operations will be handled by CatalogProxy and BaseCatalog classes.

6.  class DeltaCatalogBaseTable extends AnyRef

   Data object used by Delta Catalog implementation API that wraps Flink's ObjectPath represented by databaseName.tableName and CatalogBaseTable containing table properties and schema from DDL.

7.  class DeltaCatalogContext extends Context

   Basic implementation of Flink's CatalogFactory.Context that is needed as an argument of Flink's CatalogFactory#createCatalog(Context) used by CatalogLoader.

   Basic implementation of Flink's CatalogFactory.Context that is needed as an argument of Flink's CatalogFactory#createCatalog(Context) used by CatalogLoader.

   All Flink's implementations of CatalogFactory.Context are marked as @Internal so not meant to be used by users. This implementation is based on Flink's org.apache.flink.table.factories.FactoryUtil.DefaultCatalogContext

8.  class DeltaCatalogFactory extends CatalogFactory

   The catalog factory implementation for Delta Catalog.

   The catalog factory implementation for Delta Catalog. This factory will be discovered by Flink runtime using Java’s Service Provider Interfaces (SPI) based on resources/META-INF/services/org.apache.flink.table.factories.Factory file.

   Flink runtime will call #createCatalog(Context) method that will return new Delta Catalog instance.

9.  final class DeltaCatalogTableHelper extends AnyRef
10.  class DeltaDynamicTableFactory extends DynamicTableSinkFactory with DynamicTableSourceFactory

    Creates a DynamicTableSink and DynamicTableSource instance representing DeltaLake table.

    Creates a DynamicTableSink and DynamicTableSource instance representing DeltaLake table.

    This implementation automatically resolves all necessary object for creating instance of io.delta.flink.sink.DeltaSink and io.delta.flink.source.DeltaSource except Delta table's path that needs to be provided explicitly.

11.  class DeltaDynamicTableSink extends DynamicTableSink with SupportsPartitioning

    Sink of a dynamic Flink table to a Delta lake table.

    Sink of a dynamic Flink table to a Delta lake table.

    It utilizes new Flink Sink API (available for Flink >= 1.12) and interfaces (available for Flink >= 1.13) provided for interoperability between this new Sink API and Table API. It also supports static partitioning.

    For regular batch scenarios, the sink can solely accept insert-only rows and write out bounded streams.

    For regular streaming scenarios, the sink can solely accept insert-only rows and can write out unbounded streams.

12.  class DeltaDynamicTableSource extends ScanTableSource

    Implementation of ScanTableSource interface for Table/SQL support for Delta Source connector.

13.  class DeltaFlinkJobSpecificOptions extends AnyRef

    This class contains Flink job-specific options for io.delta.flink.source.DeltaSource and io.delta.flink.sink.DeltaSink that are relevant for Table API.

    This class contains Flink job-specific options for io.delta.flink.source.DeltaSource and io.delta.flink.sink.DeltaSink that are relevant for Table API. For Table API, this options can be set only using Flink, dynamic table options from DML/DQL query level, for example:

    
       SELECT * FROM my_delta_source_table /*+ OPTIONS(‘mode' = 'streaming')
     
    

    Flink job-specific options are not stored in metastore nor in Delta Log. Their scope is single Flink Job (DML/DQL query) only.

    In practice this class will contain options from io.delta.flink.source.internal.DeltaSourceOptions and io.delta.flink.sink.internal.DeltaSinkOptions + extra ones like MODE.

14.  class DeltaTableConnectorOptions extends AnyRef

    Options for the Flink TableAPI's DeltaSink connector.

15.  final class DeltaTableFactoryHelper extends AnyRef
16.  class HadoopUtils extends AnyRef

    Utility class to discover and process Hadoop configuration on Flink cluster.

    Utility class to discover and process Hadoop configuration on Flink cluster.

    This class was backport from Flink's flink-hadoop-fs module, and it contains a subset of methods comparing to the original class. We kept only needed methods.

    The reason for backport this to connector code was that original implementation requires various additional hadoop classes to be loaded on the classpath which required adding additional hadoop dependencies to the project.

17.  class QueryOptions extends AnyRef

    Data object containing information about Delta table path, query mode (streaming or batch) and used job-specific options such as startingVersion, versionAsOf etc.