Packages

c

it.agilelab.bigdata.wasp.consumers.spark.strategies.cdc

GoldenGateAdapterFlatModelStrategy

class GoldenGateAdapterFlatModelStrategy extends Strategy with Logging

Strategy that enable to map a flat mutation model to be mapped to an insert/update/delete object that can be sent to the CDC plugin that writes on DeltaLake. So having has input the raw flat mutations coming from a goldengate topic it will produce in output a dataframe composed of rows that has the shape accepted in input by the cdc plugin.

NB:

  • this strategy is used to map the mutation incoming from what in the oracle language is known as: Row Formatter, if you need to map a message that is incoming from an Operation Formatter you need to wait the new feature for that. More details are available under: operation vs row formatter
  • to enable the correct working of the strategy you need to ensure at runtime the configuration with path: goldengate.key.fields. This configuration is required and contains the list of primary keys fields for the mutation table. Suppose for example to have a table with the following structure: SHOP_TABLE ===> "PRODUCT_AMOUNT": Integer "TRANSACTION_ID": Integer "ORDER_DATE": Timestamp "PRODUCT_PRICE": Char "ORDER_ID": Integer "CUST_CODE": Long "PRODUCT_CODE": String and the primary key of this table is composed by the fields:
  • CUST_CODE
  • ORDER_DATE
  • PRODUCT_CODE
  • ORDER_ID

in this case you need to insert the configuration the following line:

goldengate.key.fields=["CUST_CODE", "ORDER_DATE", "PRODUCT_CODE", "ORDER_ID"]"

Linear Supertypes
Logging, Strategy, Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. GoldenGateAdapterFlatModelStrategy
  2. Logging
  3. Strategy
  4. Serializable
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new GoldenGateAdapterFlatModelStrategy()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val ALTER_DATE_DAY: String
    Definition Classes
    Strategy
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. val configuration: Config
    Definition Classes
    Strategy
  8. def enrichFinalDf: (DataFrame) ⇒ DataFrame

    Eventual enrichment function of the final DF.

    Eventual enrichment function of the final DF.

    returns

    final output DF

  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. val logger: WaspLogger
    Attributes
    protected
    Definition Classes
    Logging
  16. val mlModelsBroadcast: MlModelsBroadcastDB
    Definition Classes
    Strategy
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. def prepareInitialDf: (DataFrame) ⇒ DataFrame

    Eventual preparation function of the initial DF an example can be the removal of fields that are not required to be mapped in the table or a mapping with a default value.

    Eventual preparation function of the initial DF an example can be the removal of fields that are not required to be mapped in the table or a mapping with a default value.

    returns

    final output DF

  21. val sparkContext: Option[SparkContext]
    Definition Classes
    Strategy
  22. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  23. def toString(): String
    Definition Classes
    Strategy → AnyRef → Any
  24. def transform(dataFrames: Map[ReaderKey, DataFrame]): DataFrame

    Strategy that read data from Kafka, transform the format to make it compliant with the DataLake format that the writer expect.

    Strategy that read data from Kafka, transform the format to make it compliant with the DataLake format that the writer expect.

    dataFrames

    the dataframe that need to be transformed

    returns

    a dataframe transformed that can be sent to the CDCWriter

    Definition Classes
    GoldenGateAdapterFlatModelStrategy → Strategy
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Logging

Inherited from Strategy

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped