Packages

class AirflowLauncher extends LaunchHandler with StrictLogging

Airflow Launcher will submit a request for ingestion to Airflow using the REST API. The requested DAG must exist in Airflow first.

Linear Supertypes
StrictLogging, LaunchHandler, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AirflowLauncher
  2. StrictLogging
  3. LaunchHandler
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new AirflowLauncher()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def bqload(workflow: IngestionWorkflow, config: BigQueryLoadConfig)(implicit settings: Settings): Boolean

    Load to BigQuery

    Load to BigQuery

    Definition Classes
    AirflowLauncherLaunchHandler
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def esLoad(workflow: IngestionWorkflow, config: ESLoadConfig)(implicit settings: Settings): Boolean

    Index into elasticsearch

    Index into elasticsearch

    Definition Classes
    AirflowLauncherLaunchHandler
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def ingest(workflow: IngestionWorkflow, domain: Domain, schema: Schema, paths: List[Path], options: Map[String, String])(implicit settings: Settings): Try[JobResult]

    Request the execution of the "comet-ingest" DAG in Airflow

    Request the execution of the "comet-ingest" DAG in Airflow

    domain

    : Domain to which belong this dataset

    schema

    : Schema of the dataset

    paths

    : absolute paths where the source datasets (JSON / CSV / ...) are located

    returns

    success if request accepted

    Definition Classes
    AirflowLauncherLaunchHandler
  14. def ingest(workflow: IngestionWorkflow, domain: Domain, schema: Schema, path: Path, options: Map[String, String])(implicit settings: Settings): Try[JobResult]

    Submit to the cron manager a single file for ingestion

    Submit to the cron manager a single file for ingestion

    domain

    : Domain to which belong this dataset

    schema

    : Schema of the dataset

    path

    : absolute path where the source dataset (JSON / CSV / ...) is located

    returns

    success / failure

    Definition Classes
    LaunchHandler
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. def jdbcload(workflow: IngestionWorkflow, config: ConnectionLoadConfig)(implicit settings: Settings): Boolean

    Load to JDBC sink

    Load to JDBC sink

    Definition Classes
    AirflowLauncherLaunchHandler
  17. val logger: Logger
    Attributes
    protected
    Definition Classes
    StrictLogging
  18. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. def post(url: String, command: String): Try[AirflowJobResult]
    Attributes
    protected
  22. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  23. def toString(): String
    Definition Classes
    AnyRef → Any
  24. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from StrictLogging

Inherited from LaunchHandler

Inherited from AnyRef

Inherited from Any

Ungrouped