c

ai.chronon.api

StagingQuery

class StagingQuery extends TBase[StagingQuery, _Fields] with Serializable with Cloneable with Comparable[StagingQuery]

Staging Query encapsulates arbitrary spark computation. One key feature is that the computation follows a "fill-what's-missing" pattern. Basically instead of explicitly specifying dates you specify two macros. {{ start_date }} and {{end_date}}. Chronon will pass in earliest-missing-partition for start_date and execution-date / today for end_date. So the query will compute multiple partitions at once.

Annotations
@SuppressWarnings() @Generated()
Linear Supertypes
Cloneable, TBase[StagingQuery, _Fields], Serializable, TSerializable, Comparable[StagingQuery], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StagingQuery
  2. Cloneable
  3. TBase
  4. Serializable
  5. TSerializable
  6. Comparable
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new StagingQuery(other: StagingQuery)

    Performs a deep copy on other.

  2. new StagingQuery()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addToSetups(elem: String): Unit
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clear(): Unit
    Definition Classes
    StagingQuery → TBase
    Annotations
    @Override()
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  8. def compareTo(other: StagingQuery): Int
    Definition Classes
    StagingQuery → Comparable
    Annotations
    @Override()
  9. def deepCopy(): StagingQuery
    Definition Classes
    StagingQuery → TBase
  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(that: StagingQuery): Boolean
  12. def equals(that: AnyRef): Boolean
    Definition Classes
    StagingQuery → AnyRef → Any
    Annotations
    @Override()
  13. def fieldForId(fieldId: Int): _Fields
    Definition Classes
    StagingQuery → TBase
    Annotations
    @Nullable()
  14. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  15. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. def getFieldValue(field: _Fields): AnyRef
    Definition Classes
    StagingQuery → TBase
    Annotations
    @Nullable()
  17. def getMetaData(): MetaData

    Contains name, team, output_namespace, execution parameters etc.

    Contains name, team, output_namespace, execution parameters etc. Things that don't change the semantics of the computation itself.

    Annotations
    @Nullable()
  18. def getQuery(): String

    Arbitrary spark query that should be written with {{ start_date }}, {{ end_date }} and {{ latest_date }} templates

    Arbitrary spark query that should be written with {{ start_date }}, {{ end_date }} and {{ latest_date }} templates

    • {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.
    • {{ end_date }} is the end partition of the computing range.
    • {{ latest_date }} is the end partition independent of the computing range (meant for cumulative sources).
    • {{ max_date(table=namespace.my_table) }} is the max partition available for a given table.
    Annotations
    @Nullable()
  19. def getSetups(): List[String]

    Spark SQL setup statements.

    Spark SQL setup statements. Used typically to register UDFs.

    Annotations
    @Nullable()
  20. def getSetupsIterator(): Iterator[String]
    Annotations
    @Nullable()
  21. def getSetupsSize(): Int
  22. def getStartPartition(): String

    on the first run, {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.

    on the first run, {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.

    Annotations
    @Nullable()
  23. def hashCode(): Int
    Definition Classes
    StagingQuery → AnyRef → Any
    Annotations
    @Override()
  24. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  25. def isSet(field: _Fields): Boolean

    Returns true if field corresponding to fieldID is set (has been assigned a value) and false otherwise

    Returns true if field corresponding to fieldID is set (has been assigned a value) and false otherwise

    Definition Classes
    StagingQuery → TBase
  26. def isSetMetaData(): Boolean

    Returns true if field metaData is set (has been assigned a value) and false otherwise

  27. def isSetQuery(): Boolean

    Returns true if field query is set (has been assigned a value) and false otherwise

  28. def isSetSetups(): Boolean

    Returns true if field setups is set (has been assigned a value) and false otherwise

  29. def isSetStartPartition(): Boolean

    Returns true if field startPartition is set (has been assigned a value) and false otherwise

  30. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  31. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  32. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  33. def read(iprot: TProtocol): Unit
    Definition Classes
    StagingQuery → TSerializable
  34. def setFieldValue(field: _Fields, value: AnyRef): Unit
    Definition Classes
    StagingQuery → TBase
  35. def setMetaData(metaData: MetaData): StagingQuery

    Contains name, team, output_namespace, execution parameters etc.

    Contains name, team, output_namespace, execution parameters etc. Things that don't change the semantics of the computation itself.

  36. def setMetaDataIsSet(value: Boolean): Unit
  37. def setQuery(query: String): StagingQuery

    Arbitrary spark query that should be written with {{ start_date }}, {{ end_date }} and {{ latest_date }} templates

    Arbitrary spark query that should be written with {{ start_date }}, {{ end_date }} and {{ latest_date }} templates

    • {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.
    • {{ end_date }} is the end partition of the computing range.
    • {{ latest_date }} is the end partition independent of the computing range (meant for cumulative sources).
    • {{ max_date(table=namespace.my_table) }} is the max partition available for a given table.
  38. def setQueryIsSet(value: Boolean): Unit
  39. def setSetups(setups: List[String]): StagingQuery

    Spark SQL setup statements.

    Spark SQL setup statements. Used typically to register UDFs.

  40. def setSetupsIsSet(value: Boolean): Unit
  41. def setStartPartition(startPartition: String): StagingQuery

    on the first run, {{ start_date }} will be set to this user provided start date, future incremental runs will set it to the latest existing partition + 1 day.

  42. def setStartPartitionIsSet(value: Boolean): Unit
  43. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  44. def toString(): String
    Definition Classes
    StagingQuery → AnyRef → Any
    Annotations
    @Override()
  45. def unsetMetaData(): Unit
  46. def unsetQuery(): Unit
  47. def unsetSetups(): Unit
  48. def unsetStartPartition(): Unit
  49. def validate(): Unit
  50. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  51. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  52. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  53. def write(oprot: TProtocol): Unit
    Definition Classes
    StagingQuery → TSerializable

Inherited from Cloneable

Inherited from TBase[StagingQuery, _Fields]

Inherited from Serializable

Inherited from TSerializable

Inherited from Comparable[StagingQuery]

Inherited from AnyRef

Inherited from Any

Ungrouped