Packages

package utils

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. case class AirflowJobResult(response: String) extends JobResult with Product with Serializable
  2. trait CliConfig[T] extends AnyRef
  3. trait CometJacksonModule extends Module with JacksonModule

    This module handles some specific type serializers in a central way (so we don't need to pepper the model code with annotations)

  4. class CometObjectMapper extends ObjectMapper with ScalaObjectMapper
  5. class FileLock extends StrictLogging

    HDFS does not have a file locking mechanism.

    HDFS does not have a file locking mechanism. We implement it here in the following way - A file is locked when it's modification time is less than 5000ms than the current time - If the modification time is older tahn the current time of more than 5000ms, it is considered unlocked. - The owner of a lock spawn a thread that update the modification every 5s to keep the lock - The process willing to get the lock check every 5s second for the modification time of the file to gain the lock. - When a process gain the lock, it deletes the file first and tries to create a new one, this make sure that of two process gaining the lock, only one will be able to recreate it after deletion.

    To gain the lock, call tryLock, to release it, call release(). If you are going to use the lock and release it within the same call stack frame, please consider calling tryExclusively() (as in exclusive(delay) { action }) instead.

  6. trait Formatter extends AnyRef
  7. trait JobBase extends StrictLogging

    All Spark Job extend this trait.

    All Spark Job extend this trait. Build Spark session using spark variables from application.conf.

  8. trait JobResult extends AnyRef
  9. class SinkUtils extends StrictLogging
  10. trait SparkJob extends JobBase
  11. case class SparkJobResult(dataframe: Option[DataFrame]) extends JobResult with Product with Serializable
  12. case class Version(version: String) extends Comparable[Version] with Product with Serializable

Value Members

  1. object CometJacksonModule extends Module with CometJacksonModule
  2. object CometObjectMapper extends Serializable
  3. object DataTypeEx
  4. object FileLock
  5. object Formatter extends Formatter
  6. object Unpacker
  7. object Utils

Ungrouped