Packages

o

com.coxautodata

SparkDistCP

object SparkDistCP extends Logging

Spark-based DistCp application. SparkDistCP.main is the command-line entry to the application and SparkDistCP.run is the programmatic API entry to the application

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkDistCP
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. type KeyedCopyDefinition = (URI, CopyDefinitionWithDependencies)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. def isTraceEnabled: Boolean
    Attributes
    protected
    Definition Classes
    Logging
  13. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  14. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  15. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  16. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  17. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  18. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  19. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  20. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def main(args: Array[String]): Unit

    Main entry point for command-line.

    Main entry point for command-line. Arguments are currently: Usage: SparkDistCP [options] [source_path...] <target_path>

    --i Ignore failures --log <value> Write logs to a URI --dryrun Perform a trial run with no changes made --verbose Run in verbose mode --overwrite Overwrite destination --update Overwrite if source and destination differ in size, or checksum --filters <value> The path to a file containing a list of pattern strings, one string per line, such that paths matching the pattern will be excluded from the copy. --delete Delete the files existing in the dst but not in src --numListstatusThreads <value> Number of threads to use for building file listing --consistentPathBehaviour Revert the path behaviour when using overwrite or update to the path behaviour of non-overwrite/non-update --maxFilesPerTask <value> Maximum number of files to copy in a single Spark task --maxBytesPerTask <value> Maximum number of bytes to copy in a single Spark task --help prints this usage text [source_path...] <target_path>

  25. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  26. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  27. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  28. def run(sparkSession: SparkSession, sourcePaths: Seq[Path], destinationPath: Path, options: SparkDistCPOptions): Unit

    Main entry point for programmatic access to the application.

    Main entry point for programmatic access to the application.

    sparkSession

    Active Spark Session

    sourcePaths

    Source paths to copy from

    destinationPath

    Destination path to copy to

    options

    Options to use in the application

  29. def setLogLevel(level: Level): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  31. def toString(): String
    Definition Classes
    AnyRef → Any
  32. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  33. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped