Packages

class CompatibilitySparkHadoopUtil extends Logging

Contains util methods to interact with Hadoop from Spark.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CompatibilitySparkHadoopUtil
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CompatibilitySparkHadoopUtil()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addCredentials(conf: JobConf): Unit

    Add any user credentials to the job conf which are necessary for running on a secure Hadoop cluster.

  5. def addCurrentUserCredentials(creds: Credentials): Unit
  6. def addDelegationTokens(tokens: Array[Byte], sparkConf: SparkConf): Unit

    Add or overwrite current user's credentials with serialized delegation tokens, also confirms correct hadoop configuration is set.

  7. def appendS3AndSparkHadoopHiveConfigurations(conf: SparkConf, hadoopConf: Configuration): Unit

    Appends S3-specific, spark.hadoop.*, and spark.buffer.size configurations to a Hadoop configuration.

  8. def appendSparkHadoopConfigs(srcMap: Map[String, String], destMap: HashMap[String, String]): Unit

    Appends spark.hadoop.* configurations from a Map to another without the spark.hadoop.

    Appends spark.hadoop.* configurations from a Map to another without the spark.hadoop. prefix.

  9. def appendSparkHadoopConfigs(conf: SparkConf, hadoopConf: Configuration): Unit

    Appends spark.hadoop.* configurations from a SparkConf to a Hadoop configuration without the spark.hadoop.

    Appends spark.hadoop.* configurations from a SparkConf to a Hadoop configuration without the spark.hadoop. prefix.

  10. def appendSparkHiveConfigs(srcMap: Map[String, String], destMap: HashMap[String, String]): Unit
  11. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  12. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  13. val conf: Configuration
  14. def createSparkUser(): UserGroupInformation
  15. def deserialize(tokenBytes: Array[Byte]): Credentials
  16. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  18. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  19. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  20. def getFSBytesReadOnThreadCallback(): () ⇒ Long

    Returns a function that can be called to find Hadoop FileSystem bytes read.

    Returns a function that can be called to find Hadoop FileSystem bytes read. If getFSBytesReadOnThreadCallback is called from thread r at time t, the returned callback will return the bytes read on r since t.

    Annotations
    @silent( "deprecated" )
  21. def getFSBytesWrittenOnThreadCallback(): () ⇒ Long

    Returns a function that can be called to find Hadoop FileSystem bytes written.

    Returns a function that can be called to find Hadoop FileSystem bytes written. If getFSBytesWrittenOnThreadCallback is called from thread r at time t, the returned callback will return the bytes written on r since t.

    returns

    None if the required method can't be found.

    Annotations
    @silent( "deprecated" )
  22. def getSuffixForCredentialsPath(credentialsPath: Path): Int
  23. def globPath(fs: FileSystem, pattern: Path): Seq[Path]
  24. def globPath(pattern: Path): Seq[Path]
  25. def globPathIfNecessary(fs: FileSystem, pattern: Path): Seq[Path]
  26. def globPathIfNecessary(pattern: Path): Seq[Path]
  27. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  28. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  29. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def isGlobPath(pattern: Path): Boolean
  31. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  32. def isProxyUser(ugi: UserGroupInformation): Boolean
  33. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  34. def listFilesSorted(remoteFs: FileSystem, dir: Path, prefix: String, exclusionSuffix: String): Array[FileStatus]

    Lists all the files in a directory with the specified prefix, and does not end with the given suffix.

    Lists all the files in a directory with the specified prefix, and does not end with the given suffix. The returned {{FileStatus}} instances are sorted by the modification times of the respective files.

  35. def listLeafDirStatuses(fs: FileSystem, baseStatus: FileStatus): Seq[FileStatus]
  36. def listLeafDirStatuses(fs: FileSystem, basePath: Path): Seq[FileStatus]
  37. def listLeafStatuses(fs: FileSystem, baseStatus: FileStatus): Seq[FileStatus]

    Get FileStatus objects for all leaf children (files) under the given base path.

    Get FileStatus objects for all leaf children (files) under the given base path. If the given path points to a file, return a single-element collection containing FileStatus of that file.

  38. def listLeafStatuses(fs: FileSystem, basePath: Path): Seq[FileStatus]

    Get FileStatus objects for all leaf children (files) under the given base path.

    Get FileStatus objects for all leaf children (files) under the given base path. If the given path points to a file, return a single-element collection containing FileStatus of that file.

  39. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  40. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  47. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  49. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def loginUserFromKeytab(principalName: String, keytabFilename: String): Unit
  52. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  53. def newConfiguration(conf: SparkConf): Configuration

    Return an appropriate (subclass) of Configuration.

    Return an appropriate (subclass) of Configuration. Creating config can initialize some Hadoop subsystems.

  54. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  55. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  56. def runAsSparkUser(func: () ⇒ Unit): Unit

    Runs the given function with a Hadoop UserGroupInformation as a thread local variable (distributed to child threads), used for authenticating HDFS and YARN calls.

    Runs the given function with a Hadoop UserGroupInformation as a thread local variable (distributed to child threads), used for authenticating HDFS and YARN calls.

    IMPORTANT NOTE: If this function is going to be called repeated in the same process you need to look https://issues.apache.org/jira/browse/HDFS-3545 and possibly do a FileSystem.closeAllForUGI in order to avoid leaking Filesystems

  57. def serialize(creds: Credentials): Array[Byte]
  58. def substituteHadoopVariables(text: String, hadoopConf: Configuration): String

    Substitute variables by looking them up in Hadoop configs.

    Substitute variables by looking them up in Hadoop configs. Only variables that match the ${hadoopconf- .. } pattern are substituted.

  59. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  60. def toString(): String
    Definition Classes
    AnyRef → Any
  61. def transferCredentials(source: UserGroupInformation, dest: UserGroupInformation): Unit
  62. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  63. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  64. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped