Packages

c

org.apache.spark.sql.delta.streaming

SchemaTrackingLog

class SchemaTrackingLog[T <: PartitionAndDataSchema] extends HDFSMetadataLog[T]

The underlying class for a streaming log that keeps track of a sequence of schema changes.

It keeps tracks of the sequence of schema changes that this log is aware of, and it detects any concurrent modifications to the schema log to prevent accidents on a best effort basis.

Linear Supertypes
HDFSMetadataLog[T], Logging, MetadataLog[T], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SchemaTrackingLog
  2. HDFSMetadataLog
  3. Logging
  4. MetadataLog
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new SchemaTrackingLog(sparkSession: SparkSession, path: String, schemaSerializer: SchemaSerializer[T])(implicit arg0: ClassTag[T], arg1: Manifest[T])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def add(batchId: Long, metadata: T): Boolean
    Definition Classes
    HDFSMetadataLog → MetadataLog
  5. def addNewBatchByStream(batchId: Long)(fn: (OutputStream) => Unit): Boolean
    Definition Classes
    HDFSMetadataLog
  6. def addSchemaToLog(newSchema: T): T

    Main API to actually write the log entry to the schema log.

    Main API to actually write the log entry to the schema log. Clients can leverage this to save their new schema to the log.

    newSchema

    New persisted schema

    Exceptions thrown
  7. def applyFnToBatchByStream[RET](batchId: Long, skipExistingCheck: Boolean)(fn: (InputStream) => RET): RET
    Definition Classes
    HDFSMetadataLog
  8. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  9. val batchCache: Map[Long, T]
    Attributes
    protected[sql]
    Definition Classes
    HDFSMetadataLog
  10. val batchFilesFilter: PathFilter
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  11. def batchIdToPath(batchId: Long): Path
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  12. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  13. def deserialize(in: InputStream): T

    Deserializes the log entry from input stream.

    Deserializes the log entry from input stream.

    Attributes
    protected
    Definition Classes
    SchemaTrackingLog → HDFSMetadataLog
    Exceptions thrown
  14. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  16. val fileManager: CheckpointFileManager
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  17. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  18. def get(startId: Option[Long], endId: Option[Long]): Array[(Long, T)]
    Definition Classes
    HDFSMetadataLog → MetadataLog
  19. def get(batchId: Long): Option[T]
    Definition Classes
    HDFSMetadataLog → MetadataLog
  20. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  21. def getCurrentTrackedSchema: Option[T]

    Get the latest tracked schema entry by this schema log

  22. def getCurrentTrackedSeqNum: Long

    Get the latest tracked schema batch ID / seq num by this log

  23. def getLatest(): Option[(Long, T)]
    Definition Classes
    HDFSMetadataLog → MetadataLog
  24. def getLatestBatchId(): Option[Long]
    Definition Classes
    HDFSMetadataLog
  25. def getOrderedBatchFiles(): Array[FileStatus]
    Definition Classes
    HDFSMetadataLog
  26. def getPrevBatchFromStorage(batchId: Long): Option[Long]
    Definition Classes
    HDFSMetadataLog
  27. def getTrackedSchemaAtSeqNum(seqNum: Long): Option[T]

    Get the tracked schema at specified seq num.

  28. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  29. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  30. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def isBatchFile(path: Path): Boolean
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  32. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  33. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  34. def listBatches: Array[Long]
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  35. def listBatchesOnDisk: Array[Long]
    Definition Classes
    HDFSMetadataLog
  36. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  37. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  44. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. val metadataCacheEnabled: Boolean
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  49. val metadataPath: Path
    Definition Classes
    HDFSMetadataLog
  50. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  51. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  52. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  53. def pathToBatchId(path: Path): Long
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  54. def purge(thresholdBatchId: Long): Unit
    Definition Classes
    HDFSMetadataLog → MetadataLog
  55. def purgeAfter(thresholdBatchId: Long): Unit
    Definition Classes
    HDFSMetadataLog
  56. def serialize(metadata: T, out: OutputStream): Unit
    Attributes
    protected
    Definition Classes
    SchemaTrackingLog → HDFSMetadataLog
  57. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  58. def toString(): String
    Definition Classes
    AnyRef → Any
  59. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  60. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  61. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  62. def write(batchMetadataFile: Path, fn: (OutputStream) => Unit): Unit
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog

Inherited from HDFSMetadataLog[T]

Inherited from Logging

Inherited from MetadataLog[T]

Inherited from AnyRef

Inherited from Any

Ungrouped