Packages

c

org.apache.spark.sql.delta.streaming

SchemaTrackingLog

class SchemaTrackingLog[T <: PartitionAndDataSchema] extends HDFSMetadataLog[T] with LoggingShims

The underlying class for a streaming log that keeps track of a sequence of schema changes.

It keeps tracks of the sequence of schema changes that this log is aware of, and it detects any concurrent modifications to the schema log to prevent accidents on a best effort basis.

Linear Supertypes
LoggingShims, HDFSMetadataLog[T], Logging, MetadataLog[T], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SchemaTrackingLog
  2. LoggingShims
  3. HDFSMetadataLog
  4. Logging
  5. MetadataLog
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SchemaTrackingLog(sparkSession: SparkSession, path: String, schemaSerializer: SchemaSerializer[T])(implicit arg0: ClassTag[T], arg1: Manifest[T])

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    LoggingShims

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def add(batchId: Long, metadata: T): Boolean
    Definition Classes
    HDFSMetadataLog → MetadataLog
  5. def addNewBatchByStream(batchId: Long)(fn: (OutputStream) ⇒ Unit): Boolean
    Definition Classes
    HDFSMetadataLog
  6. def addSchemaToLog(newSchema: T): T

    Main API to actually write the log entry to the schema log.

    Main API to actually write the log entry to the schema log. Clients can leverage this to save their new schema to the log.

    newSchema

    New persisted schema

    Exceptions thrown
  7. def applyFnToBatchByStream[RET](batchId: Long, skipExistingCheck: Boolean)(fn: (InputStream) ⇒ RET): RET
    Definition Classes
    HDFSMetadataLog
  8. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  9. val batchCache: Map[Long, T]
    Attributes
    protected[sql]
    Definition Classes
    HDFSMetadataLog
  10. val batchFilesFilter: PathFilter
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  11. def batchIdToPath(batchId: Long): Path
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  12. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  13. def deserialize(in: InputStream): T

    Deserializes the log entry from input stream.

    Deserializes the log entry from input stream.

    Attributes
    protected
    Definition Classes
    SchemaTrackingLog → HDFSMetadataLog
    Exceptions thrown
  14. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  16. val fileManager: CheckpointFileManager
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  17. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  18. def get(startId: Option[Long], endId: Option[Long]): Array[(Long, T)]
    Definition Classes
    HDFSMetadataLog → MetadataLog
  19. def get(batchId: Long): Option[T]
    Definition Classes
    HDFSMetadataLog → MetadataLog
  20. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  21. def getCurrentTrackedSchema: Option[T]

    Get the latest tracked schema entry by this schema log

  22. def getCurrentTrackedSeqNum: Long

    Get the latest tracked schema batch ID / seq num by this log

  23. def getLatest(): Option[(Long, T)]
    Definition Classes
    HDFSMetadataLog → MetadataLog
  24. def getLatestBatchId(): Option[Long]
    Definition Classes
    HDFSMetadataLog
  25. def getOrderedBatchFiles(): Array[FileStatus]
    Definition Classes
    HDFSMetadataLog
  26. def getPrevBatchFromStorage(batchId: Long): Option[Long]
    Definition Classes
    HDFSMetadataLog
  27. def getTrackedSchemaAtSeqNum(seqNum: Long): Option[T]

    Get the tracked schema at specified seq num.

  28. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  29. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  30. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def isBatchFile(path: Path): Boolean
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  32. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  33. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  34. def listBatches: Array[Long]
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  35. def listBatchesOnDisk: Array[Long]
    Definition Classes
    HDFSMetadataLog
  36. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  37. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  38. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  39. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  42. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  43. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  46. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  47. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  49. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  50. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  51. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  52. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  54. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  55. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  56. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  57. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  58. val metadataCacheEnabled: Boolean
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  59. val metadataPath: Path
    Definition Classes
    HDFSMetadataLog
  60. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  61. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  62. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  63. def pathToBatchId(path: Path): Long
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  64. def purge(thresholdBatchId: Long): Unit
    Definition Classes
    HDFSMetadataLog → MetadataLog
  65. def purgeAfter(thresholdBatchId: Long): Unit
    Definition Classes
    HDFSMetadataLog
  66. def serialize(metadata: T, out: OutputStream): Unit
    Attributes
    protected
    Definition Classes
    SchemaTrackingLog → HDFSMetadataLog
  67. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  68. def toString(): String
    Definition Classes
    AnyRef → Any
  69. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  70. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  71. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  72. def write(batchMetadataFile: Path, fn: (OutputStream) ⇒ Unit): Unit
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog

Inherited from LoggingShims

Inherited from HDFSMetadataLog[T]

Inherited from Logging

Inherited from MetadataLog[T]

Inherited from AnyRef

Inherited from Any

Ungrouped