Packages

class Snapshot extends StateCache with PartitionFiltering with DeltaFileFormat with DeltaLogging

An immutable snapshot of the state of the log at some delta version. Internally this class manages the replay of actions stored in checkpoint or delta files.

After resolving any new actions, it caches the result and collects the following basic information to the driver:

  • Protocol Version
  • Metadata
  • Transaction state
Linear Supertypes
DeltaLogging, DatabricksLogging, DeltaProgressReporter, Logging, DeltaFileFormat, PartitionFiltering, StateCache, AnyRef, Any
Known Subclasses
InitialSnapshot
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Snapshot
  2. DeltaLogging
  3. DatabricksLogging
  4. DeltaProgressReporter
  5. Logging
  6. DeltaFileFormat
  7. PartitionFiltering
  8. StateCache
  9. AnyRef
  10. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Snapshot(path: Path, version: Long, logSegment: LogSegment, minFileRetentionTimestamp: Long, deltaLog: DeltaLog, timestamp: Long, checksumOpt: Option[VersionChecksum])

    timestamp

    The timestamp of the latest commit in milliseconds. Can also be set to -1 if the timestamp of the commit is unknown or the table has not been initialized, i.e. version = -1.

Type Members

  1. class CachedDS[A] extends AnyRef
    Definition Classes
    StateCache

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def allFiles: Dataset[AddFile]

    All of the files present in this Snapshot.

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def cacheDS[A](ds: Dataset[A], name: String): CachedDS[A]

    Create a CachedDS instance for the given Dataset and the name.

    Create a CachedDS instance for the given Dataset and the name.

    Definition Classes
    StateCache
  7. lazy val checkpointFileIndexOpt: Option[DeltaLogFileIndex]
    Attributes
    protected
  8. val checksumOpt: Option[VersionChecksum]
  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  10. lazy val computedState: State

    Computes some statistics around the transaction log, therefore on the actions made on this Delta table.

    Computes some statistics around the transaction log, therefore on the actions made on this Delta table.

    Attributes
    protected
  11. def dataSchema: StructType

    Returns the data schema of the table, the schema of the columns written out to file.

  12. lazy val deltaFileIndexOpt: Option[DeltaLogFileIndex]
    Attributes
    protected
  13. val deltaLog: DeltaLog
  14. def emptyActions: Dataset[SingleAction]
    Attributes
    protected
  15. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  17. def fileFormat: FileFormat

    Return the underlying Spark FileFormat of the Delta table.

    Return the underlying Spark FileFormat of the Delta table.

    Definition Classes
    DeltaFileFormat
  18. lazy val fileIndices: Seq[DeltaLogFileIndex]
    Attributes
    protected
  19. def filesForScan(projection: Seq[Attribute], filters: Seq[Expression], keepStats: Boolean = false): DeltaScan
    Definition Classes
    PartitionFiltering
  20. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  21. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  22. def getNumPartitions: Int
    Attributes
    protected
  23. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  24. def init(): Unit

    Performs validations during initialization

    Performs validations during initialization

    Attributes
    protected
  25. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  26. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  28. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  29. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  30. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  31. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Definition Classes
    Snapshot → Logging
  34. def logError(msg: ⇒ String): Unit
    Definition Classes
    Snapshot → Logging
  35. def logInfo(msg: ⇒ String): Unit
    Definition Classes
    Snapshot → Logging
  36. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  38. val logSegment: LogSegment
  39. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Definition Classes
    Snapshot → Logging
  42. def logWarning(msg: ⇒ String): Unit
    Definition Classes
    Snapshot → Logging
  43. def metadata: Metadata
  44. val minFileRetentionTimestamp: Long
  45. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  46. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  47. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  48. lazy val numIndexedCols: Int

    Number of columns to collect stats on for data skipping

  49. def numOfFiles: Long
  50. def numOfMetadata: Long
  51. def numOfProtocol: Long
  52. def numOfRemoves: Long
  53. def numOfSetTransactions: Long
  54. val path: Path
  55. def protocol: Protocol
  56. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  57. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

    Used to report the duration as well as the success or failure of an operation.

    Used to report the duration as well as the success or failure of an operation.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  58. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  59. def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: ⇒ S): S
    Definition Classes
    DatabricksLogging
  60. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  61. def redactedPath: String
  62. def schema: StructType

    Returns the schema of the table.

  63. def setTransactions: Seq[SetTransaction]
  64. def sizeInBytes: Long
  65. def spark: SparkSession
    Attributes
    protected
    Definition Classes
    Snapshot → StateCache
  66. def state: Dataset[SingleAction]

    The current set of actions in this Snapshot.

  67. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  68. val timestamp: Long
  69. def toString(): String
    Definition Classes
    Snapshot → AnyRef → Any
  70. def tombstones: Dataset[RemoveFile]

    All unexpired tombstones.

  71. lazy val transactions: Map[String, Long]

    A map to look up transaction version by appId.

  72. def uncache(): Unit

    Drop any cached data for this Snapshot.

    Drop any cached data for this Snapshot.

    Definition Classes
    StateCache
  73. val version: Long
  74. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  75. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  76. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  77. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: ⇒ T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes
    DeltaProgressReporter

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from Logging

Inherited from DeltaFileFormat

Inherited from PartitionFiltering

Inherited from StateCache

Inherited from AnyRef

Inherited from Any

Ungrouped