Packages

c

org.apache.spark.sql.delta

InitialSnapshot

class InitialSnapshot extends Snapshot

An initial snapshot with only metadata specified. Useful for creating a DataFrame from an existing parquet table during its conversion to delta.

Linear Supertypes
Snapshot, DeltaLogging, DatabricksLogging, DeltaProgressReporter, Logging, DeltaFileFormat, PartitionFiltering, StateCache, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. InitialSnapshot
  2. Snapshot
  3. DeltaLogging
  4. DatabricksLogging
  5. DeltaProgressReporter
  6. Logging
  7. DeltaFileFormat
  8. PartitionFiltering
  9. StateCache
  10. AnyRef
  11. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new InitialSnapshot(logPath: Path, deltaLog: DeltaLog, metadata: Metadata)

    logPath

    the path to transaction log

    deltaLog

    the delta log object

    metadata

    the metadata of the table

Type Members

  1. class CachedDS[A] extends AnyRef
    Definition Classes
    StateCache

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def allFiles: Dataset[AddFile]

    All of the files present in this Snapshot.

    All of the files present in this Snapshot.

    Definition Classes
    Snapshot
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def cacheDS[A](ds: Dataset[A], name: String): CachedDS[A]

    Create a CachedDS instance for the given Dataset and the name.

    Create a CachedDS instance for the given Dataset and the name.

    Definition Classes
    StateCache
  7. lazy val checkpointFileIndexOpt: Option[DeltaLogFileIndex]
    Attributes
    protected
    Definition Classes
    Snapshot
  8. val checksumOpt: Option[VersionChecksum]
    Definition Classes
    Snapshot
  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  10. lazy val computedState: State

    Computes some statistics around the transaction log, therefore on the actions made on this Delta table.

    Computes some statistics around the transaction log, therefore on the actions made on this Delta table.

    Attributes
    protected
    Definition Classes
    InitialSnapshot → Snapshot
  11. def dataSchema: StructType

    Returns the data schema of the table, the schema of the columns written out to file.

    Returns the data schema of the table, the schema of the columns written out to file.

    Definition Classes
    Snapshot
  12. lazy val deltaFileIndexOpt: Option[DeltaLogFileIndex]
    Attributes
    protected
    Definition Classes
    Snapshot
  13. val deltaLog: DeltaLog
    Definition Classes
    InitialSnapshot → Snapshot
  14. def emptyActions: Dataset[SingleAction]
    Attributes
    protected
    Definition Classes
    Snapshot
  15. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  17. def fileFormat: FileFormat

    Return the underlying Spark FileFormat of the Delta table.

    Return the underlying Spark FileFormat of the Delta table.

    Definition Classes
    DeltaFileFormat
  18. lazy val fileIndices: Seq[DeltaLogFileIndex]
    Attributes
    protected
    Definition Classes
    Snapshot
  19. def filesForScan(projection: Seq[Attribute], filters: Seq[Expression], keepStats: Boolean = false): DeltaScan
    Definition Classes
    PartitionFiltering
  20. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  21. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  22. def getNumPartitions: Int
    Attributes
    protected
    Definition Classes
    Snapshot
  23. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  24. def init(): Unit

    Performs validations during initialization

    Performs validations during initialization

    Attributes
    protected
    Definition Classes
    Snapshot
  25. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  26. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  28. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  29. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  30. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  31. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Definition Classes
    Snapshot → Logging
  34. def logError(msg: ⇒ String): Unit
    Definition Classes
    Snapshot → Logging
  35. def logInfo(msg: ⇒ String): Unit
    Definition Classes
    Snapshot → Logging
  36. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  38. val logPath: Path
  39. val logSegment: LogSegment
    Definition Classes
    Snapshot
  40. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Definition Classes
    Snapshot → Logging
  43. def logWarning(msg: ⇒ String): Unit
    Definition Classes
    Snapshot → Logging
  44. val metadata: Metadata
    Definition Classes
    InitialSnapshot → Snapshot
  45. val minFileRetentionTimestamp: Long
    Definition Classes
    Snapshot
  46. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  47. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  48. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  49. lazy val numIndexedCols: Int

    Number of columns to collect stats on for data skipping

    Number of columns to collect stats on for data skipping

    Definition Classes
    Snapshot
  50. def numOfFiles: Long
    Definition Classes
    Snapshot
  51. def numOfMetadata: Long
    Definition Classes
    Snapshot
  52. def numOfProtocol: Long
    Definition Classes
    Snapshot
  53. def numOfRemoves: Long
    Definition Classes
    Snapshot
  54. def numOfSetTransactions: Long
    Definition Classes
    Snapshot
  55. val path: Path
    Definition Classes
    Snapshot
  56. def protocol: Protocol
    Definition Classes
    Snapshot
  57. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  58. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

    Used to report the duration as well as the success or failure of an operation.

    Used to report the duration as well as the success or failure of an operation.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  59. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  60. def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: ⇒ S): S
    Definition Classes
    DatabricksLogging
  61. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  62. def redactedPath: String
    Definition Classes
    Snapshot
  63. def schema: StructType

    Returns the schema of the table.

    Returns the schema of the table.

    Definition Classes
    Snapshot
  64. def setTransactions: Seq[SetTransaction]
    Definition Classes
    Snapshot
  65. def sizeInBytes: Long
    Definition Classes
    Snapshot
  66. def spark: SparkSession
    Attributes
    protected
    Definition Classes
    Snapshot → StateCache
  67. def state: Dataset[SingleAction]

    The current set of actions in this Snapshot.

    The current set of actions in this Snapshot.

    Definition Classes
    InitialSnapshot → Snapshot
  68. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  69. val timestamp: Long
    Definition Classes
    Snapshot
  70. def toString(): String
    Definition Classes
    Snapshot → AnyRef → Any
  71. def tombstones: Dataset[RemoveFile]

    All unexpired tombstones.

    All unexpired tombstones.

    Definition Classes
    Snapshot
  72. lazy val transactions: Map[String, Long]

    A map to look up transaction version by appId.

    A map to look up transaction version by appId.

    Definition Classes
    Snapshot
  73. def uncache(): Unit

    Drop any cached data for this Snapshot.

    Drop any cached data for this Snapshot.

    Definition Classes
    StateCache
  74. val version: Long
    Definition Classes
    Snapshot
  75. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  76. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  77. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  78. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: ⇒ T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes
    DeltaProgressReporter

Inherited from Snapshot

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from Logging

Inherited from DeltaFileFormat

Inherited from PartitionFiltering

Inherited from StateCache

Inherited from AnyRef

Inherited from Any

Ungrouped