Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##: Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  val LAST_CHECKPOINT_FILE_NAME: String

   The name of the last checkpoint file

5.  val STRUCT_PARTITIONS_COL_NAME: String
6.  val STRUCT_STATS_COL_NAME: String
7.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

8.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws(classOf[java.lang.CloneNotSupportedException]) @native()

9.  def createCheckpointV2ParquetFile(spark: SparkSession, ds: Dataset[Row], finalPath: Path, hadoopConf: Configuration, useRename: Boolean): StructType

   Helper method to create a V2 Checkpoint parquet file or the V2 Checkpoint Compat file.

   Helper method to create a V2 Checkpoint parquet file or the V2 Checkpoint Compat file. V2 Checkpoint Compat files follow the same naming convention as classic checkpoints and they are needed so that V2Checkpoint-unaware readers can read them to understand that they don't have the capability to read table for which they were created. This is needed in cases where commit 0 has been cleaned up and the reader needs to read a checkpoint to read the Protocol.

10.  def deltaAssert(check: => Boolean, name: String, msg: String, deltaLog: DeltaLog = null, data: AnyRef = null, path: Option[Path] = None): Unit

    Helper method to check invariants in Delta code.

    Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.

    Attributes

    protected

    Definition Classes

    DeltaLogging

11.  final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

12.  def equals(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef → Any

13.  def extractPartitionValues(partitionSchema: StructType, partitionValuesColName: String): Option[Column]

    Creates a nested struct column of partition values that extract the partition values from the original MapType.

14.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.Throwable])

15.  def getCheckpointWritePath(conf: Configuration, logPath: Path, version: Long, numParts: Int, part: Int, useRename: Boolean, v2CheckpointEnabled: Boolean): (Path, Path)

    Generate a tuple of the file to write the checkpoint and where it may later need to be copied.

    Generate a tuple of the file to write the checkpoint and where it may later need to be copied. Should be used within a task, so that task or stage retries don't create the same files.

16.  final def getClass(): Class[_ <: AnyRef]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

17.  def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]

    Definition Classes

    DeltaLogging

18.  def getErrorData(e: Throwable): Map[String, Any]

    Definition Classes

    DeltaLogging

19.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

20.  def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Attributes

    protected

    Definition Classes

    Logging

21.  def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes

    protected

    Definition Classes

    Logging

22.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

23.  def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

24.  def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

25.  def logConsole(line: String): Unit

    Definition Classes

    DatabricksLogging

26.  def logDebug(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

27.  def logDebug(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

28.  def logError(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

29.  def logError(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

30.  def logInfo(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

31.  def logInfo(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

32.  def logName: String

    Attributes

    protected

    Definition Classes

    Logging

33.  def logTrace(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

34.  def logTrace(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

35.  def logWarning(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

36.  def logWarning(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

37.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

38.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

39.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

40.  def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes

    protected

    Definition Classes

    DeltaLogging

41.  def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes

    protected

    Definition Classes

    DeltaLogging

42.  def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes

    protected

    Definition Classes

    DeltaLogging

43.  def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

44.  def recordFrameProfile[T](group: String, name: String)(thunk: => T): T

    Attributes

    protected

    Definition Classes

    DeltaLogging

45.  def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: => S): S

    Definition Classes

    DatabricksLogging

46.  def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

47.  def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

48.  def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

49.  def shouldWriteStatsAsJson(snapshot: Snapshot): Boolean
50.  def shouldWriteStatsAsStruct(conf: SQLConf, snapshot: Snapshot): Boolean
51.  final def synchronized[T0](arg0: => T0): T0

    Definition Classes

    AnyRef

52.  def toString(): String

    Definition Classes

    AnyRef → Any

53.  def trimLastCheckpointV2(lastCheckpointV2: LastCheckpointV2, spark: SparkSession): LastCheckpointV2

    Bounds the size of a LastCheckpointV2 by removing any oversized optional fields

54.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

55.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

56.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException]) @native()

57.  def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes

    DeltaProgressReporter

58.  def writeTopLevelV2Checkpoint(v2CheckpointFormat: Format, sidecarCheckpointFiles: Array[SerializableFileStatus], spark: SparkSession, sidecarSchema: StructType, snapshot: Snapshot, deltaLog: DeltaLog, rowsWrittenInCheckpointJob: Long, parquetFilesSizeInBytes: Long, hadoopConf: Configuration, useRename: Boolean): (FileStatus, Seq[Action], LastCheckpointV2, Option[StructType])

    Writes a top-level V2 Checkpoint file which may point to multiple sidecar files.

    Writes a top-level V2 Checkpoint file which may point to multiple sidecar files.

    v2CheckpointFormat

    The format in which the top-level file should be written. Currently, json and parquet are supported.

    sidecarCheckpointFiles

    The list of sidecar files that have already been written. The top-level file will store this list.

    spark

    The current spark session

    sidecarSchema

    The schema of the sidecar parquet files.

    snapshot

    The snapshot for which the checkpoint is being written.

    deltaLog

    The deltaLog instance pointing to our tables deltaLog.

    rowsWrittenInCheckpointJob

    The number of rows that were written in total to the sidecar files.

    parquetFilesSizeInBytes

    The combined size of all sidecar files in bytes.

    hadoopConf

    The hadoopConf to use for the filesystem operation.

    useRename

    Whether we should first write to a temporary file and then rename it to the target file name during the write.

    returns

    A tuple containing

    1. FileStatus of the newly created top-level V2Checkpoint. 2. The sequence of actions that were written to the top-level file. 3. An instance of the LastCheckpointV2 containing V2-checkpoint related metadata which can later be written to LAST_CHECKPOINT 4. Schema of the newly written top-level file (only for parquet files)

    Attributes

    protected[delta]