Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  def apply(zOrderBy: Seq[String]): ZCubeInfo

   Preferred way of creating a ZCubeInfo for a new ZCube.

   Preferred way of creating a ZCubeInfo for a new ZCube. Automatically generates a unique zCubeID.

5.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

6.  def buildBaseRelation(spark: SparkSession, txn: OptimisticTransaction, actionType: String, rootPath: Path, inputLeafFiles: Seq[String], nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation

   Build a base relation of files that need to be rewritten as part of an update/delete/merge operation.

   Build a base relation of files that need to be rewritten as part of an update/delete/merge operation.

   Attributes

   protected

   Definition Classes

   DeltaCommand

7.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... ) @native()

8.  def createSetTransaction(sparkSession: SparkSession, deltaLog: DeltaLog, options: Option[DeltaOptions] = None): Option[SetTransaction]

   Returns SetTransaction if a valid app ID and version are present.

   Returns SetTransaction if a valid app ID and version are present. Otherwise returns an empty list.

   Attributes

   protected

   Definition Classes

   DeltaCommand

9.  def deltaAssert(check: ⇒ Boolean, name: String, msg: String, deltaLog: DeltaLog = null, data: AnyRef = null, path: Option[Path] = None): Unit

   Helper method to check invariants in Delta code.

   Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.

   Attributes

   protected

   Definition Classes

   DeltaLogging

10.  final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

11.  def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

12.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

13.  def fromAddFileTags(tags: Map[String, String]): Option[ZCubeInfo]

    Deserializes a ZCubeInfo object from an AddFile.tags map, if present.

14.  def generateCandidateFileMap(basePath: Path, candidateFiles: Seq[AddFile]): Map[String, AddFile]

    Generates a map of file names to add file entries for operations where we will need to rewrite files such as delete, merge, update.

    Generates a map of file names to add file entries for operations where we will need to rewrite files such as delete, merge, update. We expect file names to be unique, because each file contains a UUID.

    Definition Classes

    DeltaCommand

15.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

16.  def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]

    Definition Classes

    DeltaLogging

17.  def getDeltaLog(spark: SparkSession, path: Option[String], tableIdentifier: Option[TableIdentifier], operationName: String, hadoopConf: Map[String, String] = Map.empty): DeltaLog

    Utility method to return the DeltaLog of an existing Delta table referred by either the given path or tableIdentifier.

    Utility method to return the DeltaLog of an existing Delta table referred by either the given path or tableIdentifier.

    spark

    SparkSession reference to use.

    path

    Table location. Expects a non-empty tableIdentifier or path.

    tableIdentifier

    Table identifier. Expects a non-empty tableIdentifier or path.

    operationName

    Operation that is getting the DeltaLog, used in error messages.

    hadoopConf

    Hadoop file system options used to build DeltaLog.

    returns

    DeltaLog of the table

    Attributes

    protected

    Definition Classes

    DeltaCommand

    Exceptions thrown

    AnalysisException If either no Delta table exists at the given path/identifier or there is neither path nor tableIdentifier is provided.

18.  def getDeltaTable(target: LogicalPlan, cmd: String): DeltaTableV2

    Extracts the DeltaTableV2 from a LogicalPlan iff the LogicalPlan is a ResolvedTable with either a DeltaTableV2 or a V1Table that is referencing a Delta table.

    Extracts the DeltaTableV2 from a LogicalPlan iff the LogicalPlan is a ResolvedTable with either a DeltaTableV2 or a V1Table that is referencing a Delta table. In all other cases this method will throw a "Table not found" exception.

    Definition Classes

    DeltaCommand

19.  def getDeltaTablePathOrIdentifier(target: LogicalPlan, cmd: String): (Option[TableIdentifier], Option[String])

    Helper method to extract the table id or path from a LogicalPlan representing a Delta table.

    Helper method to extract the table id or path from a LogicalPlan representing a Delta table. This uses DeltaCommand.getDeltaTable to convert the LogicalPlan to a DeltaTableV2 and then extracts either the path or identifier from it. If the DeltaTableV2 has a CatalogTable, the table identifier will be returned. Otherwise, the table's path will be returned. Throws an exception if the LogicalPlan does not represent a Delta table.

    Definition Classes

    DeltaCommand

20.  def getErrorData(e: Throwable): Map[String, Any]

    Definition Classes

    DeltaLogging

21.  def getForFile(file: AddFile): Option[ZCubeInfo]

    If the given file was written by an OPTIMIZE ZORDER BY job, return the corresponding ZCubeInfo.

    If the given file was written by an OPTIMIZE ZORDER BY job, return the corresponding ZCubeInfo. Otherwise return None.

22.  def getTableCatalogTable(target: LogicalPlan, cmd: String): Option[CatalogTable]

    Extracts CatalogTable metadata from a LogicalPlan if the plan is a ResolvedTable.

    Extracts CatalogTable metadata from a LogicalPlan if the plan is a ResolvedTable. The table can be a non delta table.

    Definition Classes

    DeltaCommand

23.  def getTablePathOrIdentifier(target: LogicalPlan, cmd: String): (Option[TableIdentifier], Option[String])

    Helper method to extract the table id or path from a LogicalPlan representing a resolved table or path.

    Helper method to extract the table id or path from a LogicalPlan representing a resolved table or path. This calls getDeltaTablePathOrIdentifier if the resolved table is a delta table. For non delta table with identifier, we extract its identifier. For non delta table with path, it expects the path to be wrapped in an ResolvedPathBasedNonDeltaTable and extracts it from there.

    Definition Classes

    DeltaCommand

24.  def getTouchedFile(basePath: Path, escapedFilePath: String, nameToAddFileMap: Map[String, AddFile]): AddFile

    Find the AddFile record corresponding to the file that was read as part of a delete/update/merge operation.

    Find the AddFile record corresponding to the file that was read as part of a delete/update/merge operation.

    basePath

    The path of the table. Must not be escaped.

    escapedFilePath

    The path to a file that can be either absolute or relative. All special chars in this path must be already escaped by URI standards.

    nameToAddFileMap

    Map generated through generateCandidateFileMap().

    Definition Classes

    DeltaCommand

25.  def hasBeenExecuted(txn: OptimisticTransaction, sparkSession: SparkSession, options: Option[DeltaOptions] = None): Boolean

    Returns true if there is information in the spark session that indicates that this write has already been successfully written.

    Returns true if there is information in the spark session that indicates that this write has already been successfully written.

    Attributes

    protected

    Definition Classes

    DeltaCommand

26.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

27.  def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Attributes

    protected

    Definition Classes

    Logging

28.  def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes

    protected

    Definition Classes

    Logging

29.  def isCatalogTable(analyzer: Analyzer, tableIdent: TableIdentifier): Boolean

    Use the analyzer to see whether the provided TableIdentifier is for a path based table or not

    Use the analyzer to see whether the provided TableIdentifier is for a path based table or not

    analyzer

    The session state analyzer to call

    tableIdent

    Table Identifier to determine whether is path based or not

    returns

    Boolean where true means that the table is a table in a metastore and false means the table is a path based table

    Definition Classes

    DeltaCommand

30.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

31.  def isPathIdentifier(tableIdent: TableIdentifier): Boolean

    Checks if the given identifier can be for a delta table's path

    Checks if the given identifier can be for a delta table's path

    tableIdent

    Table Identifier for which to check

    Attributes

    protected

    Definition Classes

    DeltaCommand

32.  def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

33.  def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

34.  def logConsole(line: String): Unit

    Definition Classes

    DatabricksLogging

35.  def logDebug(entry: LogEntry, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

36.  def logDebug(entry: LogEntry): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

37.  def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

38.  def logDebug(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

39.  def logError(entry: LogEntry, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

40.  def logError(entry: LogEntry): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

41.  def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

42.  def logError(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

43.  def logInfo(entry: LogEntry, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

44.  def logInfo(entry: LogEntry): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

45.  def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

46.  def logInfo(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

47.  def logName: String

    Attributes

    protected

    Definition Classes

    Logging

48.  def logTrace(entry: LogEntry, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

49.  def logTrace(entry: LogEntry): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

50.  def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

51.  def logTrace(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

52.  def logWarning(entry: LogEntry, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

53.  def logWarning(entry: LogEntry): Unit

    Attributes

    protected

    Definition Classes

    LoggingShims

54.  def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

55.  def logWarning(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

56.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

57.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

58.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

59.  def parsePredicates(spark: SparkSession, predicate: String): Seq[Expression]

    Converts string predicates into Expressions relative to a transaction.

    Converts string predicates into Expressions relative to a transaction.

    Attributes

    protected

    Definition Classes

    DeltaCommand

    Exceptions thrown

    AnalysisException if a non-partition column is referenced.

60.  def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes

    protected

    Definition Classes

    DeltaLogging

61.  def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes

    protected

    Definition Classes

    DeltaLogging

62.  def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes

    protected

    Definition Classes

    DeltaLogging

63.  def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

64.  def recordFrameProfile[T](group: String, name: String)(thunk: ⇒ T): T

    Attributes

    protected

    Definition Classes

    DeltaLogging

65.  def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: ⇒ S): S

    Definition Classes

    DatabricksLogging

66.  def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

67.  def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

68.  def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

69.  def removeFilesFromPaths(deltaLog: DeltaLog, nameToAddFileMap: Map[String, AddFile], filesToRewrite: Seq[String], operationTimestamp: Long): Seq[RemoveFile]

    This method provides the RemoveFile actions that are necessary for files that are touched and need to be rewritten in methods like Delete, Update, and Merge.

    This method provides the RemoveFile actions that are necessary for files that are touched and need to be rewritten in methods like Delete, Update, and Merge.

    deltaLog

    The DeltaLog of the table that is being operated on

    nameToAddFileMap

    A map generated using generateCandidateFileMap.

    filesToRewrite

    Absolute paths of the files that were touched. We will search for these in candidateFiles. Obtained as the output of the input_file_name function.

    operationTimestamp

    The timestamp of the operation

    Attributes

    protected

    Definition Classes

    DeltaCommand

70.  def resolveIdentifier(analyzer: Analyzer, identifier: TableIdentifier): LogicalPlan

    Use the analyzer to resolve the identifier provided

    Use the analyzer to resolve the identifier provided

    analyzer

    The session state analyzer to call

    identifier

    Table Identifier to determine whether is path based or not

    Attributes

    protected

    Definition Classes

    DeltaCommand

71.  def sendDriverMetrics(spark: SparkSession, metrics: Map[String, SQLMetric]): Unit

    Send the driver-side metrics.

    Send the driver-side metrics.

    This is needed to make the SQL metrics visible in the Spark UI. All metrics are default initialized with 0 so that's what we're reporting in case we skip an already executed action.

    Attributes

    protected

    Definition Classes

    DeltaCommand

72.  def setForFile(file: AddFile, zCubeInfo: ZCubeInfo): AddFile

    Update the given file's metadata to make it part of the given zCubeInfo.

73.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

74.  def toAddFileTags(zCubeInfo: ZCubeInfo): Map[String, String]

    Serializes the given zCubeInfo to a Map[String, String] that can be used as or merged into AddFile.tags.

75.  def toString(): String

    Definition Classes

    AnyRef → Any

76.  def unsetForFile(file: AddFile): AddFile

    Clears the ZCubeInfo metadata of the given file to make it appear as unoptimized.

77.  def verifyPartitionPredicates(spark: SparkSession, partitionColumns: Seq[String], predicates: Seq[Expression]): Unit

    Definition Classes

    DeltaCommand

78.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

79.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

80.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()

81.  def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: ⇒ T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes

    DeltaProgressReporter