Type Members

1.  trait BackfillBatch extends DeltaLogging
2.  class BackfillBatchIterator[BackfillBatchType] extends Iterator[BackfillBatchType] with Closeable

   Construct a lazily evaluated iterator for getting BackfillBatchType.

   Construct a lazily evaluated iterator for getting BackfillBatchType. Callers should close this iterator to avoid resource leak.

3.  case class BackfillBatchStats(parentTransactionId: String, transactionId: String, batchId: Int, initialNumFiles: Long, totalExecutionTimeInMs: Long, wasSuccessful: Boolean) extends Product with Serializable

   Metrics for each BackfillBatch.

   Metrics for each BackfillBatch.

   parentTransactionId

   The transaction id associated with the parent command.

   transactionId

   The transaction id used in this batch.

   batchId

   An integer identifier of the batch within a parent BackfillCommand.

   initialNumFiles

   The number of files in BackfillBatch prior to conflict resolution.

   totalExecutionTimeInMs

   The total execution time in milliseconds.

   wasSuccessful

   Boolean indicating whether the batch was successfully committed.

4.  trait BackfillCommand extends LogicalPlan with LeafRunnableCommand with DeltaCommand

   This command will lazily materialize AllFiles and split them into multiple backfill commits if the number of files exceeds the threshold set by DeltaSQLConf.DELTA_BACKFILL_MAX_NUM_FILES_PER_COMMIT.

5.  case class BackfillCommandStats(transactionId: String, nameOfTriggeringOperation: String, maxNumBatchesInParallel: Int, totalExecutionTimeMs: Long = 0, numSuccessfulBatches: Int = 0, numFailedBatches: Int = 0, wasSuccessful: Boolean = false) extends Product with Serializable

   Metrics for the BackfillCommand.

   Metrics for the BackfillCommand.

   maxNumBatchesInParallel

   The maximum number of batches that could have run in parallel.

   totalExecutionTimeMs

   The total execution time in milliseconds.

   numSuccessfulBatches

   The number of BackfillBatch's that was successfully committed.

   numFailedBatches

   The number of BackfillBatch's that failed.

   wasSuccessful

   Boolean indicating whether this BackfillCommand didn't have any error.

6.  trait BackfillExecutor extends DeltaLogging
7.  case class RowTrackingBackfillBatch(filesInBatch: Seq[AddFile]) extends BackfillBatch with Product with Serializable
8.  case class RowTrackingBackfillCommand(deltaLog: DeltaLog, nameOfTriggeringOperation: String, catalogTable: Option[CatalogTable]) extends LogicalPlan with BackfillCommand with Product with Serializable

   This command re-commits all AddFiles in the current snapshot that do not have a base row IDs.

   This command re-commits all AddFiles in the current snapshot that do not have a base row IDs. After the backfill command finishes, the snapshot has row IDs for all files. All commits afterwards must include row IDs.

   First, we will add the table feature support, if necessary. Then, the command will lazily materialize AllFiles and split them into multiple backfill commits if the number of files exceeds the threshold set by DeltaSQLConf.DELTA_BACKFILL_MAX_NUM_FILES_PER_COMMIT.

   Note: We expect Backfill to be called before the table property is set. Furthermore, we do not set the table property DeltaConfigs.ROW_TRACKING_ENABLED as part of backfill. The metadata update needs to be handled by the caller.

9.  class RowTrackingBackfillExecutor extends BackfillExecutor