An internal latestOffsetInternal to get the latest offset.

Flag that allows user to force enable unsafe streaming read on Delta table with column mapping enabled AND drop/rename actions.

Check read-incompatible schema changes during stream (re)start so we could fail fast.

This only needs to be called ONCE in the life cycle of a stream, either at the very first latestOffset, or the very first getBatch to make sure we have detected an incompatible schema change. Typically, the verifyStreamHygiene that was called maybe good enough to detect these schema changes, there may be cases that wouldn't work, e.g. consider this sequence: 1. User starts a new stream @ startingVersion 1 2. latestOffset is called before getBatch() because there was no previous commits so getBatch won't be called as a recovery mechanism. Suppose there's a single rename/drop/nullability change S during computing next offset, S would look exactly the same as the latest schema so verifyStreamHygiene would not work. 3. latestOffset would return this new offset cross the schema boundary.

If a schema log is already initialized, we don't have to run the initialization nor schema checks any more.

Narrow waist to verify a metadata action for read-incompatible schema changes, specifically: 1. Any column mapping related schema changes (rename / drop) columns 2. Standard read-compatibility changes including: a) No missing columns b) No data type changes c) No read-incompatible nullability changes If the check fails, we throw an exception to exit the stream. If lazy log initialization is required, we also run a one time scan to safely initialize the metadata tracking log upon any non-additive schema change failures.

Given an iterator of file actions, create a DataFrame representing the files added to a table Only AddFile actions will be used to create the DataFrame.

Return the DataFrame between start and end offset.

Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.

Flag that allows user to disable the read-compatibility check during stream start which protects against an corner case in which verifyStreamHygiene could not detect. This is a bug fix but yet a potential behavior change, so we add a flag to fallback.

Flag that allow user to fallback to the legacy behavior in which user can allow nullable=false schema to read nullable=true data, which is incorrect but a behavior change regardless.

get the changes from startVersion, startIndex to the end

If the current stream metadata is not equal to the metadata change in metadataChangeOpt, return a metadata change barrier IndexedFile. Only returns something if trackingMetadataChangeis true.

Return the next offset when previous offset exists.

If the given previous Delta source offset is a schema change offset, returns the appropriate next offset. This should be called before trying any other means of determining the next offset. If this returns None, then there is no schema change, and the caller should determine the next offset in the normal way.

Returns the offset that starts from a specific delta table version. This function is called when starting a new stream query.

A global flag to mark whether we have done a per-stream start check for column mapping schema changes (rename / drop).

initialize the internal states for AvailableNow if this method is called first time after prepareForTriggerAvailableNow.

Initialize the schema tracking log if an empty schema tracking log is provided. This method also checks the range between batchStartVersion and batchEndVersion to ensure we a safe schema to be initialized in the log.

Whether we are streaming from a table with column mapping enabled

When AvailableNow is used, this offset will be the upper bound where this run of the query will process up. We may run multiple micro batches, but the query will stop itself when it reaches this offset.

The persisted schema from the schema log that must be used to read data files in this Delta streaming source.

The read schema for this source during initialization, taking in account of SchemaLog.

Create a snapshot descriptor, customizing its metadata using metadata tracking if necessary

Whether a schema tracking log is provided (and is empty), so we could initialize eagerly. This should only be used for the first write to the schema log, after then, schema tracking should not rely on this state any more.

Used to record the occurrence of a single event or report detailed, operation specific statistics.

Used to report the duration as well as the success or failure of an operation on a deltaLog.

Used to report the duration as well as the success or failure of an operation on a tahoePath.

This is called from getFileChangesWithRateLimit() during latestOffset().

Whether this DeltaSource is utilizing a schema log entry as its read schema.

If user explicitly turn on the flag to fall back to using latest schema to read (i.e. the legacy mode), we will ignore the schema log.

Write a new potentially changed metadata into the metadata tracking log. Then fail the stream to allow reanalysis if there are changes.

Update the current stream schema in the schema tracking log and fail the stream. This is called during commit(). It's ok to fail during commit() because in streaming's semantics, the batch with offset ending at end should've already being processed completely.

Report a log to indicate some command is running.