io.delta.kernel

Interface Scan

All Known Implementing Classes:

ScanImpl

public interface Scan

Represents a scan of a Delta table.

Method Summary

Modifier and Type	Method and Description
java.util.Optional<Expression>	getRemainingFilter() Get the remaining filter that is not guaranteed to be satisfied for the data Delta Kernel returns.
CloseableIterator<ColumnarBatch>	getScanFiles(TableClient tableClient) Get an iterator of data files to scan.
Row	getScanState(TableClient tableClient) Get the scan state associated with the current scan.
static CloseableIterator<DataReadResult>	readData(TableClient tableClient, Row scanState, CloseableIterator<Row> scanFileRowIter, java.util.Optional<Expression> filter) Get the data from the given scan files using the connector provided TableClient.

Method Detail

getScanFiles

CloseableIterator<ColumnarBatch> getScanFiles(TableClient tableClient)

Get an iterator of data files to scan.

Parameters:

tableClient - TableClient instance to use in Delta Kernel.

Returns:

iterator of ColumnarBatchs where each row in each batch corresponds to one scan file

getRemainingFilter

java.util.Optional<Expression> getRemainingFilter()

Get the remaining filter that is not guaranteed to be satisfied for the data Delta Kernel returns. This filter is used by Delta Kernel to do data skipping when possible.

Returns:

the remaining filter as an Expression.

getScanState

Row getScanState(TableClient tableClient)

Get the scan state associated with the current scan. This state is common across all files in the scan to be read.

Parameters:

tableClient - TableClient instance to use in Delta Kernel.

Returns:

Scan state in Row format.

readData

static CloseableIterator<DataReadResult> readData(TableClient tableClient,
                                                  Row scanState,
                                                  CloseableIterator<Row> scanFileRowIter,
                                                  java.util.Optional<Expression> filter)
                                           throws java.io.IOException

Get the data from the given scan files using the connector provided TableClient.

Parameters:

tableClient - Connector provided TableClient implementation.

scanState - Scan state returned by getScanState(TableClient)

scanFileRowIter - an iterator of Rows. Each Row represents one scan file from the ColumnarBatch returned by getScanFiles(TableClient)

filter - An optional filter that can be used for data skipping while reading the scan files.

Returns:

Data read from the input scan files as an iterator of DataReadResults. Each DataReadResult instance contains the data read and an optional selection vector that indicates data rows as valid or invalid. It is the responsibility of the caller to close this iterator.

Throws:

java.io.IOException - when error occurs while reading the data.