io.delta.kernel.internal

Class InternalScanFileUtils

Object

io.delta.kernel.internal.InternalScanFileUtils

public class InternalScanFileUtils
extends Object

Utilities to extract information out of the scan file rows returned by Scan.getScanFiles(Engine).

Field Summary

Fields

Modifier and Type	Field and Description
static int	ADD_FILE_ORDINAL
static Column	ADD_FILE_PARTITION_COL_REF Column expression referring to the `partitionValues` in scan `add` file.
static int	ADD_FILE_STATS_ORDINAL
static StructType	SCAN_FILE_SCHEMA Schema of the returned scan files.
static StructType	SCAN_FILE_SCHEMA_WITH_STATS Schema of the returned scan files when ScanImpl.getScanFiles(Engine, boolean) is called with includeStats=true.
static StructField	TABLE_ROOT_STRUCT_FIELD

Method Summary

Modifier and Type	Method and Description
static Row	generateScanFileRow(FileStatus fileStatus) Create a scan file row conforming to the schema SCAN_FILE_SCHEMA for given file status.
static FileStatus	getAddFileStatus(Row scanFileInfo) Get the FileStatus of AddFile from given scan file Row.
static DeletionVectorDescriptor	getDeletionVectorDescriptorFromRow(Row scanFile) Create a DeletionVectorDescriptor from add entry in the given scan file row.
static java.util.Map<String,String>	getPartitionValues(Row scanFileInfo) Get the partition columns and values belonging to the AddFile from given scan file row.
static Column	getPartitionValuesParsedRefInAddFile(String partitionColName) Get a references column for given partition column name in partitionValues_parsed column in scan file row.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ADD_FILE_PARTITION_COL_REF

public static final Column ADD_FILE_PARTITION_COL_REF

Column expression referring to the `partitionValues` in scan `add` file.

TABLE_ROOT_STRUCT_FIELD

public static StructField TABLE_ROOT_STRUCT_FIELD

SCAN_FILE_SCHEMA

public static final StructType SCAN_FILE_SCHEMA

Schema of the returned scan files. May have an additional column "add.stats" at the end of the "add" columns that is not represented in the schema here. This column is conditionally read when a valid data skipping filter can be generated.

SCAN_FILE_SCHEMA_WITH_STATS

public static final StructType SCAN_FILE_SCHEMA_WITH_STATS

Schema of the returned scan files when ScanImpl.getScanFiles(Engine, boolean) is called with includeStats=true.

ADD_FILE_ORDINAL

public static final int ADD_FILE_ORDINAL

ADD_FILE_STATS_ORDINAL

public static final int ADD_FILE_STATS_ORDINAL

Method Detail

getAddFileStatus

public static FileStatus getAddFileStatus(Row scanFileInfo)

Get the FileStatus of AddFile from given scan file Row. The FileStatus contains file metadata about the file.

Parameters:

scanFileInfo - Row representing one scan file.

Returns:

a FileStatus object created from the given scan file row.

getPartitionValues

public static java.util.Map<String,String> getPartitionValues(Row scanFileInfo)

Get the partition columns and values belonging to the AddFile from given scan file row.

Parameters:

scanFileInfo - Row representing one scan file.

Returns:

Map of partition column name to partition column value.

generateScanFileRow

public static Row generateScanFileRow(FileStatus fileStatus)

Create a scan file row conforming to the schema SCAN_FILE_SCHEMA for given file status. This is used when creating the ScanFile row for reading commit or checkpoint files.

Parameters:

fileStatus -

Returns:

getDeletionVectorDescriptorFromRow

public static DeletionVectorDescriptor getDeletionVectorDescriptorFromRow(Row scanFile)

Create a DeletionVectorDescriptor from add entry in the given scan file row.

Parameters:

scanFile - Row representing one scan file.

Returns:

getPartitionValuesParsedRefInAddFile

public static Column getPartitionValuesParsedRefInAddFile(String partitionColName)

Get a references column for given partition column name in partitionValues_parsed column in scan file row.

Parameters:

partitionColName - Partition column name

Returns:

Column reference