org.apache.hudi.internal.schema.utils

Class InternalSchemaUtils

java.lang.Object

org.apache.hudi.internal.schema.utils.InternalSchemaUtils

public class InternalSchemaUtils
extends Object

Util methods to help us do some operations on InternalSchema. eg: column prune, filter rebuild for query engine...

Method Summary

Modifier and Type	Method and Description
static Map<String,String>	collectRenameCols(InternalSchema oldSchema, InternalSchema newSchema) Try to find all renamed cols between oldSchema and newSchema.
static Map<Integer,Pair<Type,Type>>	collectTypeChangedCols(InternalSchema schema, InternalSchema oldSchema) Collect all type changed cols to build a colPosition -> (newColType, oldColType) map.
static String	createFullName(String name, Deque<String> fieldNames)
static InternalSchema	pruneInternalSchema(InternalSchema schema, List<String> names) Create project internalSchema, based on the project names which produced by query engine.
static InternalSchema	pruneInternalSchemaByID(InternalSchema schema, List<Integer> fieldIds, List<Integer> topParentFieldIds) Create project internalSchema.
static String	reBuildFilterName(String name, InternalSchema fileSchema, InternalSchema querySchema) A helper function to help correct the colName of pushed filters.
static InternalSchema	searchSchema(long versionId, List<InternalSchema> internalSchemas) Search target internalSchema by version number.
static InternalSchema	searchSchema(long versionId, TreeMap<Long,InternalSchema> treeMap) Search target internalSchema by version number.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

pruneInternalSchema

public static InternalSchema pruneInternalSchema(InternalSchema schema,
                                                 List<String> names)

Create project internalSchema, based on the project names which produced by query engine. support nested project.

Parameters:

schema - a internal schema.

names - project names produced by query engine.

Returns:

a project internalSchema.

pruneInternalSchemaByID

public static InternalSchema pruneInternalSchemaByID(InternalSchema schema,
                                                     List<Integer> fieldIds,
                                                     List<Integer> topParentFieldIds)

Create project internalSchema. support nested project.

Parameters:

schema - a internal schema.

fieldIds - project col field_ids.

Returns:

a project internalSchema.

reBuildFilterName

public static String reBuildFilterName(String name,
                                       InternalSchema fileSchema,
                                       InternalSchema querySchema)

A helper function to help correct the colName of pushed filters.

Parameters:

name - origin col name from pushed filters.

fileSchema - the real schema of avro/parquet file.

querySchema - the query schema which query engine produced.

Returns:

a corrected name.

collectTypeChangedCols

public static Map<Integer,Pair<Type,Type>> collectTypeChangedCols(InternalSchema schema,
                                                                  InternalSchema oldSchema)

Collect all type changed cols to build a colPosition -> (newColType, oldColType) map. only collect top level col changed. eg: a is a nest field(record(b int, d long), now a.b is changed from int to long, only a will be collected, a.b will excluded.

Parameters:

schema - a type changed internalSchema

oldSchema - an old internalSchema.

Returns:

a map.

searchSchema

public static InternalSchema searchSchema(long versionId,
                                          List<InternalSchema> internalSchemas)

Search target internalSchema by version number.

Parameters:

versionId - the internalSchema version to be search.

internalSchemas - internalSchemas to be searched.

Returns:

a internalSchema.

searchSchema

public static InternalSchema searchSchema(long versionId,
                                          TreeMap<Long,InternalSchema> treeMap)

Search target internalSchema by version number.

Parameters:

versionId - the internalSchema version to be search.

treeMap - internalSchemas collections to be searched.

Returns:

a internalSchema.

createFullName

public static String createFullName(String name,
                                    Deque<String> fieldNames)

collectRenameCols

public static Map<String,String> collectRenameCols(InternalSchema oldSchema,
                                                   InternalSchema newSchema)

Try to find all renamed cols between oldSchema and newSchema.

Parameters:

oldSchema - oldSchema

newSchema - newSchema which modified from oldSchema

Returns:

renameCols Map. (k, v) -> (colNameFromNewSchema, colNameLastPartFromOldSchema)