org.apache.hudi.hive.util

Class SchemaUtil

java.lang.Object

org.apache.hudi.hive.util.SchemaUtil

public class SchemaUtil
extends Object

Schema Utilities

Constructor Summary

Constructors

Constructor and Description
SchemaUtil()

Method Summary

Modifier and Type	Method and Description
static Map<String,String>	convertParquetSchemaToHiveSchema(org.apache.parquet.schema.MessageType messageType) Returns equivalent Hive table schema read from a parquet file
static String	generateCreateDDL(org.apache.parquet.schema.MessageType storageSchema, HiveSyncConfig config, String inputFormatClass, String outputFormatClass, String serdeClass)
static String	generateSchemaString(org.apache.parquet.schema.MessageType storageSchema)
static String	generateSchemaString(org.apache.parquet.schema.MessageType storageSchema, List<String> colsToSkip)
static SchemaDifference	getSchemaDifference(org.apache.parquet.schema.MessageType storageSchema, Map<String,String> tableSchema, List<String> partitionKeys) Get the schema difference between the storage schema and hive table schema
static boolean	isSchemaTypeUpdateAllowed(String prevType, String newType)
static org.apache.parquet.schema.MessageType	readSchemaFromLogFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Read the schema from the log file on path

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SchemaUtil

public SchemaUtil()

Method Detail

getSchemaDifference

public static SchemaDifference getSchemaDifference(org.apache.parquet.schema.MessageType storageSchema,
                                                   Map<String,String> tableSchema,
                                                   List<String> partitionKeys)

Get the schema difference between the storage schema and hive table schema

convertParquetSchemaToHiveSchema

public static Map<String,String> convertParquetSchemaToHiveSchema(org.apache.parquet.schema.MessageType messageType)
                                                           throws IOException

Returns equivalent Hive table schema read from a parquet file

Parameters:

messageType - : Parquet Schema

Returns:

: Hive Table schema read from parquet file MAP[String,String]

Throws:

IOException

isSchemaTypeUpdateAllowed

public static boolean isSchemaTypeUpdateAllowed(String prevType,
                                                String newType)

generateSchemaString

public static String generateSchemaString(org.apache.parquet.schema.MessageType storageSchema)
                                   throws IOException

Throws:

IOException

generateSchemaString

public static String generateSchemaString(org.apache.parquet.schema.MessageType storageSchema,
                                          List<String> colsToSkip)
                                   throws IOException

Throws:

IOException

generateCreateDDL

public static String generateCreateDDL(org.apache.parquet.schema.MessageType storageSchema,
                                       HiveSyncConfig config,
                                       String inputFormatClass,
                                       String outputFormatClass,
                                       String serdeClass)
                                throws IOException

Throws:

IOException

readSchemaFromLogFile

public static org.apache.parquet.schema.MessageType readSchemaFromLogFile(org.apache.hadoop.fs.FileSystem fs,
                                                                          org.apache.hadoop.fs.Path path)
                                                                   throws IOException

Read the schema from the log file on path

Returns:

Throws:

IOException