org.apache.hudi.common.table

Class TableSchemaResolver

java.lang.Object

org.apache.hudi.common.table.TableSchemaResolver

@ThreadSafe
public class TableSchemaResolver
extends Object

Helper class to read schema from data files and log files and to convert it between different formats.

Constructor Summary

Constructors

Constructor and Description
TableSchemaResolver(HoodieTableMetaClient metaClient)

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.Schema	appendPartitionColumns(org.apache.avro.Schema dataSchema, Option<String[]> partitionFields)
static org.apache.parquet.schema.MessageType	convertAvroSchemaToParquet(org.apache.avro.Schema schema, org.apache.hadoop.conf.Configuration hadoopConf)
org.apache.avro.Schema	getLatestSchema(org.apache.avro.Schema writeSchema, boolean convertTableSchemaToAddNamespace, Functions.Function1<org.apache.avro.Schema,org.apache.avro.Schema> converterFn) Deprecated. will be removed (HUDI-4472)
org.apache.avro.Schema	getTableAvroSchema() Gets full schema (user + metadata) for a hoodie table in Avro format.
org.apache.avro.Schema	getTableAvroSchema(boolean includeMetadataFields) Gets schema for a hoodie table in Avro format, can choice if include metadata fields.
org.apache.avro.Schema	getTableAvroSchema(HoodieInstant instant, boolean includeMetadataFields) Fetches tables schema in Avro format as of the given instant
org.apache.avro.Schema	getTableAvroSchema(String timestamp) Fetches tables schema in Avro format as of the given instant
org.apache.avro.Schema	getTableAvroSchemaFromDataFile()
Option<org.apache.avro.Schema>	getTableAvroSchemaFromLatestCommit(boolean includeMetadataFields) Returns table's latest Avro Schema iff table is non-empty (ie there's at least a single commit) This method differs from getTableAvroSchema(boolean) in that it won't fallback to use table's schema used at creation
org.apache.avro.Schema	getTableAvroSchemaWithoutMetadataFields() Deprecated. use getTableAvroSchema(boolean) instead
Option<String>	getTableHistorySchemaStrFromCommitMetadata() Gets the history schemas as String for a hoodie table from the HoodieCommitMetadata of the instant.
Option<InternalSchema>	getTableInternalSchemaFromCommitMetadata() Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.
Option<InternalSchema>	getTableInternalSchemaFromCommitMetadata(String timestamp) Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.
org.apache.parquet.schema.MessageType	getTableParquetSchema() Gets full schema (user + metadata) for a hoodie table in Parquet format.
boolean	hasOperationField() NOTE: This method could only be used in tests
static boolean	isSchemaCompatible(org.apache.avro.Schema oldSchema, org.apache.avro.Schema newSchema) HUDI specific validation of schema evolution.
static boolean	isSchemaCompatible(String oldSchema, String newSchema)
org.apache.parquet.schema.MessageType	readSchemaFromLastCompaction(Option<HoodieInstant> lastCompactionCommitOpt) Deprecated. please use getTableAvroSchema(HoodieInstant, boolean) instead
static org.apache.parquet.schema.MessageType	readSchemaFromLogFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Read the schema from the log file on path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TableSchemaResolver

public TableSchemaResolver(HoodieTableMetaClient metaClient)

Method Detail

getTableAvroSchemaFromDataFile

public org.apache.avro.Schema getTableAvroSchemaFromDataFile()

getTableAvroSchema

public org.apache.avro.Schema getTableAvroSchema()
                                          throws Exception

Gets full schema (user + metadata) for a hoodie table in Avro format.

Returns:

Avro schema for this table

Throws:

Exception

getTableAvroSchema

public org.apache.avro.Schema getTableAvroSchema(boolean includeMetadataFields)
                                          throws Exception

Gets schema for a hoodie table in Avro format, can choice if include metadata fields.

Parameters:

includeMetadataFields - choice if include metadata fields

Returns:

Avro schema for this table

Throws:

Exception

getTableAvroSchema

public org.apache.avro.Schema getTableAvroSchema(String timestamp)
                                          throws Exception

Fetches tables schema in Avro format as of the given instant

Parameters:

timestamp - as of which table's schema will be fetched

Throws:

Exception

getTableAvroSchema

public org.apache.avro.Schema getTableAvroSchema(HoodieInstant instant,
                                                 boolean includeMetadataFields)
                                          throws Exception

Fetches tables schema in Avro format as of the given instant

Parameters:

instant - as of which table's schema will be fetched

Throws:

Exception

getTableParquetSchema

public org.apache.parquet.schema.MessageType getTableParquetSchema()
                                                            throws Exception

Gets full schema (user + metadata) for a hoodie table in Parquet format.

Returns:

Parquet schema for the table

Throws:

Exception

getTableAvroSchemaWithoutMetadataFields

@Deprecated
public org.apache.avro.Schema getTableAvroSchemaWithoutMetadataFields()
                                                                           throws Exception

Deprecated. use getTableAvroSchema(boolean) instead

Gets users data schema for a hoodie table in Avro format.

Returns:

Avro user data schema

Throws:

Exception

convertAvroSchemaToParquet

public static org.apache.parquet.schema.MessageType convertAvroSchemaToParquet(org.apache.avro.Schema schema,
                                                                               org.apache.hadoop.conf.Configuration hadoopConf)

isSchemaCompatible

public static boolean isSchemaCompatible(org.apache.avro.Schema oldSchema,
                                         org.apache.avro.Schema newSchema)

HUDI specific validation of schema evolution. Ensures that a newer schema can be used for the dataset by checking if the data written using the old schema can be read using the new schema. HUDI requires a Schema to be specified in HoodieWriteConfig and is used by the HoodieWriteClient to create the records. The schema is also saved in the data files (parquet format) and log files (avro format). Since a schema is required each time new data is ingested into a HUDI dataset, schema can be evolved over time. New Schema is compatible only if: A1. There is no change in schema A2. A field has been added and it has a default value specified New Schema is incompatible if: B1. A field has been deleted B2. A field has been renamed (treated as delete + add) B3. A field's type has changed to be incompatible with the older type Issue with org.apache.avro.SchemaCompatibility: org.apache.avro.SchemaCompatibility checks schema compatibility between a writer schema (which originally wrote the AVRO record) and a readerSchema (with which we are reading the record). It ONLY guarantees that that each field in the reader record can be populated from the writer record. Hence, if the reader schema is missing a field, it is still compatible with the writer schema. In other words, org.apache.avro.SchemaCompatibility was written to guarantee that we can read the data written earlier. It does not guarantee schema evolution for HUDI (B1 above). Implementation: This function implements specific HUDI specific checks (listed below) and defers the remaining checks to the org.apache.avro.SchemaCompatibility code. Checks: C1. If there is no change in schema: success C2. If a field has been deleted in new schema: failure C3. If a field has been added in new schema: it should have default value specified C4. If a field has been renamed(treated as delete + add): failure C5. If a field type has changed: failure

Parameters:

oldSchema - Older schema to check.

newSchema - Newer schema to check.

Returns:

True if the schema validation is successful TODO revisit this method: it's implemented incorrectly as it might be applying different criteria to top-level record and nested record (for ex, if that nested record is contained w/in an array)

isSchemaCompatible

public static boolean isSchemaCompatible(String oldSchema,
                                         String newSchema)

getTableAvroSchemaFromLatestCommit

public Option<org.apache.avro.Schema> getTableAvroSchemaFromLatestCommit(boolean includeMetadataFields)
                                                                  throws Exception

Returns table's latest Avro Schema iff table is non-empty (ie there's at least a single commit) This method differs from getTableAvroSchema(boolean) in that it won't fallback to use table's schema used at creation

Throws:

Exception

getLatestSchema

@Deprecated
public org.apache.avro.Schema getLatestSchema(org.apache.avro.Schema writeSchema,
                                                          boolean convertTableSchemaToAddNamespace,
                                                          Functions.Function1<org.apache.avro.Schema,org.apache.avro.Schema> converterFn)

Deprecated. will be removed (HUDI-4472)

Get latest schema either from incoming schema or table schema.

Parameters:

writeSchema - incoming batch's write schema.

convertTableSchemaToAddNamespace - true if table schema needs to be converted. false otherwise.

converterFn - converter function to be called over table schema (to add namespace may be). Each caller can decide if any conversion is required.

Returns:

the latest schema.

readSchemaFromLastCompaction

public org.apache.parquet.schema.MessageType readSchemaFromLastCompaction(Option<HoodieInstant> lastCompactionCommitOpt)
                                                                   throws Exception

Deprecated. please use getTableAvroSchema(HoodieInstant, boolean) instead

Read schema from a data file from the last compaction commit done.

Throws:

Exception

readSchemaFromLogFile

public static org.apache.parquet.schema.MessageType readSchemaFromLogFile(org.apache.hadoop.fs.FileSystem fs,
                                                                          org.apache.hadoop.fs.Path path)
                                                                   throws IOException

Read the schema from the log file on path.

Returns:

Throws:

IOException

getTableInternalSchemaFromCommitMetadata

public Option<InternalSchema> getTableInternalSchemaFromCommitMetadata()

Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.

Returns:

InternalSchema for this table

getTableInternalSchemaFromCommitMetadata

public Option<InternalSchema> getTableInternalSchemaFromCommitMetadata(String timestamp)

Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.

Returns:

InternalSchema for this table

getTableHistorySchemaStrFromCommitMetadata

public Option<String> getTableHistorySchemaStrFromCommitMetadata()

Gets the history schemas as String for a hoodie table from the HoodieCommitMetadata of the instant.

Returns:

history schemas string for this table

hasOperationField

public boolean hasOperationField()

NOTE: This method could only be used in tests

appendPartitionColumns

public static org.apache.avro.Schema appendPartitionColumns(org.apache.avro.Schema dataSchema,
                                                            Option<String[]> partitionFields)