za.co.absa.abris.avro.AvroSerDe
DataframeDeserializer
Related Doc:
package AvroSerDe
Loads the schema from Schema Registry.
Loads the schema from a file.
Converts the binary Avro records contained in the Dataframe into regular Rows with a SQL schema whose specification is translated from the Avro schema informed.
Converts the binary Avro records contained in the Dataframe into regular Rows with a SQL schema whose specification is translated from the Avro schema informed.
Converts the binary Avro records contained in the Dataframe into regular Rows with a SQL schema whose specification is translated from the Avro schema informed.
Converts the binary Avro records contained in the Dataframe into regular Rows with a SQL schema whose specification is translated from the Avro schema informed.
This method supports schema changes from Schema Registry.
This method supports schema changes from Schema Registry. However, the conversion between Avro records and Spark rows relies on RowEncoders, which are defined before the job starts. Thus, although the schema changes are supported while reading, they are not translated to RowEncoders, which could take to errors in the final data.
Refer to the ScalaConfluentKafkaAvroDeserializer.deserialize() documentation to better understand how this operation is performed.
Converts Dataframes of binary Avro records into Dataframes of type Spark data.
Converts Dataframes of binary Avro records into Dataframes of type Spark data.
Highlights:
1. Either, the path to a schema stored in a file system or the configuration to access a Confluent's Schema Registry instance must be informed.
2. The RowEncoder for the resulting Dataframes will be created here, thus. If a schema path is informed, the schema under that path will be used to create the RowEncoder, otherwise, the schema retrieved from Schema Registry will be used.
To allow the retrieval of a remote schema, the API will look into the configurations for:
This class provides the method that converts binary Avro records from a Dataframe into Spark Rows on the fly.
It loads binary data from a stream and feed them into an Avro/Spark decoder, returning the resulting rows.
It requires the path to the Avro schema which defines the records to be read.