| Package | Description |
|---|---|
| org.apache.hudi.utilities | |
| org.apache.hudi.utilities.deltastreamer | |
| org.apache.hudi.utilities.schema | |
| org.apache.hudi.utilities.sources |
| Modifier and Type | Method and Description |
|---|---|
static SchemaProvider |
UtilHelpers.createSchemaProvider(String schemaProviderClass,
TypedProperties cfg,
org.apache.spark.api.java.JavaSparkContext jssc) |
| Modifier and Type | Method and Description |
|---|---|
static Source |
UtilHelpers.createSource(String sourceClass,
TypedProperties cfg,
org.apache.spark.api.java.JavaSparkContext jssc,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
| Modifier and Type | Method and Description |
|---|---|
SchemaProvider |
HoodieDeltaStreamer.DeltaSyncService.getSchemaProvider() |
| Constructor and Description |
|---|
DeltaSync(HoodieDeltaStreamer.Config cfg,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider,
HoodieTableType tableType,
TypedProperties props,
org.apache.spark.api.java.JavaSparkContext jssc,
org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.hive.conf.HiveConf hiveConf,
java.util.function.Function<HoodieWriteClient,Boolean> onInitializingHoodieWriteClient) |
| Modifier and Type | Class and Description |
|---|---|
class |
FilebasedSchemaProvider
A simple schema provider, that reads off files on DFS
|
class |
NullTargetSchemaRegistryProvider
Schema provider that will force DeltaStreamer to infer target schema from the dataset.
|
class |
RowBasedSchemaProvider |
class |
SchemaRegistryProvider
Obtains latest schema from the Confluent/Kafka schema-registry
https://github.com/confluentinc/schema-registry
|
| Modifier and Type | Method and Description |
|---|---|
SchemaProvider |
InputBatch.getSchemaProvider() |
| Constructor and Description |
|---|
AvroDFSSource(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
AvroKafkaSource(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
AvroSource(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
HiveIncrPullSource(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
HoodieIncrSource(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
InputBatch(Option<T> batch,
String checkpointForNextBatch,
SchemaProvider schemaProvider) |
JsonDFSSource(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
JsonKafkaSource(TypedProperties properties,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
JsonSource(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
RowSource(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
Source(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider) |
Source(TypedProperties props,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.spark.sql.SparkSession sparkSession,
SchemaProvider schemaProvider,
Source.SourceType sourceType) |
Copyright © 2019 The Apache Software Foundation. All rights reserved.