A B C D E F G H I J K L M N O P R S T U V W Y

A

AbstractDeltaStreamerService - Class in org.apache.hudi.utilities.deltastreamer

Base Class for running delta-sync/compaction in separate thread and controlling their life-cyle

apply(JavaSparkContext, SparkSession, Dataset<Row>, TypedProperties) - Method in class org.apache.hudi.utilities.transform.FlatteningTransformer

Configs supported

apply(JavaSparkContext, SparkSession, Dataset<Row>, TypedProperties) - Method in class org.apache.hudi.utilities.transform.IdentityTransformer

apply(JavaSparkContext, SparkSession, Dataset<Row>, TypedProperties) - Method in class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer

apply(JavaSparkContext, SparkSession, Dataset<Row>, TypedProperties) - Method in interface org.apache.hudi.utilities.transform.Transformer

Transform source RDD to target RDD

AsyncCompactService(JavaSparkContext, HoodieWriteClient) - Constructor for class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.AsyncCompactService

AvroConvertor - Class in org.apache.hudi.utilities.sources.helpers

Convert a variety of datum into Avro GenericRecords.

AvroConvertor(String) - Constructor for class org.apache.hudi.utilities.sources.helpers.AvroConvertor

AvroConvertor(Schema) - Constructor for class org.apache.hudi.utilities.sources.helpers.AvroConvertor

AvroDFSSource - Class in org.apache.hudi.utilities.sources

DFS Source that reads avro data

AvroDFSSource(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.AvroDFSSource

AvroKafkaSource - Class in org.apache.hudi.utilities.sources

Reads avro serialized Kafka data, based on the confluent schema-registry

AvroKafkaSource(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.AvroKafkaSource

AvroSource - Class in org.apache.hudi.utilities.sources

AvroSource(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.AvroSource

B

basePath - Variable in class org.apache.hudi.utilities.HoodieCleaner.Config

basePath - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

basePath - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

basePath - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

baseStorePathForFileGroups - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

buildHoodieRecordsForImport(JavaSparkContext, String) - Method in class org.apache.hudi.utilities.HDFSParquetImporter

buildProperties(List<String>) - Static method in class org.apache.hudi.utilities.UtilHelpers

buildSparkContext(String, String, Map<String, String>) - Static method in class org.apache.hudi.utilities.UtilHelpers

buildSparkContext(String, String) - Static method in class org.apache.hudi.utilities.UtilHelpers

buildSparkContext(String, String, String) - Static method in class org.apache.hudi.utilities.UtilHelpers

Build Spark Context for ingestion/compaction

C

calculateBeginAndEndInstants(JavaSparkContext, String, int, Option<String>, boolean) - Static method in class org.apache.hudi.utilities.sources.helpers.IncrSourceHelper

Find begin and end instants to be set for the next fetch

checkpoint - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

Resume Delta Streamer from this checkpoint.

CHECKPOINT_KEY - Static variable in class org.apache.hudi.utilities.deltastreamer.DeltaSync

CHECKPOINT_KEY - Static variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

CHECKPOINT_RESET_KEY - Static variable in class org.apache.hudi.utilities.deltastreamer.DeltaSync

CheckpointUtils() - Constructor for class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen.CheckpointUtils

close() - Method in class org.apache.hudi.utilities.deltastreamer.DeltaSync

Close all resources

close() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

Close all resources

command - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

commitOnErrors - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

compact(HoodieInstant) - Method in class org.apache.hudi.utilities.deltastreamer.Compactor

compact(JavaSparkContext, int) - Method in class org.apache.hudi.utilities.HoodieCompactor

COMPACT_POOL_NAME - Static variable in class org.apache.hudi.utilities.deltastreamer.SchedulerConfGenerator

compactionInstantTime - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

compactionInstantTime - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

Compactor - Class in org.apache.hudi.utilities.deltastreamer

Run one round of compaction

Compactor(HoodieWriteClient, JavaSparkContext) - Constructor for class org.apache.hudi.utilities.deltastreamer.Compactor

compactSchedulingMinShare - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

compactSchedulingWeight - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

computeOffsetRanges(HashMap<TopicAndPartition, KafkaCluster.LeaderOffset>, HashMap<TopicAndPartition, KafkaCluster.LeaderOffset>, long) - Static method in class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen.CheckpointUtils

Compute the offset ranges to read from Kafka, while handling newly added partitions, skews, event limits.

Config() - Constructor for class org.apache.hudi.utilities.adhoc.UpgradePayloadFromUberToApache.Config

Config() - Constructor for class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

Config() - Constructor for class org.apache.hudi.utilities.HDFSParquetImporter.Config

config - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller

Config() - Constructor for class org.apache.hudi.utilities.HiveIncrementalPuller.Config

Config() - Constructor for class org.apache.hudi.utilities.HoodieCleaner.Config

Config() - Constructor for class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

Config() - Constructor for class org.apache.hudi.utilities.HoodieCompactor.Config

Config() - Constructor for class org.apache.hudi.utilities.HoodieWithTimelineServer.Config

Config() - Constructor for class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

Config() - Constructor for class org.apache.hudi.utilities.schema.FilebasedSchemaProvider.Config

config - Variable in class org.apache.hudi.utilities.schema.SchemaProvider

Config() - Constructor for class org.apache.hudi.utilities.schema.SchemaRegistryProvider.Config

Config() - Constructor for class org.apache.hudi.utilities.sources.HoodieIncrSource.Config

configs - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

configs - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

configs - Variable in class org.apache.hudi.utilities.HoodieCleaner.Config

configs - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

continuousMode - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

createHoodieClient(JavaSparkContext, String, String, int, Option<String>, TypedProperties) - Static method in class org.apache.hudi.utilities.UtilHelpers

Build Hoodie write client

createSchemaProvider(String, TypedProperties, JavaSparkContext) - Static method in class org.apache.hudi.utilities.UtilHelpers

createSource(String, TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Static method in class org.apache.hudi.utilities.UtilHelpers

createTransformer(String) - Static method in class org.apache.hudi.utilities.UtilHelpers

D

dataImport(JavaSparkContext, int) - Method in class org.apache.hudi.utilities.HDFSParquetImporter

dataImport(JavaSparkContext) - Method in class org.apache.hudi.utilities.HDFSParquetImporter

delaySecs - Variable in class org.apache.hudi.utilities.HoodieWithTimelineServer.Config

DeltaSync - Class in org.apache.hudi.utilities.deltastreamer

Sync's one batch of data to hoodie dataset

DeltaSync(HoodieDeltaStreamer.Config, SparkSession, SchemaProvider, HoodieTableType, TypedProperties, JavaSparkContext, FileSystem, HiveConf, Function<HoodieWriteClient, Boolean>) - Constructor for class org.apache.hudi.utilities.deltastreamer.DeltaSync

DELTASYNC_POOL_NAME - Static variable in class org.apache.hudi.utilities.deltastreamer.SchedulerConfGenerator

deltaSyncSchedulingMinShare - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

deltaSyncSchedulingWeight - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

DeltaSyncService(HoodieDeltaStreamer.Config, JavaSparkContext, FileSystem, HiveConf) - Constructor for class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

DFSPathSelector - Class in org.apache.hudi.utilities.sources.helpers

DFSPathSelector(TypedProperties, Configuration) - Constructor for class org.apache.hudi.utilities.sources.helpers.DFSPathSelector

dryRun - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

E

enableHiveSync - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

enqueuePendingCompaction(HoodieInstant) - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.AsyncCompactService

Enqueues new Pending compaction

F

fetchNewData(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.AvroDFSSource

fetchNewData(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.AvroKafkaSource

fetchNewData(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.HiveIncrPullSource

fetchNewData(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.JsonDFSSource

fetchNewData(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.JsonKafkaSource

fetchNewData(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.RowSource

fetchNewData(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.Source

fetchNewDataInAvroFormat(Option<String>, long) - Method in class org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter

Fetch new data in avro format.

fetchNewDataInRowFormat(Option<String>, long) - Method in class org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter

Fetch new data in row format.

fetchNext(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.Source

Main API called by Hoodie Delta Streamer to fetch records

fetchNextBatch(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.HoodieIncrSource

fetchNextBatch(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.RowSource

FilebasedSchemaProvider - Class in org.apache.hudi.utilities.schema

A simple schema provider, that reads off files on DFS

FilebasedSchemaProvider(TypedProperties, JavaSparkContext) - Constructor for class org.apache.hudi.utilities.schema.FilebasedSchemaProvider

FilebasedSchemaProvider.Config - Class in org.apache.hudi.utilities.schema

Configs supported

fileId - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

filterDupes - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

FlatteningTransformer - Class in org.apache.hudi.utilities.transform

Transformer that can flatten nested objects.

FlatteningTransformer() - Constructor for class org.apache.hudi.utilities.transform.FlatteningTransformer

flattenSchema(StructType, String) - Method in class org.apache.hudi.utilities.transform.FlatteningTransformer

forceDisableCompaction - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

Compaction is enabled for MoR table by default.

format - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

FormatValidator() - Constructor for class org.apache.hudi.utilities.HDFSParquetImporter.FormatValidator

fromAvroBinary(byte[]) - Method in class org.apache.hudi.utilities.sources.helpers.AvroConvertor

fromCommitTime - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

fromJson(String) - Method in class org.apache.hudi.utilities.sources.helpers.AvroConvertor

G

getAsyncCompactService() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

getBatch() - Method in class org.apache.hudi.utilities.sources.InputBatch

getCheckpointForNextBatch() - Method in class org.apache.hudi.utilities.sources.InputBatch

getDeltaSync() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

getDeltaSyncService() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

getDurationInMs(long) - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics

getHiveSyncTimerContext() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics

getJavaSparkContext() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

getKafkaParams() - Method in class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen

getKey(GenericRecord) - Method in class org.apache.hudi.utilities.keygen.TimestampBasedKeyGenerator

getNextFilePathsAndMaxModificationTime(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.helpers.DFSPathSelector

getNextOffsetRanges(Option<String>, long) - Method in class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen

getOverallTimerContext() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics

getProps() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

getSchema() - Method in class org.apache.hudi.utilities.sources.helpers.AvroConvertor

getSchemaProvider() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

getSchemaProvider() - Method in class org.apache.hudi.utilities.sources.InputBatch

getSourceSchema() - Method in class org.apache.hudi.utilities.schema.FilebasedSchemaProvider

getSourceSchema() - Method in class org.apache.hudi.utilities.schema.RowBasedSchemaProvider

getSourceSchema() - Method in class org.apache.hudi.utilities.schema.SchemaProvider

getSourceSchema() - Method in class org.apache.hudi.utilities.schema.SchemaRegistryProvider

getSourceType() - Method in class org.apache.hudi.utilities.sources.Source

getSparkSchedulingConfigs(HoodieDeltaStreamer.Config) - Static method in class org.apache.hudi.utilities.deltastreamer.SchedulerConfGenerator

Helper to set Spark Scheduling Configs dynamically

getSparkSession() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

getSparkSession() - Method in class org.apache.hudi.utilities.sources.Source

getTargetSchema() - Method in class org.apache.hudi.utilities.schema.FilebasedSchemaProvider

getTargetSchema() - Method in class org.apache.hudi.utilities.schema.NullTargetSchemaRegistryProvider

getTargetSchema() - Method in class org.apache.hudi.utilities.schema.SchemaProvider

getTargetSchema() - Method in class org.apache.hudi.utilities.schema.SchemaRegistryProvider

getTimelinServerConfig() - Method in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

getTopicName() - Method in class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen

H

handleErrors(JavaSparkContext, String, JavaRDD<WriteStatus>) - Static method in class org.apache.hudi.utilities.UtilHelpers

HDFSParquetImporter - Class in org.apache.hudi.utilities

Loads data from Parquet Sources

HDFSParquetImporter(HDFSParquetImporter.Config) - Constructor for class org.apache.hudi.utilities.HDFSParquetImporter

HDFSParquetImporter.Config - Class in org.apache.hudi.utilities

HDFSParquetImporter.FormatValidator - Class in org.apache.hudi.utilities

help - Variable in class org.apache.hudi.utilities.adhoc.UpgradePayloadFromUberToApache.Config

help - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

help - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

help - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

help - Variable in class org.apache.hudi.utilities.HoodieCleaner.Config

help - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

help - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

help - Variable in class org.apache.hudi.utilities.HoodieWithTimelineServer.Config

help - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

HiveIncrementalPuller - Class in org.apache.hudi.utilities

Utility to pull data after a given commit, based on the supplied HiveQL and save the delta as another hive temporary table.

HiveIncrementalPuller(HiveIncrementalPuller.Config) - Constructor for class org.apache.hudi.utilities.HiveIncrementalPuller

HiveIncrementalPuller.Config - Class in org.apache.hudi.utilities

HiveIncrPullSource - Class in org.apache.hudi.utilities.sources

Source to read deltas produced by HiveIncrementalPuller, commit by commit and apply to the target table

HiveIncrPullSource(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.HiveIncrPullSource

hiveJDBCUrl - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

hivePassword - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

hiveSyncTimer - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics

hiveSyncTimerName - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics

hiveUsername - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

HOODIE_RECORD_NAMESPACE - Static variable in class org.apache.hudi.utilities.schema.RowBasedSchemaProvider

HOODIE_RECORD_STRUCT_NAME - Static variable in class org.apache.hudi.utilities.schema.RowBasedSchemaProvider

HoodieCleaner - Class in org.apache.hudi.utilities

HoodieCleaner(HoodieCleaner.Config, JavaSparkContext) - Constructor for class org.apache.hudi.utilities.HoodieCleaner

HoodieCleaner.Config - Class in org.apache.hudi.utilities

HoodieCompactionAdminTool - Class in org.apache.hudi.utilities

HoodieCompactionAdminTool(HoodieCompactionAdminTool.Config) - Constructor for class org.apache.hudi.utilities.HoodieCompactionAdminTool

HoodieCompactionAdminTool.Config - Class in org.apache.hudi.utilities

Admin Configuration Options

HoodieCompactionAdminTool.Operation - Enum in org.apache.hudi.utilities

Operation Types

HoodieCompactor - Class in org.apache.hudi.utilities

HoodieCompactor(HoodieCompactor.Config) - Constructor for class org.apache.hudi.utilities.HoodieCompactor

HoodieCompactor.Config - Class in org.apache.hudi.utilities

HoodieDeltaStreamer - Class in org.apache.hudi.utilities.deltastreamer

An Utility which can incrementally take the output from HiveIncrementalPuller and apply it to the target dataset.

HoodieDeltaStreamer(HoodieDeltaStreamer.Config, JavaSparkContext) - Constructor for class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

HoodieDeltaStreamer(HoodieDeltaStreamer.Config, JavaSparkContext, FileSystem, HiveConf) - Constructor for class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

HoodieDeltaStreamer.AsyncCompactService - Class in org.apache.hudi.utilities.deltastreamer

Async Compactor Service tha runs in separate thread.

HoodieDeltaStreamer.Config - Class in org.apache.hudi.utilities.deltastreamer

HoodieDeltaStreamer.DeltaSyncService - Class in org.apache.hudi.utilities.deltastreamer

Syncs data either in single-run or in continuous mode.

HoodieDeltaStreamer.Operation - Enum in org.apache.hudi.utilities.deltastreamer

HoodieDeltaStreamerException - Exception in org.apache.hudi.utilities.exception

HoodieDeltaStreamerException(String, Throwable) - Constructor for exception org.apache.hudi.utilities.exception.HoodieDeltaStreamerException

HoodieDeltaStreamerException(String) - Constructor for exception org.apache.hudi.utilities.exception.HoodieDeltaStreamerException

HoodieDeltaStreamerMetrics - Class in org.apache.hudi.utilities.deltastreamer

HoodieDeltaStreamerMetrics(HoodieWriteConfig) - Constructor for class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics

HoodieIncrementalPullException - Exception in org.apache.hudi.utilities.exception

HoodieIncrementalPullException(String, SQLException) - Constructor for exception org.apache.hudi.utilities.exception.HoodieIncrementalPullException

HoodieIncrementalPullException(String) - Constructor for exception org.apache.hudi.utilities.exception.HoodieIncrementalPullException

HoodieIncrementalPullSQLException - Exception in org.apache.hudi.utilities.exception

HoodieIncrementalPullSQLException(String, SQLException) - Constructor for exception org.apache.hudi.utilities.exception.HoodieIncrementalPullSQLException

HoodieIncrementalPullSQLException(String) - Constructor for exception org.apache.hudi.utilities.exception.HoodieIncrementalPullSQLException

HoodieIncrSource - Class in org.apache.hudi.utilities.sources

HoodieIncrSource(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.HoodieIncrSource

HoodieIncrSource.Config - Class in org.apache.hudi.utilities.sources

HoodieSnapshotCopier - Class in org.apache.hudi.utilities

Hoodie snapshot copy job which copies latest files from all partitions to another place, for snapshot backup.

HoodieSnapshotCopier() - Constructor for class org.apache.hudi.utilities.HoodieSnapshotCopier

hoodieTmpDir - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

HoodieWithTimelineServer - Class in org.apache.hudi.utilities

HoodieWithTimelineServer(HoodieWithTimelineServer.Config) - Constructor for class org.apache.hudi.utilities.HoodieWithTimelineServer

HoodieWithTimelineServer.Config - Class in org.apache.hudi.utilities

I

IdentityTransformer - Class in org.apache.hudi.utilities.transform

Identity transformer

IdentityTransformer() - Constructor for class org.apache.hudi.utilities.transform.IdentityTransformer

incrementalSQLFile - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

IncrSourceHelper - Class in org.apache.hudi.utilities.sources.helpers

IncrSourceHelper() - Constructor for class org.apache.hudi.utilities.sources.helpers.IncrSourceHelper

InputBatch<T> - Class in org.apache.hudi.utilities.sources

InputBatch(Option<T>, String, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.InputBatch

InputBatch(Option<T>, String) - Constructor for class org.apache.hudi.utilities.sources.InputBatch

inputPath - Variable in class org.apache.hudi.utilities.adhoc.UpgradePayloadFromUberToApache.Config

isAsyncCompactionEnabled() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

isInlineCompactionEnabled() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

J

JsonDFSSource - Class in org.apache.hudi.utilities.sources

DFS Source that reads json data

JsonDFSSource(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.JsonDFSSource

JsonKafkaSource - Class in org.apache.hudi.utilities.sources

Read json kafka data

JsonKafkaSource(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.JsonKafkaSource

JsonSource - Class in org.apache.hudi.utilities.sources

JsonSource(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.JsonSource

jssc - Variable in class org.apache.hudi.utilities.schema.SchemaProvider

K

KafkaOffsetGen - Class in org.apache.hudi.utilities.sources.helpers

Source to read data from Kafka, incrementally

KafkaOffsetGen(TypedProperties) - Constructor for class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen

KafkaOffsetGen.CheckpointUtils - Class in org.apache.hudi.utilities.sources.helpers

L

load(HoodieWriteClient, String, JavaRDD<HoodieRecord<T>>) - Method in class org.apache.hudi.utilities.HDFSParquetImporter

Imports records to Hoodie dataset

log - Static variable in class org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService

log - Static variable in class org.apache.hudi.utilities.deltastreamer.Compactor

log - Static variable in class org.apache.hudi.utilities.deltastreamer.DeltaSync

log - Static variable in class org.apache.hudi.utilities.deltastreamer.SchedulerConfGenerator

log - Static variable in class org.apache.hudi.utilities.sources.Source

M

main(String[]) - Static method in class org.apache.hudi.utilities.adhoc.UpgradePayloadFromUberToApache

main(String[]) - Static method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

main(String[]) - Static method in class org.apache.hudi.utilities.HDFSParquetImporter

main(String[]) - Static method in class org.apache.hudi.utilities.HiveIncrementalPuller

main(String[]) - Static method in class org.apache.hudi.utilities.HoodieCleaner

main(String[]) - Static method in class org.apache.hudi.utilities.HoodieCompactionAdminTool

main(String[]) - Static method in class org.apache.hudi.utilities.HoodieCompactor

main(String[]) - Static method in class org.apache.hudi.utilities.HoodieSnapshotCopier

main(String[]) - Static method in class org.apache.hudi.utilities.HoodieWithTimelineServer

main(String[]) - Static method in class org.apache.hudi.utilities.perf.TimelineServerPerf

maxCommits - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

maxPartitions - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

maxPendingCompactions - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

maxViewMemPerTableInMB - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

memFractionForCompactionPerTable - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

minSyncIntervalSeconds - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

N

NullTargetSchemaRegistryProvider - Class in org.apache.hudi.utilities.schema

Schema provider that will force DeltaStreamer to infer target schema from the dataset.

NullTargetSchemaRegistryProvider(TypedProperties, JavaSparkContext) - Constructor for class org.apache.hudi.utilities.schema.NullTargetSchemaRegistryProvider

numCoresPerExecutor - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

numExecutors - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

numIterations - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

numPartitions - Variable in class org.apache.hudi.utilities.HoodieWithTimelineServer.Config

O

offsetsToStr(OffsetRange[]) - Static method in class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen.CheckpointUtils

String representation of checkpoint

onInitializingWriteClient(HoodieWriteClient) - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

Callback to initialize write client and start compaction service if required

operation - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

operation - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

org.apache.hudi.utilities - package org.apache.hudi.utilities

org.apache.hudi.utilities.adhoc - package org.apache.hudi.utilities.adhoc

org.apache.hudi.utilities.deltastreamer - package org.apache.hudi.utilities.deltastreamer

org.apache.hudi.utilities.exception - package org.apache.hudi.utilities.exception

org.apache.hudi.utilities.keygen - package org.apache.hudi.utilities.keygen

org.apache.hudi.utilities.perf - package org.apache.hudi.utilities.perf

org.apache.hudi.utilities.schema - package org.apache.hudi.utilities.schema

org.apache.hudi.utilities.sources - package org.apache.hudi.utilities.sources

org.apache.hudi.utilities.sources.helpers - package org.apache.hudi.utilities.sources.helpers

org.apache.hudi.utilities.transform - package org.apache.hudi.utilities.transform

outputPath - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

overallTimerName - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics

P

parallelism - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

parallelism - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

parallelism - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

parseSchema(FileSystem, String) - Static method in class org.apache.hudi.utilities.UtilHelpers

Parse Schema from file

PARTITION_FORMATTER - Static variable in class org.apache.hudi.utilities.HDFSParquetImporter

partitionKey - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

partitionPath - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

payloadClassName - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

printOutput - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

props - Variable in class org.apache.hudi.utilities.sources.Source

propsFilePath - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

propsFilePath - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

propsFilePath - Variable in class org.apache.hudi.utilities.HoodieCleaner.Config

propsFilePath - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

R

readConfig(FileSystem, Path, List<String>) - Static method in class org.apache.hudi.utilities.UtilHelpers

readConfig(InputStream) - Static method in class org.apache.hudi.utilities.UtilHelpers

reportDir - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

retry - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

retry - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

rocksDBPath - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

RowBasedSchemaProvider - Class in org.apache.hudi.utilities.schema

RowBasedSchemaProvider(StructType) - Constructor for class org.apache.hudi.utilities.schema.RowBasedSchemaProvider

rowKey - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

RowSource - Class in org.apache.hudi.utilities.sources

RowSource(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.RowSource

run() - Method in class org.apache.hudi.utilities.adhoc.UpgradePayloadFromUberToApache

run() - Method in class org.apache.hudi.utilities.HoodieCleaner

run(JavaSparkContext) - Method in class org.apache.hudi.utilities.HoodieCompactionAdminTool

Executes one of compaction admin operations

run(JavaSparkContext) - Method in class org.apache.hudi.utilities.HoodieWithTimelineServer

run() - Method in class org.apache.hudi.utilities.perf.TimelineServerPerf

runLookups(JavaSparkContext, List<String>, SyncableFileSystemView, int, int) - Method in class org.apache.hudi.utilities.perf.TimelineServerPerf

runSchedule - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

S

saveDelta() - Method in class org.apache.hudi.utilities.HiveIncrementalPuller

SchedulerConfGenerator - Class in org.apache.hudi.utilities.deltastreamer

Utility Class to generate Spark Scheduling allocation file.

SchedulerConfGenerator() - Constructor for class org.apache.hudi.utilities.deltastreamer.SchedulerConfGenerator

schemaFile - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

schemaFile - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

SchemaProvider - Class in org.apache.hudi.utilities.schema

Class to provide schema for reading data and also writing into a Hoodie table

SchemaProvider(TypedProperties, JavaSparkContext) - Constructor for class org.apache.hudi.utilities.schema.SchemaProvider

schemaProviderClassName - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

SchemaRegistryProvider - Class in org.apache.hudi.utilities.schema

Obtains latest schema from the Confluent/Kafka schema-registry https://github.com/confluentinc/schema-registry

SchemaRegistryProvider(TypedProperties, JavaSparkContext) - Constructor for class org.apache.hudi.utilities.schema.SchemaRegistryProvider

SchemaRegistryProvider.Config - Class in org.apache.hudi.utilities.schema

Configs supported

sendRequest(String, int) - Method in class org.apache.hudi.utilities.HoodieWithTimelineServer

serverHost - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

serverPort - Variable in class org.apache.hudi.utilities.HoodieWithTimelineServer.Config

serverPort - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

setupWriteClient() - Method in class org.apache.hudi.utilities.deltastreamer.DeltaSync

Note that depending on configs and source-type, schemaProvider could either be eagerly or lazily created.

shutdownGracefully() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

skipValidation - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

snapshot(JavaSparkContext, String, String, boolean) - Method in class org.apache.hudi.utilities.HoodieSnapshotCopier

Source<T> - Class in org.apache.hudi.utilities.sources

Represents a source from which we can tail data.

Source(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider) - Constructor for class org.apache.hudi.utilities.sources.Source

Source(TypedProperties, JavaSparkContext, SparkSession, SchemaProvider, Source.SourceType) - Constructor for class org.apache.hudi.utilities.sources.Source

Source.SourceType - Enum in org.apache.hudi.utilities.sources

sourceClassName - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

sourceDb - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

SourceFormatAdapter - Class in org.apache.hudi.utilities.deltastreamer

Adapts data-format provided by the source to the data-format required by the client (DeltaStreamer)

SourceFormatAdapter(Source) - Constructor for class org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter

sourceLimit - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

sourceOrderingField - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

sourceTable - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

SPARK_SCHEDULER_ALLOCATION_FILE_KEY - Static variable in class org.apache.hudi.utilities.deltastreamer.SchedulerConfGenerator

SPARK_SCHEDULER_MODE_KEY - Static variable in class org.apache.hudi.utilities.deltastreamer.SchedulerConfGenerator

sparkContext - Variable in class org.apache.hudi.utilities.sources.Source

sparkMaster - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

sparkMaster - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

sparkMaster - Variable in class org.apache.hudi.utilities.HoodieCleaner.Config

sparkMaster - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

sparkMaster - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

sparkMaster - Variable in class org.apache.hudi.utilities.HoodieWithTimelineServer.Config

sparkMaster - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

sparkMemory - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

sparkMemory - Variable in class org.apache.hudi.utilities.HoodieCompactionAdminTool.Config

sparkMemory - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

sparkMemory - Variable in class org.apache.hudi.utilities.HoodieWithTimelineServer.Config

sparkSession - Variable in class org.apache.hudi.utilities.sources.Source

SqlQueryBasedTransformer - Class in org.apache.hudi.utilities.transform

A transformer that allows a sql-query template be used to transform the source before writing to Hudi data-set.

SqlQueryBasedTransformer() - Constructor for class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer

srcPath - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

start(Function<Boolean, Boolean>) - Method in class org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService

Start the service.

startService() - Method in class org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService

Service implementation

startService() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.AsyncCompactService

Start Compaction Service

startService() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

startService() - Method in class org.apache.hudi.utilities.HoodieWithTimelineServer

storageType - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

strategyClassName - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

strToOffsets(String) - Static method in class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen.CheckpointUtils

Reconstruct checkpoint from string.

sync() - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

Main method to start syncing

syncOnce() - Method in class org.apache.hudi.utilities.deltastreamer.DeltaSync

Run one round of delta sync and return new compaction instant if one got scheduled

T

tableName - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

tableName - Variable in class org.apache.hudi.utilities.HoodieCompactor.Config

tableType - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

targetBasePath - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

targetDb - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

targetPath - Variable in class org.apache.hudi.utilities.HDFSParquetImporter.Config

targetTable - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

targetTableName - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

TimelineServerPerf - Class in org.apache.hudi.utilities.perf

TimelineServerPerf(TimelineServerPerf.Config) - Constructor for class org.apache.hudi.utilities.perf.TimelineServerPerf

TimelineServerPerf.Config - Class in org.apache.hudi.utilities.perf

TimestampBasedKeyGenerator - Class in org.apache.hudi.utilities.keygen

Key generator, that relies on timestamps for partitioning field.

TimestampBasedKeyGenerator(TypedProperties) - Constructor for class org.apache.hudi.utilities.keygen.TimestampBasedKeyGenerator

tmpDb - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

topicName - Variable in class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen

totalNewMessages(OffsetRange[]) - Static method in class org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen.CheckpointUtils

Transformer - Interface in org.apache.hudi.utilities.transform

Transform source to target dataset before writing

transformerClassName - Variable in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config

U

updateDeltaStreamerMetrics(long, long) - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics

UpgradePayloadFromUberToApache - Class in org.apache.hudi.utilities.adhoc

This is an one-time use class meant for migrating the configuration for "hoodie.compaction.payload.class" in .hoodie/hoodie.properties from com.uber.hoodie to org.apache.hudi It takes in a file containing base-paths for a set of hudi datasets and does the migration

UpgradePayloadFromUberToApache(UpgradePayloadFromUberToApache.Config) - Constructor for class org.apache.hudi.utilities.adhoc.UpgradePayloadFromUberToApache

UpgradePayloadFromUberToApache.Config - Class in org.apache.hudi.utilities.adhoc

UtilHelpers - Class in org.apache.hudi.utilities

Bunch of helper methods

UtilHelpers() - Constructor for class org.apache.hudi.utilities.UtilHelpers

V

validate(String, String) - Method in class org.apache.hudi.utilities.HDFSParquetImporter.FormatValidator

validateInstantTime(Row, String, String, String) - Static method in class org.apache.hudi.utilities.sources.helpers.IncrSourceHelper

Validate instant time seen in the incoming row

valueOf(String) - Static method in enum org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Operation

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum org.apache.hudi.utilities.HoodieCompactionAdminTool.Operation

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum org.apache.hudi.utilities.sources.Source.SourceType

Returns the enum constant of this type with the specified name.

values() - Static method in enum org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Operation

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum org.apache.hudi.utilities.HoodieCompactionAdminTool.Operation

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum org.apache.hudi.utilities.sources.Source.SourceType

Returns an array containing the constants of this enum type, in the order they are declared.

viewStorageType - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

W

waitForManualQueries - Variable in class org.apache.hudi.utilities.perf.TimelineServerPerf.Config

waitTillPendingCompactionsReducesTo(int) - Method in class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.AsyncCompactService

Wait till outstanding pending compactions reduces to the passed in value

Y

yarnQueueName - Variable in class org.apache.hudi.utilities.HiveIncrementalPuller.Config

A B C D E F G H I J K L M N O P R S T U V W Y