org.apache.hudi.integ.testsuite.generator

类 DeltaGenerator

java.lang.Object

org.apache.hudi.integ.testsuite.generator.DeltaGenerator

所有已实现的接口:

Serializable

public class DeltaGenerator
extends Object
implements Serializable

The delta generator generates all types of workloads (insert, update) for the given configs.

另请参阅:

序列化表格

构造器概要

构造器

构造器和说明
DeltaGenerator(DFSDeltaConfig deltaOutputConfig, org.apache.spark.api.java.JavaSparkContext jsc, org.apache.spark.sql.SparkSession sparkSession, String schemaStr, org.apache.hudi.keygen.BuiltinKeyGenerator keyGenerator)

方法概要

限定符和类型	方法和说明
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	adjustRDDToGenerateExactNumUpdates(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> updates, org.apache.spark.api.java.JavaSparkContext jsc, long totalRecordsRequired)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	generateDeletes(DeltaConfig.Config config)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	generateInserts(DeltaConfig.Config operation)
org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>	generateUpdates(DeltaConfig.Config config)
Map<Integer,Long>	getAdjustedPartitionsCount(Map<Integer,Long> partitionCountMap, long recordsToRemove)
Map<Integer,Long>	getPartitionToCountMap(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> records)
org.apache.spark.api.java.JavaRDD<DeltaWriteStats>	writeRecords(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> records)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

DeltaGenerator

public DeltaGenerator(DFSDeltaConfig deltaOutputConfig,
                      org.apache.spark.api.java.JavaSparkContext jsc,
                      org.apache.spark.sql.SparkSession sparkSession,
                      String schemaStr,
                      org.apache.hudi.keygen.BuiltinKeyGenerator keyGenerator)

方法详细资料

writeRecords

public org.apache.spark.api.java.JavaRDD<DeltaWriteStats> writeRecords(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> records)

generateInserts

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> generateInserts(DeltaConfig.Config operation)

generateUpdates

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> generateUpdates(DeltaConfig.Config config)
                                                                                         throws IOException

抛出:

IOException

generateDeletes

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> generateDeletes(DeltaConfig.Config config)
                                                                                         throws IOException

抛出:

IOException

getPartitionToCountMap

public Map<Integer,Long> getPartitionToCountMap(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> records)

getAdjustedPartitionsCount

public Map<Integer,Long> getAdjustedPartitionsCount(Map<Integer,Long> partitionCountMap,
                                                    long recordsToRemove)

adjustRDDToGenerateExactNumUpdates

public org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> adjustRDDToGenerateExactNumUpdates(org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord> updates,
                                                                                                                   org.apache.spark.api.java.JavaSparkContext jsc,
                                                                                                                   long totalRecordsRequired)