org.apache.hudi

Class QuickstartUtils.DataGenerator

java.lang.Object

org.apache.hudi.QuickstartUtils.DataGenerator

Enclosing class:

QuickstartUtils

public static class QuickstartUtils.DataGenerator
extends Object

Constructor Summary

Constructors

Constructor and Description
DataGenerator()
DataGenerator(String[] partitionPaths)

Method Summary

Modifier and Type	Method and Description
void	close()
List<String>	generateDeletes(List<org.apache.spark.sql.Row> rows) Generates delete records for the passed in rows.
static org.apache.avro.generic.GenericRecord	generateGenericRecord(String rowKey, String riderName, String driverName, long timestamp)
List<HoodieRecord>	generateInserts(Integer n) Generates new inserts, uniformly across the partition paths above.
Stream<HoodieRecord>	generateInsertsStream(String randomString, Integer n) Generates new inserts, uniformly across the partition paths above.
static OverwriteWithLatestAvroPayload	generateRandomValue(HoodieKey key, String riderDriverSuffix) Generates a new avro record of the above schema format, retaining the key if optionally provided.
List<HoodieRecord>	generateUniqueUpdates(Integer n) Generates new updates, one for each of the keys above list
HoodieRecord	generateUpdateRecord(HoodieKey key, String randomString)
List<HoodieRecord>	generateUpdates(Integer n) Generates new updates, randomly distributed across the keys above.
int	getNumExistingKeys()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataGenerator

public DataGenerator()

DataGenerator

public DataGenerator(String[] partitionPaths)

Method Detail

getNumExistingKeys

public int getNumExistingKeys()

generateGenericRecord

public static org.apache.avro.generic.GenericRecord generateGenericRecord(String rowKey,
                                                                          String riderName,
                                                                          String driverName,
                                                                          long timestamp)

generateRandomValue

public static OverwriteWithLatestAvroPayload generateRandomValue(HoodieKey key,
                                                                 String riderDriverSuffix)
                                                          throws IOException

Generates a new avro record of the above schema format, retaining the key if optionally provided. The riderDriverSuffix string is a random String to simulate updates by changing the rider driver fields for records belonging to the same commit. It is purely used for demo purposes. In real world, the actual updates are assumed to be provided based on the application requirements.

Throws:

IOException

generateInsertsStream

public Stream<HoodieRecord> generateInsertsStream(String randomString,
                                                  Integer n)

Generates new inserts, uniformly across the partition paths above. It also updates the list of existing keys.

generateInserts

public List<HoodieRecord> generateInserts(Integer n)
                                   throws IOException

Generates new inserts, uniformly across the partition paths above. It also updates the list of existing keys.

Throws:

IOException

generateUpdateRecord

public HoodieRecord generateUpdateRecord(HoodieKey key,
                                         String randomString)
                                  throws IOException

Throws:

IOException

generateUpdates

public List<HoodieRecord> generateUpdates(Integer n)

Generates new updates, randomly distributed across the keys above. There can be duplicates within the returned list

Parameters:

n - Number of updates (including dups)

Returns:

list of hoodie record updates

generateUniqueUpdates

public List<HoodieRecord> generateUniqueUpdates(Integer n)

Generates new updates, one for each of the keys above list

Parameters:

n - Number of updates (must be no more than number of existing keys)

Returns:

list of hoodie record updates

generateDeletes

public List<String> generateDeletes(List<org.apache.spark.sql.Row> rows)

Generates delete records for the passed in rows.

Parameters:

rows - List of Rows for which delete record need to be generated

Returns:

list of hoodie records to delete

close

public void close()