org.apache.hudi.integ.testsuite.generator

Class GenericRecordFullPayloadGenerator

java.lang.Object

org.apache.hudi.integ.testsuite.generator.GenericRecordFullPayloadGenerator

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

GenericRecordPartialPayloadGenerator

public class GenericRecordFullPayloadGenerator
extends Object
implements Serializable

This is a GenericRecord payload generator that generates full generic records GenericRecord. Every field of a generic record created using this generator contains a random value.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_HOODIE_IS_DELETED_COL
static int	DEFAULT_NUM_DATE_PARTITIONS
static int	DEFAULT_PAYLOAD_SIZE
static int	DEFAULT_START_PARTITION
protected Random	random

Constructor Summary

Constructors

Constructor and Description
GenericRecordFullPayloadGenerator(org.apache.avro.Schema schema)
GenericRecordFullPayloadGenerator(org.apache.avro.Schema schema, int minPayloadSize)
GenericRecordFullPayloadGenerator(org.apache.avro.Schema schema, int minPayloadSize, int numDatePartitions, int startPartition)

Method Summary

Modifier and Type	Method and Description
protected org.apache.avro.generic.GenericRecord	convertPartial(org.apache.avro.Schema schema) Create a new GenericRecord with random values.
protected org.apache.avro.generic.GenericRecord	create(org.apache.avro.Schema schema, Set<String> partitionPathFieldNames)
protected org.apache.avro.generic.GenericRecord	generateDeleteRecord(org.apache.avro.generic.GenericRecord record) Set _hoodie_is_deleted column value to true.
int	getEstimatedFullPayloadSize()
org.apache.avro.generic.GenericRecord	getNewPayload() Create a new GenericRecord with random value according to given schema.
protected org.apache.avro.generic.GenericRecord	getNewPayload(org.apache.avro.Schema schema)
org.apache.avro.generic.GenericRecord	getNewPayload(Set<String> partitionPathFieldNames) Create a new GenericRecord with random value according to given schema.
org.apache.avro.generic.GenericRecord	getNewPayloadWithTimestamp(String tsFieldName)
protected org.apache.avro.Schema	getNonNull(org.apache.avro.Schema schema)
org.apache.avro.generic.GenericRecord	getUpdatePayload(org.apache.avro.generic.GenericRecord record, Set<String> blacklistFields) Update a given GenericRecord with random value.
org.apache.avro.generic.GenericRecord	getUpdatePayloadWithTimestamp(org.apache.avro.generic.GenericRecord record, Set<String> blacklistFields, String tsFieldName)
protected boolean	isOption(org.apache.avro.Schema schema) Check whether a schema is option.
protected static boolean	isPrimitive(org.apache.avro.Schema localSchema)
protected org.apache.avro.generic.GenericRecord	randomize(org.apache.avro.generic.GenericRecord record, Set<String> blacklistFields) Set random value to GenericRecord according to the schema type of field.
org.apache.avro.generic.GenericRecord	updateTimestamp(org.apache.avro.generic.GenericRecord record, String fieldName) Generates a sequential timestamp (daily increment), and updates the timestamp field of the record.
boolean	validate(org.apache.avro.generic.GenericRecord record) Validate whether the record match schema.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_PAYLOAD_SIZE

public static final int DEFAULT_PAYLOAD_SIZE

See Also:

Constant Field Values

DEFAULT_NUM_DATE_PARTITIONS

public static final int DEFAULT_NUM_DATE_PARTITIONS

See Also:

Constant Field Values

DEFAULT_HOODIE_IS_DELETED_COL

public static final String DEFAULT_HOODIE_IS_DELETED_COL

See Also:

Constant Field Values

DEFAULT_START_PARTITION

public static final int DEFAULT_START_PARTITION

See Also:

Constant Field Values

random

protected final Random random

Constructor Detail

GenericRecordFullPayloadGenerator

public GenericRecordFullPayloadGenerator(org.apache.avro.Schema schema)

GenericRecordFullPayloadGenerator

public GenericRecordFullPayloadGenerator(org.apache.avro.Schema schema,
                                         int minPayloadSize,
                                         int numDatePartitions,
                                         int startPartition)

GenericRecordFullPayloadGenerator

public GenericRecordFullPayloadGenerator(org.apache.avro.Schema schema,
                                         int minPayloadSize)

Method Detail

isPrimitive

protected static boolean isPrimitive(org.apache.avro.Schema localSchema)

getNewPayload

public org.apache.avro.generic.GenericRecord getNewPayload()

Create a new GenericRecord with random value according to given schema.

Returns:

GenericRecord with random value

getNewPayload

protected org.apache.avro.generic.GenericRecord getNewPayload(org.apache.avro.Schema schema)

getNewPayload

public org.apache.avro.generic.GenericRecord getNewPayload(Set<String> partitionPathFieldNames)

Create a new GenericRecord with random value according to given schema. Long fields which are specified within partitionPathFieldNames are constrained to the value of the partition for which records are being generated.

Returns:

GenericRecord with random value

getNewPayloadWithTimestamp

public org.apache.avro.generic.GenericRecord getNewPayloadWithTimestamp(String tsFieldName)

getUpdatePayloadWithTimestamp

public org.apache.avro.generic.GenericRecord getUpdatePayloadWithTimestamp(org.apache.avro.generic.GenericRecord record,
                                                                           Set<String> blacklistFields,
                                                                           String tsFieldName)

create

protected org.apache.avro.generic.GenericRecord create(org.apache.avro.Schema schema,
                                                       Set<String> partitionPathFieldNames)

getUpdatePayload

public org.apache.avro.generic.GenericRecord getUpdatePayload(org.apache.avro.generic.GenericRecord record,
                                                              Set<String> blacklistFields)

Update a given GenericRecord with random value. The fields in blacklistFields will not be updated.

Parameters:

record - GenericRecord to update

blacklistFields - Fields whose value should not be touched

Returns:

The updated GenericRecord

convertPartial

protected org.apache.avro.generic.GenericRecord convertPartial(org.apache.avro.Schema schema)

Create a new GenericRecord with random values. Not all the fields have value, it is random, and its value is random too.

Parameters:

schema - Schema to create with.

Returns:

A GenericRecord with random value.

randomize

protected org.apache.avro.generic.GenericRecord randomize(org.apache.avro.generic.GenericRecord record,
                                                          Set<String> blacklistFields)

Set random value to GenericRecord according to the schema type of field. The field in blacklist will not be set.

Parameters:

record - GenericRecord to randomize.

blacklistFields - blacklistFields where the filed will not be randomized.

Returns:

Randomized GenericRecord.

generateDeleteRecord

protected org.apache.avro.generic.GenericRecord generateDeleteRecord(org.apache.avro.generic.GenericRecord record)

Set _hoodie_is_deleted column value to true.

Parameters:

record - GenericRecord to delete.

Returns:

GenericRecord representing deleted record.

validate

public boolean validate(org.apache.avro.generic.GenericRecord record)

Validate whether the record match schema.

Parameters:

record - Record to validate.

Returns:

True if matches.

updateTimestamp

public org.apache.avro.generic.GenericRecord updateTimestamp(org.apache.avro.generic.GenericRecord record,
                                                             String fieldName)

Generates a sequential timestamp (daily increment), and updates the timestamp field of the record. Note: When generating records, number of records to be generated must be more than numDatePartitions * parallelism, to guarantee that at least numDatePartitions are created.

isOption

protected boolean isOption(org.apache.avro.Schema schema)

Check whether a schema is option. return true if it match the follows: 1. Its type is Type.UNION 2. Has two types 3. Has a NULL type.

getNonNull

protected org.apache.avro.Schema getNonNull(org.apache.avro.Schema schema)

getEstimatedFullPayloadSize

public int getEstimatedFullPayloadSize()