org.apache.hudi.integ.testsuite.dag.nodes

Class BaseValidateDatasetNode

java.lang.Object

org.apache.hudi.integ.testsuite.dag.nodes.DagNode<Boolean>

org.apache.hudi.integ.testsuite.dag.nodes.BaseValidateDatasetNode

All Implemented Interfaces:

Comparable<DagNode<Boolean>>

Direct Known Subclasses:

ValidateDatasetNode

public abstract class BaseValidateDatasetNode
extends DagNode<Boolean>

This nodes validates contents from input path are in tact with Hudi. By default no configs are required for this node. But there is an optional config "delete_input_data" that you can set for this node. If set, once validation completes, contents from inputPath are deleted. This will come in handy for long running test suites. README has more details under docker set up for usages of this node.

Field Summary

Fields inherited from class org.apache.hudi.integ.testsuite.dag.nodes.DagNode

childNodes, config, log, parentNodes, result

Constructor Summary

Constructors

Constructor and Description
BaseValidateDatasetNode(DeltaConfig.Config config)

Method Summary

Modifier and Type	Method and Description
void	execute(ExecutionContext context, int curItrCount) Execute the DagNode.
abstract org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	getDatasetToValidate(org.apache.spark.sql.SparkSession session, ExecutionContext context, org.apache.spark.sql.types.StructType inputSchema)
abstract org.slf4j.Logger	getLogger()

Methods inherited from class org.apache.hudi.integ.testsuite.dag.nodes.DagNode

addChildNode, addParentNode, clone, compareTo, equals, getChildNodes, getConfig, getName, getParentNodes, getResult, hashCode, isCompleted, setCompleted, setParentNodes

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BaseValidateDatasetNode

public BaseValidateDatasetNode(DeltaConfig.Config config)

Method Detail

getLogger

public abstract org.slf4j.Logger getLogger()

Returns:

Logger instance to use.

getDatasetToValidate

public abstract org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> getDatasetToValidate(org.apache.spark.sql.SparkSession session,
                                                                                            ExecutionContext context,
                                                                                            org.apache.spark.sql.types.StructType inputSchema)

Parameters:

session - SparkSession instance to use.

context - ExecutionContext instance to use.

inputSchema - input schema in StructType

Returns:

data in Dataset to validate.

execute

public void execute(ExecutionContext context,
                    int curItrCount)
             throws Exception

Description copied from class: DagNode

Execute the DagNode.

Specified by:

execute in class DagNode<Boolean>

Parameters:

context - The context needed for an execution of a node.

curItrCount - iteration count for executing the node.

Throws:

Exception - Thrown if the execution failed.