org.apache.hudi.integ.testsuite.dag.nodes

Class ValidateDatasetNode

java.lang.Object

org.apache.hudi.integ.testsuite.dag.nodes.DagNode<Boolean>

org.apache.hudi.integ.testsuite.dag.nodes.BaseValidateDatasetNode

org.apache.hudi.integ.testsuite.dag.nodes.ValidateDatasetNode

All Implemented Interfaces:

Comparable<DagNode<Boolean>>

public class ValidateDatasetNode
extends BaseValidateDatasetNode

This validation node uses spark datasource for comparison purposes.

Field Summary

Fields inherited from class org.apache.hudi.integ.testsuite.dag.nodes.DagNode

childNodes, config, log, parentNodes, result

Constructor Summary

Constructors

Constructor and Description
ValidateDatasetNode(DeltaConfig.Config config)

Method Summary

Modifier and Type	Method and Description
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	getDatasetToValidate(org.apache.spark.sql.SparkSession session, ExecutionContext context, org.apache.spark.sql.types.StructType inputSchema)
org.slf4j.Logger	getLogger()

Methods inherited from class org.apache.hudi.integ.testsuite.dag.nodes.BaseValidateDatasetNode

execute

Methods inherited from class org.apache.hudi.integ.testsuite.dag.nodes.DagNode

addChildNode, addParentNode, clone, compareTo, equals, getChildNodes, getConfig, getName, getParentNodes, getResult, hashCode, isCompleted, setCompleted, setParentNodes

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ValidateDatasetNode

public ValidateDatasetNode(DeltaConfig.Config config)

Method Detail

getLogger

public org.slf4j.Logger getLogger()

Specified by:

getLogger in class BaseValidateDatasetNode

Returns:

Logger instance to use.

getDatasetToValidate

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> getDatasetToValidate(org.apache.spark.sql.SparkSession session,
                                                                                   ExecutionContext context,
                                                                                   org.apache.spark.sql.types.StructType inputSchema)

Specified by:

getDatasetToValidate in class BaseValidateDatasetNode

Parameters:

session - SparkSession instance to use.

context - ExecutionContext instance to use.

inputSchema - input schema in StructType

Returns:

data in Dataset to validate.