Package org.apache.hop.pipeline.transform

Interface ITransformMeta

All Known Subinterfaces:

IInputFileMeta

All Known Implementing Classes:

AbstractTransformMeta, BaseFileInputMeta, BaseTransformMeta, DummyMeta, InjectorMeta, Missing, TransformWithMappingMeta

public interface ITransformMeta

This interface allows custom transforms to talk to Hop. The ITransformMeta is the main Java interface that a plugin implements. The responsibilities of the implementing class are listed below:

• Keep track of the transform settings
  The implementing class typically keeps track of transform settings using private fields with corresponding getters and setters. The dialog class implementing ITransformDialog is using the getters and setters to copy the user supplied configuration in and out of the dialog.

  The following interface methods also fall into the area of maintaining settings:

  void setDefault()

  This method is called every time a new transform is created and should allocate or set the transform configuration to sensible defaults. The values set here will be used by HopGui when a new transform is created. This is often a good place to ensure that the transform’s settings are initialized to non-null values. Null values can be cumbersome to deal with in serialization and dialog population, so most Apache Hop transform implementations stick to non-null values for all transform settings.

  public Object clone()

  This method is called when a transform is duplicated in HopGui. It needs to return a deep copy of this transform meta object. It is essential that the implementing class creates proper deep copies if the transform configuration is stored in modifiable objects, such as lists or custom helper objects. See org.apache.hop.pipeline.transforms.rowgenerator.RowGeneratorMeta.clone() for an example on creating a deep copy.

• Serialize transform settings
  The plugin needs to be able to serialize its settings to XML . The interface methods are as follows.

  public String getXml()

  This method is called by Apache Hop whenever a transform needs to serialize its settings to XML. It is called when saving a pipeline in HopGui. The method returns an XML string, containing the serialized transform settings. The string contains a series of XML tags, typically one tag per setting. The helper class org.apache.hop.core.xml.XmlHandler is typically used to construct the XML string.

  public void loadXml(...)

  This method is called by Apache Hop whenever a transform needs to read its settings from XML. The XML node containing the transform's settings is passed in as an argument. Again, the helper class org.apache.hop.core.xml.XmlHandler is typically used to conveniently read the transform settings from the XML node.

• Provide instances of other plugin classes
  The ITransformMeta plugin class is the main class tying in with the rest of Apache Hop architecture. It is responsible for supplying instances of the other plugin classes implementing ITransformDialog, ITransform and ITransformData. The following methods cover these responsibilities. Each of the method’s implementation is typically constructing a new instance of the corresponding class forwarding the passed in arguments to the constructor. The methods are as follows.

  public ITransformDialog getDialog(...)
  public ITransform getTransform(...)
  public ITransformData getTransformData()
  

  Each of the above methods returns a new instance of the plugin class implementing ITransformDialog, ITransform and ITransformData.

• Report the transform’s changes to the row stream Apache Hop needs to know how a transform affects the row structure. A transform may be adding or removing fields, as well as modifying the metadata of a field. The method implementing this aspect of a transform plugin is getFields().

  public void getFields(...)

  Given a description of the input rows, the plugin needs to modify it to match the structure for its output fields. The implementation modifies the passed in IRowMeta object to reflect any changes to the row stream. Typically a transform adds fields to the row structure, which is done by creating ValueMeta objects (default implementation of IValueMeta), and appending them to the IRowMeta object. The section Working with Fields goes into deeper detail on IValueMeta.

• Validate transform settings
  HopGui supports a “validate pipeline” feature, which triggers a self-check of all transforms. Apache Hop invokes the check() method of each transform on the canvas allowing each transform to validate its settings.

  public void check()

  Each transform has the opportunity to validate its settings and verify that the configuration given by the user is reasonable. In addition to that a transform typically checks if it is connected to preceding or following transforms, if the nature of the transform requires that kind of connection. An input transform may expect to not have a preceding transform for example. The check method passes in a list of check remarks that the method should append its validation results to. HopGui then displays the list of remarks collected from the transforms, allowing the user to take corrective action in case of validation warnings or errors. Given a description of the input rows, the plugin needs to modify it to match the structure for its output fields. The implementation modifies the passed in IRowMeta object to reflect any changes to the row stream. Typically a transform adds fields to the row structure, which is done by creating ValueMeta objects (default implementation of IValueMeta), and appending them to the IRowMeta object. The section Working with Fields goes into deeper detail on IValueMeta.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	ITransformMeta.TransformFactory

Method Summary

Modifier and Type	Method	Description
void	analyseImpact(IVariables variables, List<DatabaseImpact> impact, PipelineMeta pipelineMeta, TransformMeta transformMeta, IRowMeta prev, String[] input, String[] output, IRowMeta info, IHopMetadataProvider metadataProvider)	Each transform must be able to report on the impact it has on a database, table field, etc.
void	cancelQueries()	Call this to cancel trailing database queries (too long running, etc)
void	check(List<ICheckResult> remarks, PipelineMeta pipelineMeta, TransformMeta transformMeta, IRowMeta prev, String[] input, String[] output, IRowMeta info, IVariables variables, IHopMetadataProvider metadataProvider)	Checks the settings of this transform and puts the findings in a remarks List.
default boolean	cleanAfterHopFromRemove()	Action remove hop exiting this transform
default boolean	cleanAfterHopFromRemove(TransformMeta toTransform)	Action remove hop exiting this transform
default boolean	cleanAfterHopToRemove(TransformMeta fromTransform)	Action remove hop entering this transform.
Object	clone()	Make an exact copy of this transform, make sure to explicitly copy Collections etc.
void	convertIOMetaToTransformNames()	Convert defined IOMeta information to transform metadata.
ITransform	createTransform(TransformMeta transformMeta, ITransformData data, int copyNr, PipelineMeta pipelineMeta, Pipeline pipeline)
ITransformData	createTransformData()	Create a new instance of the appropriate data class.
boolean	excludeFromCopyDistributeVerification()	This method is added to exclude certain transforms from copy/distribute checking.
boolean	excludeFromRowLayoutVerification()	This method is added to exclude certain transforms from layout checking.
String	exportResources(IVariables variables, Map<String,ResourceDefinition> definitions, IResourceNaming iResourceNaming, IHopMetadataProvider metadataProvider)
String	getActiveReferencedObjectDescription()
String	getDialogClassName()	 * This returns the expected name for the dialog that edits a action.
void	getFields(IRowMeta inputRowMeta, String name, IRowMeta[] info, TransformMeta nextTransform, IVariables variables, IHopMetadataProvider metadataProvider)	Gets the fields.
List<IStream>	getOptionalStreams()
TransformMeta	getParentTransformMeta()
String[]	getReferencedObjectDescriptions()
IRowMeta	getRequiredFields(IVariables variables)	The natural way of data flow in a pipeline is source-to-target.
List<ResourceReference>	getResourceDependencies(IVariables variables, TransformMeta transformMeta)	Get a list of all the resource dependencies that the transform is depending on.
SqlStatement	getSqlStatements(IVariables variables, PipelineMeta pipelineMeta, TransformMeta transformMeta, IRowMeta prev, IHopMetadataProvider metadataProvider)	Standard method to return an SqlStatement object with Sql statements that the transform needs in order to work correctly.
IRowMeta	getTableFields(IVariables variables)
ITransformIOMeta	getTransformIOMeta()	Returns the Input/Output metadata for this transform.
String	getXml()	Get the XML that represents the values in this transform
void	handleStreamSelection(IStream stream)	When an optional stream is selected, this method is called to handled the ETL metadata implications of that.
boolean	hasChanged()
boolean[]	isReferencedObjectEnabled()
IHasFilename	loadReferencedObject(int index, IHopMetadataProvider metadataProvider, IVariables variables)	Load the referenced object
void	loadXml(Node transformNode, IHopMetadataProvider metadataProvider)	Load the values for this transform from an XML Node
default boolean	passDataToServletOutput()	True if the transform passes it's result data straight to the servlet output.
void	resetTransformIoMeta()	For transforms that handle dynamic input (info) or output (target) streams, it is useful to be able to force the recreate the TransformIoMeta definition.
void	searchInfoAndTargetTransforms(List<TransformMeta> transforms)	Change transform names into transform objects to allow them to be name-changed etc.
void	setChanged()
void	setDefault()	Set default values
void	setParentTransformMeta(TransformMeta parentTransformMeta)	Provide original lineage for this metadata object
boolean	supportsErrorHandling()

Method Detail

setDefault

void setDefault()

Set default values

getFields

void getFields(IRowMeta inputRowMeta,
               String name,
               IRowMeta[] info,
               TransformMeta nextTransform,
               IVariables variables,
               IHopMetadataProvider metadataProvider)
        throws HopTransformException

Gets the fields.

Parameters:

inputRowMeta - the input row meta that is modified in this method to reflect the output row metadata of the transform

name - Name of the transform to use as input for the origin field in the values

info - Fields used as extra lookup information

nextTransform - the next transform that is targeted

variables - the variables The variable variables to use to replace variables

metadataProvider - the MetaStore to use to load additional external data or metadata impacting the output fields

Throws:

HopTransformException - the hop transform exception

getXml

String getXml()
       throws HopException

Get the XML that represents the values in this transform

Returns:

the XML that represents the metadata in this transform

Throws:

HopException - in case there is a conversion or XML encoding error

loadXml

void loadXml(Node transformNode,
             IHopMetadataProvider metadataProvider)
      throws HopXmlException

Load the values for this transform from an XML Node

Parameters:

transformNode - the Node to get the info from

metadataProvider - the metadata to optionally load external reference metadata from

Throws:

HopXmlException - When an unexpected XML error occurred. (malformed etc.)

check

void check(List<ICheckResult> remarks,
           PipelineMeta pipelineMeta,
           TransformMeta transformMeta,
           IRowMeta prev,
           String[] input,
           String[] output,
           IRowMeta info,
           IVariables variables,
           IHopMetadataProvider metadataProvider)

Checks the settings of this transform and puts the findings in a remarks List.

Parameters:

remarks - The list to put the remarks in @see org.apache.hop.core.CheckResult

transformMeta - The transformMeta to help checking

prev - The fields coming from the previous transform

input - The input transform names

output - The output transform names

info - The fields that are used as information by the transform

variables - the variable variables to resolve variable expressions with

metadataProvider - the MetaStore to use to load additional external data or metadata impacting the output fields

clone

Object clone()

Make an exact copy of this transform, make sure to explicitly copy Collections etc.

Returns:

an exact copy of this transform

getTableFields

IRowMeta getTableFields(IVariables variables)

Parameters:

variables -

Returns:

The fields used by this transform, this is being used for the Impact analyses.

excludeFromRowLayoutVerification

boolean excludeFromRowLayoutVerification()

This method is added to exclude certain transforms from layout checking.

excludeFromCopyDistributeVerification

boolean excludeFromCopyDistributeVerification()

This method is added to exclude certain transforms from copy/distribute checking.

createTransform

ITransform createTransform(TransformMeta transformMeta,
                           ITransformData data,
                           int copyNr,
                           PipelineMeta pipelineMeta,
                           Pipeline pipeline)

createTransformData

ITransformData createTransformData()

Create a new instance of the appropriate data class. This data class implements the ITransformData. It basically contains the persisting data that needs to live on, even if a worker thread is terminated.

Returns:

The appropriate ITransformData class.

analyseImpact

void analyseImpact(IVariables variables,
                   List<DatabaseImpact> impact,
                   PipelineMeta pipelineMeta,
                   TransformMeta transformMeta,
                   IRowMeta prev,
                   String[] input,
                   String[] output,
                   IRowMeta info,
                   IHopMetadataProvider metadataProvider)
            throws HopTransformException

Each transform must be able to report on the impact it has on a database, table field, etc.

Parameters:

variables - the variables to resolve expression with

impact - The list of impacts @see org.apache.hop.pipelineMeta.DatabaseImpact

pipelineMeta - The pipeline information

transformMeta - The transform information

prev - The fields entering this transform

input - The previous transform names

output - The output transform names

info - The fields used as information by this transform

metadataProvider - the MetaStore to use to load additional external data or metadata impacting the output fields

Throws:

HopTransformException

getSqlStatements

SqlStatement getSqlStatements(IVariables variables,
                              PipelineMeta pipelineMeta,
                              TransformMeta transformMeta,
                              IRowMeta prev,
                              IHopMetadataProvider metadataProvider)
                       throws HopTransformException

Standard method to return an SqlStatement object with Sql statements that the transform needs in order to work correctly. This can mean "create table", "create index" statements but also "alter table ... add/drop/modify" statements.

Parameters:

variables - the variables to resolve expressions with

pipelineMeta - PipelineMeta object containing the complete pipeline

transformMeta - TransformMeta object containing the complete transform

prev - Row containing meta-data for the input fields (no data)

metadataProvider - the MetaStore to use to load additional external data or metadata impacting the output fields

Returns:

The Sql Statements for this transform. If nothing has to be done, the SqlStatement.getSql() == null. @see SqlStatement

Throws:

HopTransformException

cancelQueries

void cancelQueries()
            throws HopDatabaseException

Call this to cancel trailing database queries (too long running, etc)

Throws:

HopDatabaseException

getRequiredFields

IRowMeta getRequiredFields(IVariables variables)
                    throws HopException

The natural way of data flow in a pipeline is source-to-target. However, this makes mapping to target tables difficult to do. To help out here, we supply information to the pipeline meta-data model about which fields are required for a transform. This allows us to automate certain tasks like the mapping to pre-defined tables. The Table Output transform in this case will output the fields in the target table using this method.

Parameters:

variables - the variable variables to reference

Returns:

the required fields for this transforms metadata.

Throws:

HopException - in case the required fields can't be determined.

supportsErrorHandling

boolean supportsErrorHandling()

Returns:

true if this transform supports error "reporting" on rows: the ability to send rows to a certain target transform.

getResourceDependencies

List<ResourceReference> getResourceDependencies(IVariables variables,
                                                TransformMeta transformMeta)

Get a list of all the resource dependencies that the transform is depending on.

Parameters:

transformMeta -

Returns:

a list of all the resource dependencies that the transform is depending on

exportResources

String exportResources(IVariables variables,
                       Map<String,ResourceDefinition> definitions,
                       IResourceNaming iResourceNaming,
                       IHopMetadataProvider metadataProvider)
                throws HopException

Parameters:

variables - the variable variables to use

definitions -

iResourceNaming -

metadataProvider - the metadataProvider in which non-hop metadata could reside.

Returns:

the filename of the exported resource

Throws:

HopException

getParentTransformMeta

TransformMeta getParentTransformMeta()

Returns:

The TransformMeta object to which this metadata class belongs. With this, we can see to which pipeline metadata (etc) this metadata pertains to. (hierarchy)

setParentTransformMeta

void setParentTransformMeta(TransformMeta parentTransformMeta)

Provide original lineage for this metadata object

Parameters:

parentTransformMeta - the parent transform metadata container object

getTransformIOMeta

ITransformIOMeta getTransformIOMeta()

Returns the Input/Output metadata for this transform.

getOptionalStreams

List<IStream> getOptionalStreams()

Returns:

The list of optional input streams. It allows the user to select f rom a list of possible actions like "New target transform"

handleStreamSelection

void handleStreamSelection(IStream stream)

When an optional stream is selected, this method is called to handled the ETL metadata implications of that.

Parameters:

stream - The optional stream to handle.

resetTransformIoMeta

void resetTransformIoMeta()

For transforms that handle dynamic input (info) or output (target) streams, it is useful to be able to force the recreate the TransformIoMeta definition. By default this definition is cached.

searchInfoAndTargetTransforms

void searchInfoAndTargetTransforms(List<TransformMeta> transforms)

Change transform names into transform objects to allow them to be name-changed etc.

Parameters:

transforms - the transforms to reference

convertIOMetaToTransformNames

void convertIOMetaToTransformNames()

Convert defined IOMeta information to transform metadata. This is the inverse of method searchInfoAndTargetTransforms()

setChanged

void setChanged()

hasChanged

boolean hasChanged()

getReferencedObjectDescriptions

String[] getReferencedObjectDescriptions()

Returns:

The objects referenced in the transform, like a mapping, a pipeline, a workflow, ...

isReferencedObjectEnabled

boolean[] isReferencedObjectEnabled()

Returns:

true for each referenced object that is enabled or has a valid reference definition.

getActiveReferencedObjectDescription

String getActiveReferencedObjectDescription()

Returns:

A description of the active referenced object in a pipeline. Null if nothing special needs to be done or if the active metadata isn't different from design.

loadReferencedObject

IHasFilename loadReferencedObject(int index,
                                  IHopMetadataProvider metadataProvider,
                                  IVariables variables)
                           throws HopException

Load the referenced object

Parameters:

index - the referenced object index to load (in case there are multiple references)

metadataProvider - the MetaStore to use

variables - the variable variables to use

Returns:

the referenced object once loaded

Throws:

HopException

cleanAfterHopFromRemove

default boolean cleanAfterHopFromRemove()

Action remove hop exiting this transform

Returns:

transform was changed

cleanAfterHopFromRemove

default boolean cleanAfterHopFromRemove(TransformMeta toTransform)

Action remove hop exiting this transform

Parameters:

toTransform - the to-transform of the hop being removed

Returns:

transform was changed

cleanAfterHopToRemove

default boolean cleanAfterHopToRemove(TransformMeta fromTransform)

Action remove hop entering this transform.

Sometimes, in addition to the hops themselves, a Transform has internal configuration referencing some of the transforms that connect to it (to identify the main transform or the one to be used for lookup, for instance). If the hop being deleted comes from one of those transforms, the references to them should be removed.

Parameters:

fromTransform - the from-transform of the hop being removed

Returns:

transform was changed

passDataToServletOutput

default boolean passDataToServletOutput()

True if the transform passes it's result data straight to the servlet output. See exposing Hop data over a web service

Returns:

True if the transform passes it's result data straight to the servlet output, false otherwise

getDialogClassName

String getDialogClassName()

 * This returns the expected name for the dialog that edits a action. The expected name is in the org.apache.hop.ui  * tree and has a class name that is the name of the action with 'Dialog' added to the end.  *

 * e.g. if the action is org.apache.hop.workflow.actions.zipfile.JobEntryZipFile the dialog would be  * org.apache.hop.ui.workflow.actions.zipfile.JobEntryZipFileDialog  *

 * If the dialog class for a action does not match this pattern it should override this method and return the  * appropriate class name  *  * @return full class name of the dialog