Class PipelineMeta

    • Field Detail

      • XML_TAG

        public static final String XML_TAG
        A constant specifying the tag value for the XML node of the pipeline.
        See Also:
        Constant Field Values
      • transforms

        protected List<TransformMeta> transforms
        The list of transforms associated with the pipeline.
      • pipelineStatus

        protected int pipelineStatus
        The status of the pipeline.
      • changedTransforms

        protected boolean changedTransforms
        Indicators for changes in transforms, databases, hops, and notes.
      • changedHops

        protected boolean changedHops
      • previousResult

        protected Result previousResult
        The previous result.
      • transformFieldsCache

        protected Map<String,​IRowMeta> transformFieldsCache
        The transforms fields cache.
      • descTypeUndo

        public static final String[] descTypeUndo
        A list of localized strings corresponding to string descriptions of the undo/redo actions.
      • XML_TAG_INFO

        protected static final String XML_TAG_INFO
        A constant specifying the tag value for the XML node of the pipeline information.
        See Also:
        Constant Field Values
      • XML_TAG_ORDER

        public static final String XML_TAG_ORDER
        A constant specifying the tag value for the XML node of the order of transforms.
        See Also:
        Constant Field Values
      • XML_TAG_NOTEPADS

        public static final String XML_TAG_NOTEPADS
        A constant specifying the tag value for the XML node of the notes.
        See Also:
        Constant Field Values
      • XML_TAG_PARAMETERS

        public static final String XML_TAG_PARAMETERS
        A constant specifying the tag value for the XML node of the pipeline parameters.
        See Also:
        Constant Field Values
      • XML_TAG_TRANSFORM_ERROR_HANDLING

        public static final String XML_TAG_TRANSFORM_ERROR_HANDLING
        A constant specifying the tag value for the XML node of the transforms' error-handling information.
        See Also:
        Constant Field Values
    • Constructor Detail

      • PipelineMeta

        public PipelineMeta()
        Builds a new empty pipeline. The pipeline will have default logging capability and no variables, and all internal meta-data is cleared to defaults.
      • PipelineMeta

        public PipelineMeta​(String fname,
                            IHopMetadataProvider metadataProvider,
                            IVariables parentVariableSpace)
                     throws HopXmlException,
                            HopMissingPluginsException
        Parses a file containing the XML that describes the pipeline.
        Parameters:
        fname - The filename
        metadataProvider - the metadata store to reference (or null if there is none)
        parentVariableSpace - the parent variable variables to use during PipelineMeta construction
        Throws:
        HopXmlException - if any errors occur during parsing of the specified file
        HopMissingPluginsException - in case missing plugins were found (details are in the exception in that case)
    • Method Detail

      • compare

        public int compare​(PipelineMeta t1,
                           PipelineMeta t2)
        Compares two pipeline on name and filename. The comparison algorithm is as follows:
        1. The first pipeline's filename is checked first; if it has none, the pipeline is generated If the second pipeline is not generated, -1 is returned.
        2. If the pipelines are both generated, the pipelines' names are compared. If the first pipeline has no name and the second one does, a -1 is returned. If the opposite is true, 1 is returned.
        3. If they both have names they are compared as strings. If the result is non-zero it is returned. Otherwise the repository directories are compared using the same technique of checking empty values and then performing a string comparison, returning any non-zero result.
        Specified by:
        compare in interface Comparator<PipelineMeta>
        Parameters:
        t1 - the first pipeline to compare
        t2 - the second pipeline to compare
        Returns:
        0 if the two pipelines are equal, 1 or -1 depending on the values (see description above)
      • equals

        public boolean equals​(Object obj)
        Checks whether this pipeline's meta-data object is equal to the specified object. If the specified object is not an instance of PipelineMeta, false is returned. Otherwise the method returns whether a call to compare() indicates equality (i.e. compare(this, (PipelineMeta)obj)==0).
        Specified by:
        equals in interface Comparator<PipelineMeta>
        Overrides:
        equals in class Object
        Parameters:
        obj - the obj
        Returns:
        true, if successful
        See Also:
        compare(PipelineMeta, PipelineMeta), Object.equals(Object)
      • clone

        public Object clone()
        Clones the pipeline meta-data object.
        Overrides:
        clone in class Object
        Returns:
        a clone of the pipeline meta-data object
        See Also:
        Object.clone()
      • realClone

        public Object realClone​(boolean doClear)
        Perform a real clone of the pipeline meta-data object, including cloning all lists and copying all values. If the doClear parameter is true, the clone will be cleared of ALL values before the copy. If false, only the copied fields will be cleared.
        Parameters:
        doClear - Whether to clear all of the clone's data before copying from the source object
        Returns:
        a real clone of the calling object
      • clear

        public void clear()
        Clears the pipeline's meta-data, including the lists of databases, transforms, hops, notes, dependencies, partition schemas, hop servers, and cluster schemas. Logging information and timeouts are reset to defaults, and recent connection info is cleared.
        Overrides:
        clear in class AbstractMeta
      • addTransform

        public void addTransform​(TransformMeta transformMeta)
        Add a new transform to the pipeline. Also marks that the pipeline's transforms have changed.
        Parameters:
        transformMeta - The meta-data for the transform to be added.
      • addOrReplaceTransform

        public void addOrReplaceTransform​(TransformMeta transformMeta)
        Add a new transform to the pipeline if that transform didn't exist yet. Otherwise, replace the transform. This method also marks that the pipeline's transforms have changed.
        Parameters:
        transformMeta - The meta-data for the transform to be added.
      • addPipelineHop

        public void addPipelineHop​(PipelineHopMeta hi)
        Add a new hop to the pipeline. The hop information (source and target transforms, e.g.) should be configured in the PipelineHopMeta object before calling addPipelineHop(). Also marks that the pipeline's hops have changed.
        Parameters:
        hi - The hop meta-data to be added.
      • addTransform

        public void addTransform​(int p,
                                 TransformMeta transformMeta)
        Add a new transform to the pipeline at the specified index. This method sets the transform's parent pipeline to the this pipeline, and marks that the pipelines' transforms have changed.
        Parameters:
        p - The index into the transform list
        transformMeta - The transform to be added.
      • addPipelineHop

        public void addPipelineHop​(int p,
                                   PipelineHopMeta hi)
        Add a new hop to the pipeline on a certain location (i.e. the specified index). Also marks that the pipeline's hops have changed.
        Parameters:
        p - the index into the hop list
        hi - The hop to be added.
      • getTransforms

        public List<TransformMeta> getTransforms()
        Get a list of defined transforms in this pipeline.
        Returns:
        an ArrayList of defined transforms.
      • getTransform

        public TransformMeta getTransform​(int i)
        Retrieves a transform on a certain location (i.e. the specified index).
        Parameters:
        i - The index into the transforms list.
        Returns:
        The desired transform's meta-data.
      • getPipelineHops

        public List<PipelineHopMeta> getPipelineHops()
        Get a list of defined hops in this pipeline.
        Returns:
        a list of defined hops.
      • getPipelineHop

        public PipelineHopMeta getPipelineHop​(int i)
        Retrieves a hop on a certain location (i.e. the specified index).
        Parameters:
        i - The index into the hops list.
        Returns:
        The desired hop's meta-data.
      • removeTransform

        public void removeTransform​(int i)
        Removes a transform from the pipeline on a certain location (i.e. the specified index). Also marks that the pipeline's transforms have changed.
        Parameters:
        i - The index
      • removePipelineHop

        public void removePipelineHop​(int i)
        Removes a hop from the pipeline on a certain location (i.e. the specified index). Also marks that the pipeline's hops have changed.
        Parameters:
        i - The index into the hops list
      • removePipelineHop

        public void removePipelineHop​(PipelineHopMeta hop)
        Removes a hop from the pipeline. Also marks that the pipeline's hops have changed.
        Parameters:
        hop - The hop to remove from the list of hops
      • nrTransforms

        public int nrTransforms()
        Gets the number of transforms in the pipeline.
        Returns:
        The number of transforms in the pipeline.
      • nrPipelineHops

        public int nrPipelineHops()
        Gets the number of hops in the pipeline.
        Returns:
        The number of hops in the pipeline.
      • nrTransformChangeListeners

        public int nrTransformChangeListeners()
        Gets the number of transformChangeListeners in the pipeline.
        Returns:
        The number of transformChangeListeners in the pipeline.
      • setTransform

        public void setTransform​(int i,
                                 TransformMeta transformMeta)
        Changes the content of a transform on a certain position. This is accomplished by setting the transform's metadata at the specified index to the specified meta-data object. The new transform's parent pipeline is updated to be this pipeline.
        Parameters:
        i - The index into the transforms list
        transformMeta - The transform meta-data to set
      • setPipelineHop

        public void setPipelineHop​(int i,
                                   PipelineHopMeta hi)
        Changes the content of a hop on a certain position. This is accomplished by setting the hop's metadata at the specified index to the specified meta-data object.
        Parameters:
        i - The index into the hops list
        hi - The hop meta-data to set
      • getUsedTransforms

        public List<TransformMeta> getUsedTransforms()
        Gets the list of used transforms, which are the transforms that are connected by hops.
        Returns:
        a list with all the used transforms
      • findTransform

        public TransformMeta findTransform​(String name)
        Searches the list of transforms for a transform with a certain name.
        Parameters:
        name - The name of the transform to look for
        Returns:
        The transform information or null if no nothing was found.
      • findTransform

        public TransformMeta findTransform​(String name,
                                           TransformMeta exclude)
        Searches the list of transforms for a transform with a certain name while excluding one transform.
        Parameters:
        name - The name of the transform to look for
        exclude - The transform information to exclude.
        Returns:
        The transform information or null if nothing was found.
      • findPipelineHop

        public PipelineHopMeta findPipelineHop​(String name)
        Searches the list of hops for a hop with a certain name.
        Parameters:
        name - The name of the hop to look for
        Returns:
        The hop information or null if nothing was found.
      • findPipelineHopFrom

        public PipelineHopMeta findPipelineHopFrom​(TransformMeta fromTransform)
        Search all hops for a hop where a certain transform is at the start.
        Parameters:
        fromTransform - The transform at the start of the hop.
        Returns:
        The hop or null if no hop was found.
      • findPipelineHop

        public PipelineHopMeta findPipelineHop​(PipelineHopMeta hi)
        Find a certain hop in the pipeline.
        Parameters:
        hi - The hop information to look for.
        Returns:
        The hop or null if no hop was found.
      • findPipelineHop

        public PipelineHopMeta findPipelineHop​(TransformMeta from,
                                               TransformMeta to)
        Search all hops for a hop where a certain transform is at the start and another is at the end.
        Parameters:
        from - The transform at the start of the hop.
        to - The transform at the end of the hop.
        Returns:
        The hop or null if no hop was found.
      • findPipelineHop

        public PipelineHopMeta findPipelineHop​(TransformMeta from,
                                               TransformMeta to,
                                               boolean disabledToo)
        Search all hops for a hop where a certain transform is at the start and another is at the end.
        Parameters:
        from - The transform at the start of the hop.
        to - The transform at the end of the hop.
        disabledToo - the disabled too
        Returns:
        The hop or null if no hop was found.
      • findPipelineHopTo

        public PipelineHopMeta findPipelineHopTo​(TransformMeta toTransform)
        Search all hops for a hop where a certain transform is at the end.
        Parameters:
        toTransform - The transform at the end of the hop.
        Returns:
        The hop or null if no hop was found.
      • isTransformInformative

        public boolean isTransformInformative​(TransformMeta thisTransform,
                                              TransformMeta prevTransform)
        Determines whether or not a certain transform is informative. This means that the previous transform is sending information to this transform, but only informative. This means that this transform is using the information to process the actual stream of data. We use this in StreamLookup, TableInput and other types of transforms.
        Parameters:
        thisTransform - The transform that is receiving information.
        prevTransform - The transform that is sending information
        Returns:
        true if prevTransform if informative for thisTransform.
      • findPreviousTransforms

        public List<TransformMeta> findPreviousTransforms​(TransformMeta transformMeta)
        Get the list of previous transforms for a certain reference transform. This includes the info transforms.
        Parameters:
        transformMeta - The reference transform
        Returns:
        The list of the preceding transforms, including the info transforms.
      • findPreviousTransforms

        public List<TransformMeta> findPreviousTransforms​(TransformMeta transformMeta,
                                                          boolean info)
        Get the previous transforms on a certain location taking into account the transforms being informational or not.
        Parameters:
        transformMeta - The name of the transform
        info - true if we only want the informational transforms.
        Returns:
        The list of the preceding transforms
      • getInfoTransform

        public TransformMeta[] getInfoTransform​(TransformMeta transformMeta)
        Get the informational transforms for a certain transform. An informational transform is a transform that provides information for lookups, etc.
        Parameters:
        transformMeta - The name of the transform
        Returns:
        An array of the informational transforms found
      • findNrInfoTransforms

        public int findNrInfoTransforms​(TransformMeta transformMeta)
        Find the the number of informational transforms for a certain transform.
        Parameters:
        transformMeta - The transform
        Returns:
        The number of informational transforms found.
      • getPrevInfoFields

        public IRowMeta getPrevInfoFields​(IVariables variables,
                                          String transformName)
                                   throws HopTransformException
        Find the informational fields coming from an informational transform into the transform specified.
        Parameters:
        transformName - The name of the transform
        Returns:
        A row containing fields with origin.
        Throws:
        HopTransformException - the hop transform exception
      • getPrevInfoFields

        public IRowMeta getPrevInfoFields​(IVariables variables,
                                          TransformMeta transformMeta)
                                   throws HopTransformException
        Find the informational fields coming from an informational transform into the transform specified.
        Parameters:
        transformMeta - The receiving transform
        Returns:
        A row containing fields with origin.
        Throws:
        HopTransformException - the hop transform exception
      • getPrevTransforms

        public TransformMeta[] getPrevTransforms​(TransformMeta transformMeta)
        Retrieve an array of preceding transforms for a certain destination transform. This includes the info transforms.
        Parameters:
        transformMeta - The destination transform
        Returns:
        An array containing the preceding transforms.
      • getPrevTransformNames

        public String[] getPrevTransformNames​(String transformName)
        Retrieve an array of succeeding transform names for a certain originating transform name.
        Parameters:
        transformName - The originating transform name
        Returns:
        An array of succeeding transform names
      • getPrevTransformNames

        public String[] getPrevTransformNames​(TransformMeta transformMeta)
        Retrieve an array of preceding transforms for a certain destination transform.
        Parameters:
        transformMeta - The destination transform
        Returns:
        an array of preceding transform names.
      • findNextTransforms

        public List<TransformMeta> findNextTransforms​(TransformMeta transformMeta)
        Retrieve a list of succeeding transforms for a certain originating transform.
        Parameters:
        transformMeta - The originating transform
        Returns:
        an array of succeeding transforms.
      • getNextTransformNames

        public String[] getNextTransformNames​(TransformMeta transformMeta)
        Retrieve an array of succeeding transform names for a certain originating transform.
        Parameters:
        transformMeta - The originating transform
        Returns:
        an array of succeeding transform names.
      • getTransform

        public TransformMeta getTransform​(int x,
                                          int y,
                                          int iconsize)
        Find the transform that is located on a certain point on the canvas, taking into account the icon size.
        Parameters:
        x - the x-coordinate of the point queried
        y - the y-coordinate of the point queried
        iconsize - the iconsize
        Returns:
        The transform information if a transform is located at the point. Otherwise, if no transform was found: null.
      • partOfPipelineHop

        public boolean partOfPipelineHop​(TransformMeta transformMeta)
        Determines whether or not a certain transform is part of a hop.
        Parameters:
        transformMeta - The transform queried
        Returns:
        true if the transform is part of a hop.
      • getTransformFields

        public IRowMeta getTransformFields​(IVariables variables,
                                           String transformName)
                                    throws HopTransformException
        Returns the fields that are emitted by a certain transform name.
        Parameters:
        transformName - The transformName of the transform to be queried.
        Returns:
        A row containing the fields emitted.
        Throws:
        HopTransformException - the hop transform exception
      • getTransformFields

        public IRowMeta getTransformFields​(IVariables variables,
                                           TransformMeta transformMeta)
                                    throws HopTransformException
        Returns the fields that are emitted by a certain transform.
        Parameters:
        transformMeta - The transform to be queried.
        Returns:
        A row containing the fields emitted.
        Throws:
        HopTransformException - the hop transform exception
      • getTransformFields

        public IRowMeta getTransformFields​(IVariables variables,
                                           TransformMeta[] transformMeta)
                                    throws HopTransformException
        Gets the fields for each of the specified transforms and merges them into a single set
        Parameters:
        transformMeta - the transform meta
        Returns:
        an interface to the transform fields
        Throws:
        HopTransformException - the hop transform exception
      • getTransformFields

        public IRowMeta getTransformFields​(IVariables variables,
                                           TransformMeta transformMeta,
                                           IProgressMonitor monitor)
                                    throws HopTransformException
        Returns the fields that are emitted by a certain transform.
        Parameters:
        transformMeta - The transform to be queried.
        monitor - The progress monitor for progress dialog. (null if not used!)
        Returns:
        A row containing the fields emitted.
        Throws:
        HopTransformException - the hop transform exception
      • getTransformFields

        public IRowMeta getTransformFields​(IVariables variables,
                                           TransformMeta transformMeta,
                                           TransformMeta targetTransform,
                                           IProgressMonitor monitor)
                                    throws HopTransformException
        Returns the fields that are emitted by a certain transform.
        Parameters:
        transformMeta - The transform to be queried.
        targetTransform - the target transform
        monitor - The progress monitor for progress dialog. (null if not used!)
        Returns:
        A row containing the fields emitted.
        Throws:
        HopTransformException - the hop transform exception
      • getPrevTransformFields

        public IRowMeta getPrevTransformFields​(IVariables variables,
                                               String transformName)
                                        throws HopTransformException
        Find the fields that are entering a transform with a certain name.
        Parameters:
        transformName - The name of the transform queried
        Returns:
        A row containing the fields (w/ origin) entering the transform
        Throws:
        HopTransformException - the hop transform exception
      • getPrevTransformFields

        public IRowMeta getPrevTransformFields​(IVariables variables,
                                               TransformMeta transformMeta)
                                        throws HopTransformException
        Find the fields that are entering a certain transform.
        Parameters:
        transformMeta - The transform queried
        Returns:
        A row containing the fields (w/ origin) entering the transform
        Throws:
        HopTransformException - the hop transform exception
      • getPrevTransformFields

        public IRowMeta getPrevTransformFields​(IVariables variables,
                                               TransformMeta transformMeta,
                                               IProgressMonitor monitor)
                                        throws HopTransformException
        Find the fields that are entering a certain transform.
        Parameters:
        transformMeta - The transform queried
        monitor - The progress monitor for progress dialog. (null if not used!)
        Returns:
        A row containing the fields (w/ origin) entering the transform
        Throws:
        HopTransformException - the hop transform exception
      • getThisTransformFields

        public IRowMeta getThisTransformFields​(IVariables variables,
                                               String transformName,
                                               IRowMeta row)
                                        throws HopTransformException
        Return the fields that are emitted by a transform with a certain name.
        Parameters:
        transformName - The name of the transform that's being queried.
        row - A row containing the input fields or an empty row if no input is required.
        Returns:
        A Row containing the output fields.
        Throws:
        HopTransformException - the hop transform exception
      • getThisTransformFields

        public IRowMeta getThisTransformFields​(IVariables variables,
                                               TransformMeta transformMeta,
                                               TransformMeta nextTransform,
                                               IRowMeta row)
                                        throws HopTransformException
        Returns the fields that are emitted by a transform.
        Parameters:
        transformMeta - : The TransformMeta object that's being queried
        nextTransform - : if non-null this is the next transform that's call back to ask what's being sent
        row - : A row containing the input fields or an empty row if no input is required.
        Returns:
        A Row containing the output fields.
        Throws:
        HopTransformException - the hop transform exception
      • getThisTransformFields

        public IRowMeta getThisTransformFields​(IVariables variables,
                                               TransformMeta transformMeta,
                                               TransformMeta nextTransform,
                                               IRowMeta row,
                                               IProgressMonitor monitor)
                                        throws HopTransformException
        Returns the fields that are emitted by a transform.
        Parameters:
        transformMeta - : The TransformMeta object that's being queried
        nextTransform - : if non-null this is the next transform that's call back to ask what's being sent
        row - : A row containing the input fields or an empty row if no input is required.
        monitor - the monitor
        Returns:
        A Row containing the output fields.
        Throws:
        HopTransformException - the hop transform exception
      • isUsingPartitionSchema

        public boolean isUsingPartitionSchema​(PartitionSchema partitionSchema)
        Checks if the pipeline is using the specified partition schema.
        Parameters:
        partitionSchema - the partition schema
        Returns:
        true if the pipeline is using the partition schema, false otherwise
      • indexOfPipelineHop

        public int indexOfPipelineHop​(PipelineHopMeta hi)
        Finds the location (index) of the specified hop.
        Parameters:
        hi - The hop queried
        Returns:
        The location of the hop, or -1 if nothing was found.
      • indexOfTransform

        public int indexOfTransform​(TransformMeta transformMeta)
        Finds the location (index) of the specified transform.
        Parameters:
        transformMeta - The transform queried
        Returns:
        The location of the transform, or -1 if nothing was found.
      • loadXml

        public void loadXml​(Node pipelineNode,
                            String filename,
                            IHopMetadataProvider metadataProvider,
                            IVariables variables)
                     throws HopXmlException,
                            HopMissingPluginsException
        Parses an XML DOM (starting at the specified Node) that describes the pipeline.
        Parameters:
        pipelineNode - The XML node to load from
        filename - The filename
        variables - the parent variable variables to use during PipelineMeta construction
        Throws:
        HopXmlException - if any errors occur during parsing of the specified file
        HopMissingPluginsException - in case missing plugins were found (details are in the exception in that case)
      • getPipelineHopTransforms

        public List<TransformMeta> getPipelineHopTransforms​(boolean all)
        Gets a List of all the transforms that are used in at least one active hop. These transforms will be used to execute the pipeline. The others will not be executed.
        Update 3.0 : we also add those transforms that are not linked to another hop, but have at least one remote input or output transform defined.
        Parameters:
        all - true if you want to get ALL the transforms from the pipeline, false otherwise
        Returns:
        A List of transforms
      • isTransformUsedInPipelineHops

        public boolean isTransformUsedInPipelineHops​(TransformMeta transformMeta)
        Checks if a transform has been used in a hop or not.
        Parameters:
        transformMeta - The transform queried.
        Returns:
        true if a transform is used in a hop (active or not), false otherwise
      • isAnySelectedTransformUsedInPipelineHops

        public boolean isAnySelectedTransformUsedInPipelineHops()
        Checks if any selected transform has been used in a hop or not.
        Returns:
        true if a transform is used in a hop (active or not), false otherwise
      • haveTransformsChanged

        public boolean haveTransformsChanged()
        Checks whether or not the transforms have changed.
        Returns:
        true if the transforms have been changed, false otherwise
      • haveHopsChanged

        public boolean haveHopsChanged()
        Checks whether or not any of the hops have been changed.
        Returns:
        true if a hop has been changed, false otherwise
      • hasChanged

        public boolean hasChanged()
        Checks whether or not the pipeline has changed.
        Specified by:
        hasChanged in interface IChanged
        Overrides:
        hasChanged in class AbstractMeta
        Returns:
        true if the pipeline has changed, false otherwise
      • hasLoop

        public boolean hasLoop​(TransformMeta transformMeta)
        See if there are any loops in the pipeline, starting at the indicated transform. This works by looking at all the previous transforms. If you keep going backward and find the transform, there is a loop. Both the informational and the normal transforms need to be checked for loops!
        Parameters:
        transformMeta - The transform position to start looking
        Returns:
        true if a loop has been found, false if no loop is found.
      • hasLoop

        public boolean hasLoop​(TransformMeta transformMeta,
                               TransformMeta lookup)
        Checks for loop.
        Parameters:
        transformMeta - the transformmeta
        lookup - the lookup
        Returns:
        true, if successful
      • selectAll

        public void selectAll()
        Mark all transforms in the pipeline as selected.
      • unselectAll

        public void unselectAll()
        Clear the selection of all transforms.
      • getSelectedTransformLocations

        public Point[] getSelectedTransformLocations()
        Get an array of all the selected transform locations.
        Returns:
        The selected transform locations.
      • getSelectedNoteLocations

        public Point[] getSelectedNoteLocations()
        Get an array of all the selected note locations.
        Returns:
        The selected note locations.
      • getSelectedTransforms

        public List<TransformMeta> getSelectedTransforms()
        Gets a list of the selected transforms.
        Returns:
        A list of all the selected transforms.
      • getSelectedTransformNames

        public String[] getSelectedTransformNames()
        Gets an array of all the selected transform names.
        Returns:
        An array of all the selected transform names.
      • getTransformIndexes

        public int[] getTransformIndexes​(List<TransformMeta> transforms)
        Gets an array of the locations of an array of transforms.
        Parameters:
        transforms - An array of transforms
        Returns:
        an array of the locations of an array of transforms
      • getMaximum

        public Point getMaximum()
        Gets the maximum size of the canvas by calculating the maximum location of a transform.
        Returns:
        Maximum coordinate of a transform in the pipeline + (100,100) for safety.
      • getMinimum

        public Point getMinimum()
        Gets the minimum point on the canvas of a pipeline.
        Returns:
        Minimum coordinate of a transform in the pipeline
      • getTransformNames

        public String[] getTransformNames()
        Gets the names of all the transforms.
        Returns:
        An array of transform names.
      • getTransformsArray

        public TransformMeta[] getTransformsArray()
        Gets all the transforms as an array.
        Returns:
        An array of all the transforms in the pipeline.
      • findPrevious

        public boolean findPrevious​(TransformMeta startTransform,
                                    TransformMeta transformToFind)
        Looks in the pipeline to find a transform in a previous location starting somewhere.
        Parameters:
        startTransform - The starting transform
        transformToFind - The transform to look for backward in the pipeline
        Returns:
        true if we can find the transform in an earlier location in the pipeline.
      • sortTransforms

        public void sortTransforms()
        Puts the transforms in alphabetical order.
      • sortHops

        public void sortHops()
        Sorts all the hops in the pipeline.
      • sortTransformsNatural

        public Map<TransformMeta,​Map<TransformMeta,​Boolean>> sortTransformsNatural()
        Puts the transforms in a more natural order: from start to finish. For the moment, we ignore splits and joins. Splits and joins can't be listed sequentially in any case!
        Returns:
        a map containing all the previous transforms per transform
      • sortHopsNatural

        public void sortHopsNatural()
        Sorts the hops in a natural way: from beginning to end.
      • analyseImpact

        public void analyseImpact​(IVariables variables,
                                  List<DatabaseImpact> impact,
                                  IProgressMonitor monitor)
                           throws HopTransformException
        Determines the impact of the different transforms in a pipeline on databases, tables and field.
        Parameters:
        impact - An ArrayList of DatabaseImpact objects.
        monitor - a progress monitor listener to be updated as the pipeline is analyzed
        Throws:
        HopTransformException - if any errors occur during analysis
      • getAlternativeTransformName

        public String getAlternativeTransformName​(String transformName)
        Proposes an alternative transformName when the original already exists.
        Parameters:
        transformName - The transformName to find an alternative for
        Returns:
        The suggested alternative transformName.
      • getSqlStatements

        public List<SqlStatement> getSqlStatements​(IVariables variables)
                                            throws HopTransformException
        Builds a list of all the Sql statements that this pipeline needs in order to work properly.
        Parameters:
        variables - the variables to resolve variable expressions with
        Returns:
        An ArrayList of SqlStatement objects.
        Throws:
        HopTransformException - if any errors occur during Sql statement generation
      • getSqlStatements

        public List<SqlStatement> getSqlStatements​(IVariables variables,
                                                   IProgressMonitor monitor)
                                            throws HopTransformException
        Builds a list of all the Sql statements that this pipeline needs in order to work properly.
        Parameters:
        variables - the variables to resolve variable expressions with
        monitor - a progress monitor listener to be updated as the Sql statements are generated
        Returns:
        An ArrayList of SqlStatement objects.
        Throws:
        HopTransformException - if any errors occur during Sql statement generation
      • getSqlStatementsString

        public String getSqlStatementsString​(IVariables variables)
                                      throws HopTransformException
        Get the Sql statements (needed to run this pipeline) as a single String.
        Returns:
        the Sql statements needed to run this pipeline
        Throws:
        HopTransformException - if any errors occur during Sql statement generation
      • checkTransforms

        public void checkTransforms​(List<ICheckResult> remarks,
                                    boolean onlySelected,
                                    IProgressMonitor monitor,
                                    IVariables variables,
                                    IHopMetadataProvider metadataProvider)
        Checks all the transforms and fills a List of (CheckResult) remarks.
        Parameters:
        remarks - The remarks list to add to.
        onlySelected - true to check only the selected transforms, false for all transforms
        monitor - a progress monitor listener to be updated as the Sql statements are generated
      • getPipelineVersion

        public String getPipelineVersion()
        Gets the version of the pipeline.
        Returns:
        The version of the pipeline
      • setPipelineVersion

        public void setPipelineVersion​(String pipelineVersion)
        Sets the version of the pipeline.
        Parameters:
        pipelineVersion - The new version description of the pipeline
      • setPipelineStatus

        public void setPipelineStatus​(int pipelineStatus)
        Sets the status of the pipeline.
        Parameters:
        pipelineStatus - The new status description of the pipeline
      • getPipelineStatus

        public int getPipelineStatus()
        Gets the status of the pipeline.
        Returns:
        The status of the pipeline
      • toString

        public String toString()
        Gets a textual representation of the pipeline. If its name has been set, it will be returned, otherwise the classname is returned.
        Overrides:
        toString in class Object
        Returns:
        the textual representation of the pipeline.
      • cancelQueries

        public void cancelQueries()
                           throws HopDatabaseException
        Cancels queries opened for checking & fieldprediction.
        Throws:
        HopDatabaseException - if any errors occur during query cancellation
      • getStringList

        public List<StringSearchResult> getStringList​(boolean searchTransforms,
                                                      boolean searchDatabases,
                                                      boolean searchNotes,
                                                      boolean includePasswords)
        Gets a list of all the strings used in this pipeline. The parameters indicate which collections to search and which to exclude.
        Parameters:
        searchTransforms - true if transforms should be searched, false otherwise
        searchDatabases - true if databases should be searched, false otherwise
        searchNotes - true if notes should be searched, false otherwise
        includePasswords - true if passwords should be searched, false otherwise
        Returns:
        a list of search results for strings used in the pipeline.
      • getStringList

        public List<StringSearchResult> getStringList​(boolean searchTransforms,
                                                      boolean searchDatabases,
                                                      boolean searchNotes)
        Get a list of all the strings used in this pipeline. The parameters indicate which collections to search and which to exclude.
        Parameters:
        searchTransforms - true if transforms should be searched, false otherwise
        searchDatabases - true if databases should be searched, false otherwise
        searchNotes - true if notes should be searched, false otherwise
        Returns:
        a list of search results for strings used in the pipeline.
      • getUsedVariables

        public List<String> getUsedVariables()
        Gets a list of the used variables in this pipeline.
        Returns:
        a list of the used variables in this pipeline.
      • checkRowMixingStatically

        public void checkRowMixingStatically​(IVariables variables,
                                             TransformMeta transformMeta,
                                             IProgressMonitor monitor)
                                      throws HopRowException
        Check a transform to see if there are no multiple transforms to read from. If so, check to see if the receiving rows are all the same in layout. We only want to ONLY use the DbCache for this to prevent GUI stalls.
        Parameters:
        transformMeta - the transform to check
        monitor - the monitor
        Throws:
        HopRowException - in case we detect a row mixing violation
      • setInternalNameHopVariable

        protected void setInternalNameHopVariable​(IVariables variables)
        Sets the internal name hop variable.
        Specified by:
        setInternalNameHopVariable in class AbstractMeta
        Parameters:
        variables - the new internal name hop variable
      • setInternalFilenameHopVariables

        protected void setInternalFilenameHopVariables​(IVariables variables)
        Sets the internal filename hop variables.
        Specified by:
        setInternalFilenameHopVariables in class AbstractMeta
        Parameters:
        variables - the new internal filename hop variables
      • setInternalEntryCurrentDirectory

        protected void setInternalEntryCurrentDirectory​(IVariables variables)
      • findMappingInputTransform

        public TransformMeta findMappingInputTransform​(String transformName)
                                                throws HopTransformException
        Finds the mapping input transform with the specified name. If no mapping input transform is found, null is returned
        Parameters:
        transformName - the name to search for
        Returns:
        the transform meta-data corresponding to the desired mapping input transform, or null if no transform was found
        Throws:
        HopTransformException - if any errors occur during the search
      • findMappingOutputTransform

        public TransformMeta findMappingOutputTransform​(String transformName)
                                                 throws HopTransformException
        Finds the mapping output transform with the specified name. If no mapping output transform is found, null is returned.
        Parameters:
        transformName - the name to search for
        Returns:
        the transform meta-data corresponding to the desired mapping input transform, or null if no transform was found
        Throws:
        HopTransformException - if any errors occur during the search
      • getResourceDependencies

        public List<ResourceReference> getResourceDependencies​(IVariables variables)
        Gets a list of the resource dependencies.
        Returns:
        a list of ResourceReferences
      • exportResources

        public String exportResources​(IVariables variables,
                                      Map<String,​ResourceDefinition> definitions,
                                      IResourceNaming iResourceNaming,
                                      IHopMetadataProvider metadataProvider)
                               throws HopException
        Exports the specified objects to a flat-file system, adding content with filename keys to a set of definitions. The supplied resource naming interface allows the object to name appropriately without worrying about those parts of the implementation specific details.
        Specified by:
        exportResources in interface IResourceExport
        Parameters:
        variables - the variable variables to use
        definitions -
        iResourceNaming -
        metadataProvider - the metadataProvider in which non-hop metadata could reside.
        Returns:
        the filename of the exported resource
        Throws:
        HopException - in case something goes wrong during the export
      • isCapturingTransformPerformanceSnapShots

        public boolean isCapturingTransformPerformanceSnapShots()
        Checks whether the pipeline is capturing transform performance snapshots.
        Returns:
        true if the pipeline is capturing transform performance snapshots, false otherwise
      • setCapturingTransformPerformanceSnapShots

        public void setCapturingTransformPerformanceSnapShots​(boolean capturingTransformPerformanceSnapShots)
        Sets whether the pipeline is capturing transform performance snapshots.
        Parameters:
        capturingTransformPerformanceSnapShots - true if the pipeline is capturing transform performance snapshots, false otherwise
      • getTransformPerformanceCapturingDelay

        public long getTransformPerformanceCapturingDelay()
        Gets the transform performance capturing delay.
        Returns:
        the transform performance capturing delay
      • setTransformPerformanceCapturingDelay

        public void setTransformPerformanceCapturingDelay​(long transformPerformanceCapturingDelay)
        Sets the transform performance capturing delay.
        Parameters:
        transformPerformanceCapturingDelay - the transformPerformanceCapturingDelay to set
      • getTransformPerformanceCapturingSizeLimit

        public String getTransformPerformanceCapturingSizeLimit()
        Gets the transform performance capturing size limit.
        Returns:
        the transform performance capturing size limit
      • setTransformPerformanceCapturingSizeLimit

        public void setTransformPerformanceCapturingSizeLimit​(String transformPerformanceCapturingSizeLimit)
        Sets the transform performance capturing size limit.
        Parameters:
        transformPerformanceCapturingSizeLimit - the transform performance capturing size limit to set
      • clearCaches

        public void clearCaches()
        Clears the transform fields and loop caches.
      • setPipelineType

        public void setPipelineType​(PipelineMeta.PipelineType pipelineType)
        Sets the pipeline type.
        Parameters:
        pipelineType - the pipelineType to set
      • containsTransformMeta

        public boolean containsTransformMeta​(TransformMeta transformMeta)
      • getMissingPipeline

        public List<Missing> getMissingPipeline()
      • addMissingPipeline

        public void addMissingPipeline​(Missing pipeline)
      • removeMissingPipeline

        public void removeMissingPipeline​(Missing pipeline)
      • isEmpty

        public boolean isEmpty()
      • setTransforms

        public void setTransforms​(List<TransformMeta> transforms)
        Sets transforms
        Parameters:
        transforms - value of transforms
      • setHops

        public void setHops​(List<PipelineHopMeta> hops)
        Sets hops
        Parameters:
        hops - value of hops
      • setInfo

        public void setInfo​(PipelineMetaInfo info)
        Sets info
        Parameters:
        info - value of info
      • getName

        public String getName()
        Get the name of the pipeline. If the name is synchronized with the filename, we return the base filename.
        Specified by:
        getName in interface IEngineMeta
        Specified by:
        getName in class AbstractMeta
        Returns:
        The name of the pipeline
      • getDescription

        public String getDescription()
        Gets the description of the workflow.
        Specified by:
        getDescription in class AbstractMeta
        Returns:
        The description of the workflow
      • setDescription

        public void setDescription​(String description)
        Set the description of the workflow.
        Specified by:
        setDescription in class AbstractMeta
        Parameters:
        description - The new description of the workflow
      • getExtendedDescription

        public String getExtendedDescription()
        Gets the extended description of the workflow.
        Specified by:
        getExtendedDescription in class AbstractMeta
        Returns:
        The extended description of the workflow
      • setExtendedDescription

        public void setExtendedDescription​(String extendedDescription)
        Set the description of the workflow.
        Specified by:
        setExtendedDescription in class AbstractMeta
        Parameters:
        extendedDescription - The new extended description of the workflow