Package org.apache.hop.pipeline.engine

Interface IPipelineEngine<T extends PipelineMeta>

    • Method Detail

      • getPipelineMeta

        T getPipelineMeta()

      • setPipelineMeta

        void setPipelineMeta​(T t)

      • getPluginId

        String getPluginId()

      • setPluginId

        void setPluginId​(String pluginId)

      • createDefaultPipelineEngineRunConfiguration

        IPipelineEngineRunConfiguration createDefaultPipelineEngineRunConfiguration()
        Ask the engine to generate a default pipeline engine configuration for this engine
        Returns:
        a new pipeline engine run configuration

      • getEngineCapabilities

        PipelineEngineCapabilities getEngineCapabilities()
        See which operations are supported by the pipeline engine.
        Returns:
        The engine capabilities

      • execute

        void execute()
      throws HopException
        Executes the object/subject: calls prepareExecution and startThreads in sequence.
        Throws:
        HopException - if the object execution could not be prepared (initialized)

      • prepareExecution

        void prepareExecution()
               throws HopException
        Prepares for execution. This includes setting the parameters as well as preparing anything needed in terms of resources, threads,.... It runs initialization on all the work that is needed to be done up-front.
        Throws:
        HopException - In case of an initialization error

      • isPreparing

        boolean isPreparing()
        Returns:
        true if the engine is preparing execution

      • isReadyToStart

        boolean isReadyToStart()
        Check to see if the pipeline engine is ready to start threads.
        Returns:
        True if the pipeline engine was prepared and is ready to start.

      • startThreads

        void startThreads()
           throws HopException
        Starts the engine itself after prepareExecution.
        Throws:
        HopException - If there is an engine error during execution.

      • getEngineMetrics

        EngineMetrics getEngineMetrics()
        Collect metrics from the various components in the engine.
        Returns:
        The engine metrics

      • getEngineMetrics

        EngineMetrics getEngineMetrics​(String componentName,
                               int copyNr)
        Get the engine metrics for a specific component name and/or copy nr
        Parameters:
        componentName - the name of the component or null for all components
        copyNr - The copy nr to select or a negative number for all components
        Returns:
        The engine metrics for the given

      • cleanup

        void cleanup()
        This method performs any cleanup operations on the various sub-components of the engine after execution.

      • waitUntilFinished

        void waitUntilFinished()
        Wait until the execution is done.

      • pipelineCompleted

        void pipelineCompleted()
                throws HopException
        This method is called when all the finished listeners are executed and when the pipeline has completed.
        Throws:
        HopException

      • stopAll

        void stopAll()
        Stops all parts of the execution from running and alerts any registered listeners.

      • hasHaltedComponents

        boolean hasHaltedComponents()
        See if there are any halted components in the engine: actions, auditing, ...
        Returns:
        True if there are halted components

      • isRunning

        boolean isRunning()
        Indicates whether or not the engine is running
        Returns:
        True is the engine is running

      • isStopped

        boolean isStopped()
        Indicates whether or not the engine is stopped.
        Returns:
        True is the engine execution has stopped

      • pauseExecution

        void pauseExecution()
        Pauses the execution (all components).

      • resumeExecution

        void resumeExecution()
        Resume the execution (all components).

      • getErrors

        int getErrors()
        Returns:
        a number larger than 0 in case of errors

      • isFinished

        boolean isFinished()
        See if the engine has finished executing.
        Returns:
        True if the execution has finished

      • isPaused

        boolean isPaused()
        See if the engine is paused
        Returns:
        true if the engine is paused

      • firePipelineExecutionStartedListeners

        void firePipelineExecutionStartedListeners()
                                    throws HopException
        Throws:
        HopException

      • firePipelineExecutionFinishedListeners

        void firePipelineExecutionFinishedListeners()
                                     throws HopException
        Throws:
        HopException

      • firePipelineExecutionStoppedListeners

        void firePipelineExecutionStoppedListeners()
                                    throws HopException
        Throws:
        HopException

      • getComponentLogText

        String getComponentLogText​(String componentName,
                           int copyNr)
        Retrieve the logging text of a particular component in the engine
        Parameters:
        componentName - The name of the component (transform)
        copyNr - The copy number if multiple components are running in parallel (0 based).
        Returns:
        The logging text

      • getComponents

        List<IEngineComponent> getComponents()
        Get a list of components
        Returns:
        The list of components in the engine

      • getComponentCopies

        List<IEngineComponent> getComponentCopies​(String name)
        Find all component copies by name
        Parameters:
        name - The name of the component to look for (transform)
        Returns:
        The list of components found

      • findComponent

        IEngineComponent findComponent​(String name,
                               int copyNr)
        Find a component by name and copy nr
        Parameters:
        name - The name of the component to look for (transform)
        copyNr - The copy number to match
        Returns:
        The component or null if nothing was found

      • getLogChannel

        ILogChannel getLogChannel()
        Gets the log channel interface for the pipeline.
        Returns:
        the log channel

      • setLogChannel

        void setLogChannel​(ILogChannel log)
        Sets the log channel. For testing only
        Parameters:
        log -

      • setParent

        void setParent​(ILoggingObject parent)
        Sets the parent logging object.
        Parameters:
        parent - the new parent

      • setParentWorkflow

        void setParentWorkflow​(IWorkflowEngine<WorkflowMeta> parentWorkflow)
        Inform the pipeline that it's being executed as part of a workflow
        Parameters:
        parentWorkflow -

      • setParentPipeline

        void setParentPipeline​(IPipelineEngine parentPipeline)
        Inform the pipeline that it's being executed as part of another pipeline
        Parameters:
        parentPipeline -

      • setPreviousResult

        void setPreviousResult​(Result previousResult)
        Inform the pipeline about a previous execution result in a workflow or pipeline
        Parameters:
        previousResult -

      • getResult

        Result getResult()
        Get the result of the execution after it's done, a resume
        Returns:
        The Result of the execution of the object

      • setLogLevel

        void setLogLevel​(LogLevel logLevel)

      • setPreview

        @Deprecated(since="2.0")
void setPreview​(boolean preview)
        Deprecated.
        Temporary until we find a better way to preview/debug
        Parameters:
        preview -

      • isPreview

        @Deprecated(since="2.0")
boolean isPreview()
        Deprecated.
        Temporary until we find a better way to preview/debug
        Returns:

      • retrieveComponentOutput

        void retrieveComponentOutput​(IVariables variables,
                             String componentName,
                             int copyNr,
                             int nrRows,
                             IPipelineComponentRowsReceived rowsReceived)
                      throws HopException
        Retrieve output rows from a component copy. Pass the rows to the rows received lambda.
        Parameters:
        variables -
        componentName -
        copyNr -
        nrRows -
        rowsReceived -
        Throws:
        HopException

      • getParentPipeline

        IPipelineEngine<T> getParentPipeline()
        Determine the pipeline engine which is executing this pipeline engine.
        Returns:
        The executing pipeline or null if none is known.

      • getParentWorkflow

        IWorkflowEngine<WorkflowMeta> getParentWorkflow()
        Determine the workflow engine which is executing this pipeline engine.
        Returns:
        The executing workflow of null if none is known.

      • isSafeModeEnabled

        boolean isSafeModeEnabled()
        True if the engine is doing extra validations at runtime to detect possible issues with data types and so on.
        Returns:
        True if safe mode is enabled.

      • findRowSet

        IRowSet findRowSet​(String fromTransformName,
                   int fromTransformCopy,
                   String toTransformName,
                   int toTransformCopy)
        For engines that support it we allow the retrieval of a rowset from one transform copy to another
        Parameters:
        fromTransformName -
        fromTransformCopy -
        toTransformName -
        toTransformCopy -
        Returns:
        The rowset if one was found.
        Throws:
        RuntimeException - in case the engine doesn't support this operation.

      • isFeedbackShown

        @Deprecated(since="2.0")
boolean isFeedbackShown()
        Deprecated.
        True if feedback need to be given every X rows
        Returns:
        True if feedback needs to be given

      • getFeedbackSize

        @Deprecated(since="2.0")
int getFeedbackSize()
        Deprecated.
        The feedback size in rows
        Returns:
        The feedback size in rows

      • getPreviousResult

        Result getPreviousResult()
        Get the execution result of a previous execution in a workflow
        Returns:
        the previous execution result

      • getExecutionStartDate

        Date getExecutionStartDate()
        Returns:
        The start date of the pipeline execution

      • getExecutionEndDate

        Date getExecutionEndDate()
        Returns:
        The end date of the pipeline preparation

      • addActiveSubPipeline

        void addActiveSubPipeline​(String transformName,
                          IPipelineEngine executorPipeline)
        Add an active sub-pipeline to allow drill-down in the GUI
        Parameters:
        transformName -
        executorPipeline -

      • getActiveSubPipeline

        IPipelineEngine getActiveSubPipeline​(String subPipelineName)
        Get the active sub-pipeline with the given name
        Parameters:
        subPipelineName -
        Returns:
        The active pipeline engine or null if it was not found

      • addActiveSubWorkflow

        void addActiveSubWorkflow​(String subWorkflowName,
                          IWorkflowEngine<WorkflowMeta> subWorkflow)
        Add an active sub-workflow to allow drill-down in the GUI
        Parameters:
        subWorkflowName -
        subWorkflow -

      • getActiveSubWorkflow

        IWorkflowEngine<WorkflowMeta> getActiveSubWorkflow​(String subWorkflowName)
        Get the active sub-workflow with the given name
        Parameters:
        subWorkflowName -
        Returns:
        The active workflow or null if nothing was found

      • setInternalHopVariables

        void setInternalHopVariables​(IVariables var)
        Make the engine define internal hop variables based on the given variables, the filename and so on
        Parameters:
        var - TODO get rid of this method, internal variables should be available

      • setContainerId

        void setContainerId​(String containerId)
        The unique ID this pipeline engine has while executing in a container like a web-server and so on. When we have multiple pipelines running with the same name
        Parameters:
        containerId - The unique ID to identify this executing engine with

      • getStatusDescription

        String getStatusDescription()
        Returns:
        Get a status description of the state of the engine (running, stopped, finished, paused, halted, ...)

      • getExtensionDataMap

        Map<String,​Object> getExtensionDataMap()
        A map in which the various transforms and plugins can store objects to remember during the lifecycle of the pipeline. TODO This is obviously harder to do in a clustered or remote execution environment. As such it's not clear that it makes sense to leave this method here over time
        Specified by:
        getExtensionDataMap in interface IExtensionData
        Returns:
        The extension data map to use

      • addExecutionDataSampler

        <Store extends IExecutionDataSamplerStore,​Sampler extends IExecutionDataSampler<Store>> void addExecutionDataSampler​(Sampler sampler)
                                                                                                                    throws HopException
        Plugins can add individual data samplers which will act upon the rows passing through the transforms in the pipeline. The execution information collected at this level will be sent to the execution information location if any is defined in the run configuration used to execute this pipeline.
        Parameters:
        sampler - The sampler to use
        Throws:
        HopException