Package org.apache.hop.pipeline
Class Pipeline
- java.lang.Object
-
- org.apache.hop.pipeline.Pipeline
-
- All Implemented Interfaces:
IExecutor,IExtensionData,IHasLogChannel,ILoggingObject,INamedParameterDefinitions,INamedParameters,IVariables,IPipelineEngine<PipelineMeta>
- Direct Known Subclasses:
LocalPipelineEngine,LocalSinglePipelineEngine
public abstract class Pipeline extends Object implements IVariables, INamedParameters, IHasLogChannel, ILoggingObject, IExecutor, IExtensionData, IPipelineEngine<PipelineMeta>
This class represents the information and operations associated with the execution of a Pipeline. It loads, instantiates, initializes, runs, and monitors the execution of the pipeline contained in the specified PipelineMeta object.
-
-
Field Summary
Fields Modifier and Type Field Description protected String[]argumentsThe command line arguments for the pipeline.static StringCONFIGURATION_IN_EXPORT_FILENAMEConstant specifying a filename containing XML to inject into a ZIP file created during resource export.protected StringcontainerObjectIdThe container object id.protected List<IExecutionDataSampler<? extends IExecutionDataSamplerStore>>dataSamplersPlugins can use this to add additional data samplers to the pipeline.protected booleanfeedbackShownWhether the feedback is shown.protected intfeedbackSizeThe feedback size.protected ILogChannellogThe log channel interface.protected intlogCommitSizeThe log commit size.protected LogLevellogLevelThe log level.protected IHopMetadataProvidermetadataProviderThe MetaStore to usestatic IEngineMetricMETRIC_BUFFER_INstatic IEngineMetricMETRIC_BUFFER_OUTstatic IEngineMetricMETRIC_ERRORstatic IEngineMetricMETRIC_FLUSH_BUFFERstatic IEngineMetricMETRIC_INITstatic IEngineMetricMETRIC_INPUTstatic StringMETRIC_NAME_BUFFER_INstatic StringMETRIC_NAME_BUFFER_OUTstatic StringMETRIC_NAME_ERRORstatic StringMETRIC_NAME_FLUSH_BUFFERstatic StringMETRIC_NAME_INITstatic StringMETRIC_NAME_INPUTstatic StringMETRIC_NAME_OUTPUTstatic StringMETRIC_NAME_READstatic StringMETRIC_NAME_REJECTEDstatic StringMETRIC_NAME_UPDATEDstatic StringMETRIC_NAME_WRITTENstatic IEngineMetricMETRIC_OUTPUTstatic IEngineMetricMETRIC_READstatic IEngineMetricMETRIC_REJECTEDstatic IEngineMetricMETRIC_UPDATEDstatic IEngineMetricMETRIC_WRITTENprotected PipelineMetapipelineMetaThe pipeline metadata to execute.protected PipelineRunConfigurationpipelineRunConfigurationprotected StringpluginIdprotected List<ResultFile>resultFilesprotected List<RowMetaAndData>resultRowsList<IRowSet>rowsetsA list of all the row sets.protected introwSetSizestatic StringSTRING_FINISHEDConstant indicating a pipeline status of Finished.static StringSTRING_FINISHED_WITH_ERRORSConstant indicating a pipeline status of Finished (with errors).static StringSTRING_HALTINGConstant indicating a pipeline status of Halting.static StringSTRING_INITIALIZINGConstant indicating a pipeline status of Initializing.static StringSTRING_PAUSEDConstant indicating a pipeline status of Paused.static StringSTRING_PREPARINGConstant indicating a pipeline status of Preparing for execution.static StringSTRING_RUNNINGConstant indicating a pipeline status of Running.static StringSTRING_STOPPEDConstant indicating a pipeline status of Stopped.static StringSTRING_STOPPED_WITH_ERRORSConstant indicating a pipeline status of Stopped (with errors).static StringSTRING_WAITINGConstant indicating a pipeline status of Waiting.static intTYPE_DISP_1_1Constant indicating a dispatch type of 1-to-1.static intTYPE_DISP_1_NConstant indicating a dispatch type of 1-to-N.static intTYPE_DISP_N_1Constant indicating a dispatch type of N-to-1.static intTYPE_DISP_N_MConstant indicating a dispatch type of N-to-M.static intTYPE_DISP_N_NConstant indicating a dispatch type of N-to-N.-
Fields inherited from interface org.apache.hop.pipeline.engine.IPipelineEngine
PIPELINE_IN_PREVIEW_MODE
-
-
Constructor Summary
Constructors Constructor Description Pipeline()Instantiates a new pipeline.Pipeline(PipelineMeta pipelineMeta)Initializes a pipeline from pipeline meta-data defined in memory.Pipeline(PipelineMeta pipelineMeta, IVariables variables, ILoggingObject parent)Initializes a pipeline from pipeline meta-data defined in memory.Pipeline(Parent parent, String name, String filename, IHopMetadataProvider metadataProvider)Instantiates a new pipeline using any of the provided parameters including the variable bindings, a name and a filename.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidactivateParameters(IVariables variables)Activates all parameters by setting their values.voidaddActiveSubPipeline(String subPipelineName, IPipelineEngine subPipeline)Add an active sub-pipeline to allow drill-down in the GUIvoidaddActiveSubWorkflow(String subWorkflowName, IWorkflowEngine<WorkflowMeta> subWorkflow)Add an active sub-workflow to allow drill-down in the GUI<Store extends IExecutionDataSamplerStore,Sampler extends IExecutionDataSampler<Store>>
voidaddExecutionDataSampler(Sampler sampler)Plugins can add individual data samplers which will act upon the rows passing through the transforms in the pipeline.voidaddExecutionFinishedListener(IExecutionFinishedListener executionFinishedListener)Adds a pipeline finished listener.voidaddExecutionStartedListener(IExecutionStartedListener executionStartedListener)Adds a pipeline started listener.voidaddExecutionStoppedListener(IExecutionStoppedListener executionStoppedListener)Adds a pipeline stopped listener.voidaddFinishedListener(IExecutionFinishedListener<IPipelineEngine<PipelineMeta>> listener)voidaddParameterDefinition(String key, String defValue, String description)Adds a parameter definition to this pipeline.voidaddPipelineStoppedListener(IExecutionStoppedListener<IPipelineEngine<PipelineMeta>> pipelineStoppedListener)Adds a stop-event listener to the pipeline.RowProduceraddRowProducer(String transformName, int copynr)This adds a row producer to the pipeline that just got set up.voidaddStartedListener(IExecutionStartedListener<IPipelineEngine<PipelineMeta>> listener)protected voidaddTransformPerformanceSnapShot()Adds a transform performance snapshot.voidbeginProcessing()Begin processing.voidcalculateBatchIdAndDateRange()Calculate the batch id and date range for the pipeline.voidcleanup()This method performs any cleanup operations, typically called after the pipeline has finished.voidclearError()Clear the error in the pipeline, clear all the rows from all the row sets, to make sure the pipeline can continue with other data.voidclearParameterValues()Clear the values of all parameters.voidcopyFrom(IVariables variables)Copies variables from a given variable variables to this pipeline.voidcopyParametersFromDefinitions(INamedParameterDefinitions definitions)IPipelineEngineRunConfigurationcreateDefaultPipelineEngineRunConfiguration()Ask the engine to generate a default pipeline engine configuration for this enginevoiddoTopologySortOfTransforms()voidexecute()Executes the pipeline.IEngineComponentfindComponent(String name, int copyNr)Find a component by name and copy nrIRowSetfindRowSet(String from, int fromcopy, String to, int tocopy)Finds the IRowSet between two transforms (or copies of transforms).ITransformfindRunThread(String transformName)Find the run thread for the transform with the specified name.voidfirePipelineExecutionFinishedListeners()Make attempt to fire all registered finished listeners if possible.voidfirePipelineExecutionStartedListeners()Fires the start-event listeners (if any are registered).voidfirePipelineExecutionStoppedListeners()IPipelineEnginegetActiveSubPipeline(String subPipelineName)Get the active sub-pipeline with the given nameMap<String,IPipelineEngine>getActiveSubPipelines()Gets activeSubPipelinesIWorkflowEngine<WorkflowMeta>getActiveSubWorkflow(String subWorkflowName)Get the active sub-workflow with the given nameMap<String,IWorkflowEngine<WorkflowMeta>>getActiveSubWorkflows()Gets the active sub-workflows.List<IEngineComponent>getComponentCopies(String name)Find all component copies by nameStringgetComponentLogText(String componentName, int copyNr)Retrieve the logging text of a particular component in the engineList<IEngineComponent>getComponents()Get a list of componentsStringgetContainerId()Gets the container object ID.List<IExecutionDataSampler<? extends IExecutionDataSamplerStore>>getDataSamplers()Gets dataSamplersEngineMetricsgetEngineMetrics()Collect metrics from the various components in the engine.EngineMetricsgetEngineMetrics(String componentName, int copyNr)Get the engine metrics for a specific component name and/or copy nrintgetErrors()Gets the number of errors that have occurred during execution of the pipeline.StringgetExecutingServer()Gets the name of the executing server.StringgetExecutingUser()Gets the name of the executing user.DategetExecutionEndDate()Gets executionEndDateList<IExecutionFinishedListener<IPipelineEngine<PipelineMeta>>>getExecutionFinishedListeners()Gets executionFinishedListenersDategetExecutionStartDate()Gets executionStartDateList<IExecutionStartedListener<IPipelineEngine<PipelineMeta>>>getExecutionStartedListeners()Gets executionStartedListenersList<IExecutionStoppedListener<IPipelineEngine<PipelineMeta>>>getExecutionStoppedListeners()Gets the list of stop-event listeners for the pipeline.Map<String,Object>getExtensionDataMap()A map in which the various transforms and plugins can store objects to remember during the lifecycle of the pipeline.intgetFeedbackSize()Gets feedbackSizeStringgetFilename()Gets the filename of the pipeline, or null if no filename existsILogChannelgetLogChannel()Gets the log channel interface for the pipeline.StringgetLogChannelId()Gets the log channel ID.List<LoggingHierarchy>getLoggingHierarchy()Gets the logging hierarchy.LogLevelgetLogLevel()Gets the log level.IHopMetadataProvidergetMetadataProvider()StringgetName()Gets the name of the pipeline.StringgetObjectCopy()Gets the object copy.StringgetObjectName()Gets the object name.LoggingObjectTypegetObjectType()Gets the object type.StringgetParameterDefault(String key)Gets the default value of the specified parameter.StringgetParameterDescription(String key)Gets the description of the specified parameter.StringgetParameterValue(String key)Gets the value of the specified parameter.ILoggingObjectgetParent()Gets the parent logging object interface.IPipelineEnginegetParentPipeline()Gets the parent pipeline, which is null if no parent pipeline exists.IVariablesgetParentVariables()Gets the parent variable variables.IWorkflowEngine<WorkflowMeta>getParentWorkflow()Gets the parent workflow, or null if there is no parent.PipelineMetagetPipelineMeta()Gets the meta-data for the pipeline.PipelineRunConfigurationgetPipelineRunConfiguration()Gets pipelineRunConfigurationEngineComponent.ComponentExecutionStatus[]getPipelineTransformExecutionStatusLookup()Checks the execution status of each transform in the pipelines.boolean[]getPipelineTransformIsRunningLookup()Checks whether the pipeline transforms are running lookup.StringgetPluginId()Gets pluginIdResultgetPreviousResult()Get the execution result of a previous execution in a workflowDategetRegistrationDate()Gets the registration date.ResultgetResult()Gets the result of the pipeline.List<ResultFile>getResultFiles()List<RowMetaAndData>getResultRows()List<IRowSet>getRowsets()Gets the rowsets for the pipeline.intgetRowSetSize()Gets rowSetSizeITransformgetRunThread(int i)Gets the run thread for the transform at the specified index.ITransformgetRunThread(String name, int copy)Gets the run thread for the transform with the specified name and copy number.javax.servlet.http.HttpServletRequestgetServletRequest()javax.servlet.http.HttpServletResponsegetServletResponse()StringgetStatus()Gets the status of the pipeline (Halting, Finished, Paused, etc.)ITransformgetTransform(String name, int copyNr)Find the ITransform (thread) by looking it up using the name.ITransformDatagetTransformData(String name, int copyNr)Finds the ITransformData (currently) associated with the specified transform.Map<String,List<PerformanceSnapShot>>getTransformPerformanceSnapShots()Gets a named list (map) of transform performance snapshots.List<TransformMetaDataCombi>getTransforms()Gets a list of transforms in the pipeline.List<ITransform>getTransforms(String name)Find the available executing transform copies for the transform with the specified nameStringgetVariable(String variableName)Gets the value of the specified variable, or returns a default value if no such variable exists.StringgetVariable(String variableName, String defaultValue)Gets the value of the specified variable, or returns a default value if no such variable exists.booleangetVariableBoolean(String variableName, boolean defaultValue)Returns a boolean representation of the specified variable after performing any necessary substitution.String[]getVariableNames()Gets a list of variable names for the pipeline.IVariablesgetVariables()Gets variablesbooleanhasHaltedComponents()Checks whether the pipeline has any components that are halted.booleanhasTransformStarted(String sname, int copy)Checks whether the specified transform (or transform copy) has started.voidinitializeFrom(IVariables parent)Sets the values of the pipeline's variables to the values from the parent variables.booleanisFeedbackShown()Gets feedbackShownbooleanisFinished()Checks if the pipeline is finished\.booleanisFinishedOrStopped()booleanisForcingSeparateLogging()booleanisGatheringMetrics()booleanisInitializing()Checks whether the pipeline is initializing.booleanisPaused()Checks if the pipeline is paused.booleanisPreparing()Checks whether the pipeline is preparing for execution.booleanisPreview()Checks whether the pipeline is being previewed.booleanisReadyToStart()Checks whether the pipeline is ready to start (i.e.booleanisRunning()Checks whether the pipeline is running.booleanisSafeModeEnabled()Checks whether safe mode is enabled.booleanisSortingTransformsTopologically()Gets sortingTransformsTopologicallybooleanisStopped()Checks if the pipeline is stopped.String[]listParameters()Gets a list of the parameters for the pipeline.intnrTransforms()Gets the number of transforms in this pipeline.voidpauseExecution()Pauses the pipeline (pause all transforms).voidprepareExecution()Prepares the pipeline for execution.voidremoveAllParameters()Remove all parameters.Stringresolve(String aString)Substitutes any variable values into the given string, and returns the resolved string.String[]resolve(String[] aString)Substitutes any variable values into each of the given strings, and returns an array containing the resolved string(s).Stringresolve(String aString, IRowMeta rowMeta, Object[] rowData)voidresumeExecution()Resumes running the pipeline after a pause (resume all transforms).voidretrieveComponentOutput(IVariables variables, String componentName, int copyNr, int nrRows, IPipelineComponentRowsReceived rowsReceived)Retrieve output rows from a component copy.voidsafeStop()Stops only input transforms so that all downstream transforms can finish processing rows that have already been inputvoidsetActiveSubPipelines(Map<String,IPipelineEngine> activeSubPipelines)voidsetActiveSubWorkflows(Map<String,IWorkflowEngine<WorkflowMeta>> activeSubWorkflows)voidsetContainerId(String containerId)Sets the container object ID.voidsetDataSamplers(List<IExecutionDataSampler<? extends IExecutionDataSamplerStore>> dataSamplers)Sets dataSamplersvoidsetExecutingServer(String executingServer)Sets the name of the executing server.voidsetExecutingUser(String executingUser)Sets the name of the executing user.voidsetExecutionEndDate(Date executionEndDate)voidsetExecutionFinishedListeners(List<IExecutionFinishedListener<IPipelineEngine<PipelineMeta>>> executionFinishedListeners)voidsetExecutionStartDate(Date executionStartDate)voidsetExecutionStartedListeners(List<IExecutionStartedListener<IPipelineEngine<PipelineMeta>>> executionStartedListeners)voidsetExecutionStoppedListeners(List<IExecutionStoppedListener<IPipelineEngine<PipelineMeta>>> executionStoppedListeners)Sets the list of stop-event listeners for the pipeline.voidsetFeedbackShown(boolean feedbackShown)voidsetFeedbackSize(int feedbackSize)protected voidsetFinished(boolean finished)voidsetForcingSeparateLogging(boolean forcingSeparateLogging)voidsetGatheringMetrics(boolean gatheringMetrics)voidsetInitializing(boolean initializing)Sets whether the pipeline is initializing.protected voidsetInternalEntryCurrentDirectory(boolean hasFilename)voidsetInternalHopVariables(IVariables var)Sets the internal Hop variables.voidsetLogChannel(ILogChannel log)Sets the log channel interface for the pipeline.voidsetLogLevel(LogLevel logLevel)Sets the log level.voidsetMetadataProvider(IHopMetadataProvider metadataProvider)voidsetParameterValue(String key, String value)Sets the value for the specified parameter.voidsetParent(ILoggingObject parent)Sets the parent logging object.voidsetParentPipeline(IPipelineEngine parentPipeline)Sets the parent pipeline.voidsetParentVariables(IVariables parent)Sets the parent variable variables.voidsetParentWorkflow(IWorkflowEngine<WorkflowMeta> parentWorkflow)Sets the parent workflow for the pipeline.voidsetPaused(boolean paused)voidsetPipelineMeta(PipelineMeta pipelineMeta)Sets the meta-data for the pipeline.voidsetPipelineRunConfiguration(PipelineRunConfiguration pipelineRunConfiguration)voidsetPluginId(String pluginId)voidsetPreparing(boolean preparing)Sets whether the pipeline is preparing for execution.voidsetPreview(boolean preview)Sets whether the pipeline is being previewed.voidsetPreviousResult(Result previousResult)Inform the pipeline about a previous execution result in a workflow or pipelineprotected voidsetReadyToStart(boolean ready)voidsetResultFiles(List<ResultFile> resultFiles)voidsetResultRows(List<RowMetaAndData> resultRows)voidsetRowSetSize(int rowSetSize)voidsetRunning(boolean running)Sets whether the pipeline is running.voidsetSafeModeEnabled(boolean safeModeEnabled)Turn on safe mode during running: the pipeline will run slower but with more checking enabled.voidsetServletReponse(javax.servlet.http.HttpServletResponse response)Sets encoding of HttpServletResponse according to System encoding.Check if system encoding is null or an empty and set it to HttpServletResponse when not and writes error to log if null.voidsetServletRequest(javax.servlet.http.HttpServletRequest request)voidsetSortingTransformsTopologically(boolean sortingTransformsTopologically)voidsetStopped(boolean stopped)voidsetTransformPerformanceSnapShots(Map<String,List<PerformanceSnapShot>> transformPerformanceSnapShots)Sets the named list (map) of transform performance snapshots.protected voidsetTransforms(List<TransformMetaDataCombi> transforms)voidsetVariable(String variableName, String variableValue)Sets the value of the specified variable to the specified value.voidsetVariables(Map<String,String> map)Injects variables using the given Map.voidsetVariables(IVariables variables)voidshareWith(IVariables variables)Shares a variable variables from another variable variables.voidstartThreads()Starts the threads prepared by prepareThreads().voidstopAll()Stops all transforms from running, and alerts any registered listeners.voidstopTransform(TransformMetaDataCombi combi, boolean safeStop)StringtoString()Gets a string representation of the pipeline.voidwaitUntilFinished()Waits until all RunThreads have finished.-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.hop.pipeline.engine.IPipelineEngine
getEngineCapabilities, getStatusDescription, pipelineCompleted
-
-
-
-
Field Detail
-
METRIC_NAME_INPUT
public static final String METRIC_NAME_INPUT
- See Also:
- Constant Field Values
-
METRIC_NAME_OUTPUT
public static final String METRIC_NAME_OUTPUT
- See Also:
- Constant Field Values
-
METRIC_NAME_ERROR
public static final String METRIC_NAME_ERROR
- See Also:
- Constant Field Values
-
METRIC_NAME_READ
public static final String METRIC_NAME_READ
- See Also:
- Constant Field Values
-
METRIC_NAME_WRITTEN
public static final String METRIC_NAME_WRITTEN
- See Also:
- Constant Field Values
-
METRIC_NAME_UPDATED
public static final String METRIC_NAME_UPDATED
- See Also:
- Constant Field Values
-
METRIC_NAME_REJECTED
public static final String METRIC_NAME_REJECTED
- See Also:
- Constant Field Values
-
METRIC_NAME_BUFFER_IN
public static final String METRIC_NAME_BUFFER_IN
- See Also:
- Constant Field Values
-
METRIC_NAME_BUFFER_OUT
public static final String METRIC_NAME_BUFFER_OUT
- See Also:
- Constant Field Values
-
METRIC_NAME_FLUSH_BUFFER
public static final String METRIC_NAME_FLUSH_BUFFER
- See Also:
- Constant Field Values
-
METRIC_NAME_INIT
public static final String METRIC_NAME_INIT
- See Also:
- Constant Field Values
-
pluginId
protected String pluginId
-
pipelineRunConfiguration
protected PipelineRunConfiguration pipelineRunConfiguration
-
log
protected ILogChannel log
The log channel interface.
-
logLevel
protected LogLevel logLevel
The log level.
-
containerObjectId
protected String containerObjectId
The container object id.
-
logCommitSize
protected int logCommitSize
The log commit size.
-
pipelineMeta
protected PipelineMeta pipelineMeta
The pipeline metadata to execute.
-
metadataProvider
protected IHopMetadataProvider metadataProvider
The MetaStore to use
-
TYPE_DISP_1_1
public static final int TYPE_DISP_1_1
Constant indicating a dispatch type of 1-to-1.- See Also:
- Constant Field Values
-
TYPE_DISP_1_N
public static final int TYPE_DISP_1_N
Constant indicating a dispatch type of 1-to-N.- See Also:
- Constant Field Values
-
TYPE_DISP_N_1
public static final int TYPE_DISP_N_1
Constant indicating a dispatch type of N-to-1.- See Also:
- Constant Field Values
-
TYPE_DISP_N_N
public static final int TYPE_DISP_N_N
Constant indicating a dispatch type of N-to-N.- See Also:
- Constant Field Values
-
TYPE_DISP_N_M
public static final int TYPE_DISP_N_M
Constant indicating a dispatch type of N-to-M.- See Also:
- Constant Field Values
-
STRING_FINISHED
public static final String STRING_FINISHED
Constant indicating a pipeline status of Finished.- See Also:
- Constant Field Values
-
STRING_FINISHED_WITH_ERRORS
public static final String STRING_FINISHED_WITH_ERRORS
Constant indicating a pipeline status of Finished (with errors).- See Also:
- Constant Field Values
-
STRING_RUNNING
public static final String STRING_RUNNING
Constant indicating a pipeline status of Running.- See Also:
- Constant Field Values
-
STRING_PAUSED
public static final String STRING_PAUSED
Constant indicating a pipeline status of Paused.- See Also:
- Constant Field Values
-
STRING_PREPARING
public static final String STRING_PREPARING
Constant indicating a pipeline status of Preparing for execution.- See Also:
- Constant Field Values
-
STRING_INITIALIZING
public static final String STRING_INITIALIZING
Constant indicating a pipeline status of Initializing.- See Also:
- Constant Field Values
-
STRING_WAITING
public static final String STRING_WAITING
Constant indicating a pipeline status of Waiting.- See Also:
- Constant Field Values
-
STRING_STOPPED
public static final String STRING_STOPPED
Constant indicating a pipeline status of Stopped.- See Also:
- Constant Field Values
-
STRING_STOPPED_WITH_ERRORS
public static final String STRING_STOPPED_WITH_ERRORS
Constant indicating a pipeline status of Stopped (with errors).- See Also:
- Constant Field Values
-
STRING_HALTING
public static final String STRING_HALTING
Constant indicating a pipeline status of Halting.- See Also:
- Constant Field Values
-
CONFIGURATION_IN_EXPORT_FILENAME
public static final String CONFIGURATION_IN_EXPORT_FILENAME
Constant specifying a filename containing XML to inject into a ZIP file created during resource export.- See Also:
- Constant Field Values
-
dataSamplers
protected List<IExecutionDataSampler<? extends IExecutionDataSamplerStore>> dataSamplers
Plugins can use this to add additional data samplers to the pipeline.
-
resultRows
protected List<RowMetaAndData> resultRows
-
resultFiles
protected List<ResultFile> resultFiles
-
arguments
protected String[] arguments
The command line arguments for the pipeline.
-
rowSetSize
protected int rowSetSize
-
feedbackShown
protected boolean feedbackShown
Whether the feedback is shown.
-
feedbackSize
protected int feedbackSize
The feedback size.
-
METRIC_INPUT
public static final IEngineMetric METRIC_INPUT
-
METRIC_READ
public static final IEngineMetric METRIC_READ
-
METRIC_WRITTEN
public static final IEngineMetric METRIC_WRITTEN
-
METRIC_OUTPUT
public static final IEngineMetric METRIC_OUTPUT
-
METRIC_UPDATED
public static final IEngineMetric METRIC_UPDATED
-
METRIC_REJECTED
public static final IEngineMetric METRIC_REJECTED
-
METRIC_ERROR
public static final IEngineMetric METRIC_ERROR
-
METRIC_BUFFER_IN
public static final IEngineMetric METRIC_BUFFER_IN
-
METRIC_BUFFER_OUT
public static final IEngineMetric METRIC_BUFFER_OUT
-
METRIC_INIT
public static final IEngineMetric METRIC_INIT
-
METRIC_FLUSH_BUFFER
public static final IEngineMetric METRIC_FLUSH_BUFFER
-
-
Constructor Detail
-
Pipeline
public Pipeline()
Instantiates a new pipeline.
-
Pipeline
public Pipeline(PipelineMeta pipelineMeta)
Initializes a pipeline from pipeline meta-data defined in memory.- Parameters:
pipelineMeta- the pipeline meta-data to use.
-
Pipeline
public Pipeline(PipelineMeta pipelineMeta, IVariables variables, ILoggingObject parent)
Initializes a pipeline from pipeline meta-data defined in memory. Also take into account the parent log channel interface (workflow or pipeline) for logging lineage purposes.- Parameters:
pipelineMeta- the pipeline meta-data to use.variables- The variables to inherit fromparent- the parent workflow that is executing this pipeline
-
Pipeline
public Pipeline(Parent parent, String name, String filename, IHopMetadataProvider metadataProvider) throws HopExceptionInstantiates a new pipeline using any of the provided parameters including the variable bindings, a name and a filename. This contstructor loads the specified pipeline from a file.- Parameters:
parent- the parent variable variables and named paramsname- the name of the pipelinefilename- the filename containing the pipeline definitionmetadataProvider- The MetaStore to use when referencing metadata objects- Throws:
HopException- if any error occurs during loading, parsing, or creation of the pipeline
-
-
Method Detail
-
setParent
public void setParent(ILoggingObject parent)
Sets the parent logging object.- Specified by:
setParentin interfaceIPipelineEngine<PipelineMeta>- Parameters:
parent- the new parent
-
getLogChannel
public ILogChannel getLogChannel()
Gets the log channel interface for the pipeline.- Specified by:
getLogChannelin interfaceIHasLogChannel- Specified by:
getLogChannelin interfaceIPipelineEngine<PipelineMeta>- Returns:
- the log channel
- See Also:
IHasLogChannel.getLogChannel()
-
setLogChannel
public void setLogChannel(ILogChannel log)
Sets the log channel interface for the pipeline.- Specified by:
setLogChannelin interfaceIPipelineEngine<PipelineMeta>- Parameters:
log- the new log channel interface
-
getName
public String getName()
Gets the name of the pipeline.- Returns:
- the pipeline name
-
execute
public void execute() throws HopExceptionExecutes the pipeline. This method will prepare the pipeline for execution and then start all the threads associated with the pipeline and its transforms.- Specified by:
executein interfaceIPipelineEngine<PipelineMeta>- Throws:
HopException- if the pipeline could not be prepared (initialized)
-
prepareExecution
public void prepareExecution() throws HopExceptionPrepares the pipeline for execution. This includes setting the arguments and parameters as well as preparing and tracking the transforms and hops in the pipeline.- Specified by:
prepareExecutionin interfaceIPipelineEngine<PipelineMeta>- Throws:
HopException- in case the pipeline could not be prepared (initialized)
-
startThreads
public void startThreads() throws HopExceptionStarts the threads prepared by prepareThreads(). Before you start the threads, you can add RowListeners to them.- Specified by:
startThreadsin interfaceIPipelineEngine<PipelineMeta>- Throws:
HopException- if there is a communication error with a remote output socket.
-
firePipelineExecutionFinishedListeners
public void firePipelineExecutionFinishedListeners() throws HopExceptionMake attempt to fire all registered finished listeners if possible.- Specified by:
firePipelineExecutionFinishedListenersin interfaceIPipelineEngine<PipelineMeta>- Throws:
HopException- if any errors occur during notification
-
firePipelineExecutionStartedListeners
public void firePipelineExecutionStartedListeners() throws HopExceptionFires the start-event listeners (if any are registered).- Specified by:
firePipelineExecutionStartedListenersin interfaceIPipelineEngine<PipelineMeta>- Throws:
HopException- if any errors occur during notification
-
addTransformPerformanceSnapShot
protected void addTransformPerformanceSnapShot()
Adds a transform performance snapshot.
-
cleanup
public void cleanup()
This method performs any cleanup operations, typically called after the pipeline has finished.- Specified by:
cleanupin interfaceIPipelineEngine<PipelineMeta>
-
waitUntilFinished
public void waitUntilFinished()
Waits until all RunThreads have finished.- Specified by:
waitUntilFinishedin interfaceIPipelineEngine<PipelineMeta>
-
getErrors
public int getErrors()
Gets the number of errors that have occurred during execution of the pipeline.- Specified by:
getErrorsin interfaceIPipelineEngine<PipelineMeta>- Returns:
- the number of errors
-
isFinished
public boolean isFinished()
Checks if the pipeline is finished\.- Specified by:
isFinishedin interfaceIPipelineEngine<PipelineMeta>- Returns:
- true if the pipeline is finished, false otherwise
-
setFinished
protected void setFinished(boolean finished)
-
isFinishedOrStopped
public boolean isFinishedOrStopped()
-
findRowSet
public IRowSet findRowSet(String from, int fromcopy, String to, int tocopy)
Finds the IRowSet between two transforms (or copies of transforms).- Specified by:
findRowSetin interfaceIPipelineEngine<PipelineMeta>- Parameters:
from- the name of the "from" transformfromcopy- the copy number of the "from" transformto- the name of the "to" transformtocopy- the copy number of the "to" transform- Returns:
- the row set, or null if none found
-
hasTransformStarted
public boolean hasTransformStarted(String sname, int copy)
Checks whether the specified transform (or transform copy) has started.- Parameters:
sname- the transform namecopy- the copy number- Returns:
- true the specified transform (or transform copy) has started, false otherwise
-
safeStop
public void safeStop()
Stops only input transforms so that all downstream transforms can finish processing rows that have already been input
-
stopAll
public void stopAll()
Stops all transforms from running, and alerts any registered listeners.- Specified by:
stopAllin interfaceIPipelineEngine<PipelineMeta>
-
stopTransform
public void stopTransform(TransformMetaDataCombi combi, boolean safeStop)
-
firePipelineExecutionStoppedListeners
public void firePipelineExecutionStoppedListeners()
- Specified by:
firePipelineExecutionStoppedListenersin interfaceIPipelineEngine<PipelineMeta>
-
nrTransforms
public int nrTransforms()
Gets the number of transforms in this pipeline.- Returns:
- the number of transforms
-
getPipelineTransformIsRunningLookup
public boolean[] getPipelineTransformIsRunningLookup()
Checks whether the pipeline transforms are running lookup.- Returns:
- a boolean array associated with the transform list, indicating whether that transform is running a lookup.
-
getPipelineTransformExecutionStatusLookup
public EngineComponent.ComponentExecutionStatus[] getPipelineTransformExecutionStatusLookup()
Checks the execution status of each transform in the pipelines.- Returns:
- an array associated with the transform list, indicating the status of that transform.
-
getRunThread
public ITransform getRunThread(int i)
Gets the run thread for the transform at the specified index.- Parameters:
i- the index of the desired transform- Returns:
- a ITransform object corresponding to the run thread for the specified transform
-
getRunThread
public ITransform getRunThread(String name, int copy)
Gets the run thread for the transform with the specified name and copy number.- Parameters:
name- the transform namecopy- the copy number- Returns:
- a ITransform object corresponding to the run thread for the specified transform
-
calculateBatchIdAndDateRange
public void calculateBatchIdAndDateRange() throws HopPipelineExceptionCalculate the batch id and date range for the pipeline.- Throws:
HopPipelineException- if there are any errors during calculation
-
beginProcessing
public void beginProcessing() throws HopPipelineExceptionBegin processing. Also handle logging operations related to the start of the pipeline- Throws:
HopPipelineException- the hop pipeline exception
-
getResult
public Result getResult()
Gets the result of the pipeline. The Result object contains such measures as the number of errors, number of lines read/written/input/output/updated/rejected, etc.- Specified by:
getResultin interfaceIPipelineEngine<PipelineMeta>- Returns:
- the Result object containing resulting measures from execution of the pipeline
-
findRunThread
public ITransform findRunThread(String transformName)
Find the run thread for the transform with the specified name.- Parameters:
transformName- the transform name- Returns:
- a ITransform object corresponding to the run thread for the specified transform
-
isSortingTransformsTopologically
public boolean isSortingTransformsTopologically()
Gets sortingTransformsTopologically- Returns:
- value of sortingTransformsTopologically
-
setSortingTransformsTopologically
public void setSortingTransformsTopologically(boolean sortingTransformsTopologically)
- Parameters:
sortingTransformsTopologically- The sortingTransformsTopologically to set
-
getPipelineMeta
public PipelineMeta getPipelineMeta()
Gets the meta-data for the pipeline.- Specified by:
getPipelineMetain interfaceIPipelineEngine<PipelineMeta>- Returns:
- Returns the pipeline meta-data
-
setPipelineMeta
public void setPipelineMeta(PipelineMeta pipelineMeta)
Sets the meta-data for the pipeline.- Specified by:
setPipelineMetain interfaceIPipelineEngine<PipelineMeta>- Parameters:
pipelineMeta- The pipeline meta-data to set.
-
getRowsets
public List<IRowSet> getRowsets()
Gets the rowsets for the pipeline.- Returns:
- a list of rowsets
-
getTransforms
public List<TransformMetaDataCombi> getTransforms()
Gets a list of transforms in the pipeline.- Returns:
- a list of the transforms in the pipeline
-
setTransforms
protected void setTransforms(List<TransformMetaDataCombi> transforms)
-
toString
public String toString()
Gets a string representation of the pipeline.- Overrides:
toStringin classObject- Returns:
- the string representation of the pipeline
- See Also:
Object.toString()
-
getTransform
public ITransform getTransform(String name, int copyNr)
Find the ITransform (thread) by looking it up using the name.- Parameters:
name- The name of the transform to look forcopyNr- The copy number of the transform to look for- Returns:
- the ITransform or null if nothing was found.
-
getTransforms
public List<ITransform> getTransforms(String name)
Find the available executing transform copies for the transform with the specified name- Parameters:
name- The transform name- Returns:
- the list of executing transform copies found or null if no transforms are available yet (incorrect usage)
-
setSafeModeEnabled
public void setSafeModeEnabled(boolean safeModeEnabled)
Turn on safe mode during running: the pipeline will run slower but with more checking enabled.- Parameters:
safeModeEnabled- true for safe mode
-
isSafeModeEnabled
public boolean isSafeModeEnabled()
Checks whether safe mode is enabled.- Specified by:
isSafeModeEnabledin interfaceIPipelineEngine<PipelineMeta>- Returns:
- Returns true if the safe mode is enabled: the pipeline will run slower but with more checking enabled
-
addRowProducer
public RowProducer addRowProducer(String transformName, int copynr) throws HopException
This adds a row producer to the pipeline that just got set up. It is preferable to run this BEFORE execute() but after prepareExecution()- Parameters:
transformName- The transform to produce rows forcopynr- The copynr of the transform to produce row for (normally 0 unless you have multiple copies running)- Returns:
- the row producer
- Throws:
HopException- in case the thread/transform to produce rows for could not be found.- See Also:
execute(),prepareExecution()
-
getParentWorkflow
public IWorkflowEngine<WorkflowMeta> getParentWorkflow()
Gets the parent workflow, or null if there is no parent.- Specified by:
getParentWorkflowin interfaceIPipelineEngine<PipelineMeta>- Returns:
- the parent workflow, or null if there is no parent
-
setParentWorkflow
public void setParentWorkflow(IWorkflowEngine<WorkflowMeta> parentWorkflow)
Sets the parent workflow for the pipeline.- Specified by:
setParentWorkflowin interfaceIPipelineEngine<PipelineMeta>- Parameters:
parentWorkflow- The parent workflow to set
-
getTransformData
public ITransformData getTransformData(String name, int copyNr)
Finds the ITransformData (currently) associated with the specified transform.- Parameters:
name- The name of the transform to look forcopyNr- The copy number (0 based) of the transform- Returns:
- The ITransformData or null if non found.
-
hasHaltedComponents
public boolean hasHaltedComponents()
Checks whether the pipeline has any components that are halted.- Specified by:
hasHaltedComponentsin interfaceIPipelineEngine<PipelineMeta>- Returns:
- true if one or more components are halted, false otherwise
-
getStatus
public String getStatus()
Gets the status of the pipeline (Halting, Finished, Paused, etc.)- Returns:
- the status of the pipeline
-
isInitializing
public boolean isInitializing()
Checks whether the pipeline is initializing.- Returns:
- true if the pipeline is initializing, false otherwise
-
setInitializing
public void setInitializing(boolean initializing)
Sets whether the pipeline is initializing.- Parameters:
initializing- true if the pipeline is initializing, false otherwise
-
isPreparing
public boolean isPreparing()
Checks whether the pipeline is preparing for execution.- Specified by:
isPreparingin interfaceIPipelineEngine<PipelineMeta>- Returns:
- true if the pipeline is preparing for execution, false otherwise
-
setPreparing
public void setPreparing(boolean preparing)
Sets whether the pipeline is preparing for execution.- Parameters:
preparing- true if the pipeline is preparing for execution, false otherwise
-
isRunning
public boolean isRunning()
Checks whether the pipeline is running.- Specified by:
isRunningin interfaceIPipelineEngine<PipelineMeta>- Returns:
- true if the pipeline is running, false otherwise
-
setRunning
public void setRunning(boolean running)
Sets whether the pipeline is running.- Parameters:
running- true if the pipeline is running, false otherwise
-
isReadyToStart
public boolean isReadyToStart()
Checks whether the pipeline is ready to start (i.e. execution preparation was successful)- Specified by:
isReadyToStartin interfaceIPipelineEngine<PipelineMeta>- Returns:
- true if the pipeline was prepared for execution successfully, false otherwise
- See Also:
prepareExecution()
-
setReadyToStart
protected void setReadyToStart(boolean ready)
-
setInternalHopVariables
public void setInternalHopVariables(IVariables var)
Sets the internal Hop variables.- Specified by:
setInternalHopVariablesin interfaceIPipelineEngine<PipelineMeta>- Parameters:
var- the new internal hop variables
-
setInternalEntryCurrentDirectory
protected void setInternalEntryCurrentDirectory(boolean hasFilename)
-
copyFrom
public void copyFrom(IVariables variables)
Copies variables from a given variable variables to this pipeline.- Specified by:
copyFromin interfaceIVariables- Parameters:
variables- the variable variables- See Also:
IVariables.copyFrom(IVariables)
-
resolve
public String resolve(String aString)
Substitutes any variable values into the given string, and returns the resolved string.- Specified by:
resolvein interfaceIVariables- Parameters:
aString- the string to resolve against environment variables- Returns:
- the string after variables have been resolved/susbstituted
- See Also:
IVariables.resolve(String)
-
resolve
public String[] resolve(String[] aString)
Substitutes any variable values into each of the given strings, and returns an array containing the resolved string(s).- Specified by:
resolvein interfaceIVariables- Parameters:
aString- an array of strings to resolve against environment variables- Returns:
- the array of strings after variables have been resolved/susbstituted
- See Also:
IVariables.resolve(String[])
-
resolve
public String resolve(String aString, IRowMeta rowMeta, Object[] rowData) throws HopValueException
- Specified by:
resolvein interfaceIVariables- Throws:
HopValueException
-
getParentVariables
public IVariables getParentVariables()
Gets the parent variable variables.- Specified by:
getParentVariablesin interfaceIVariables- Returns:
- the parent variable variables
- See Also:
IVariables.getParentVariables()
-
setParentVariables
public void setParentVariables(IVariables parent)
Sets the parent variable variables.- Specified by:
setParentVariablesin interfaceIVariables- Parameters:
parent- the new parent variable variables- See Also:
IVariables.setParentVariables( IVariables)
-
getVariable
public String getVariable(String variableName, String defaultValue)
Gets the value of the specified variable, or returns a default value if no such variable exists.- Specified by:
getVariablein interfaceIVariables- Parameters:
variableName- the variable namedefaultValue- the default value- Returns:
- the value of the specified variable, or returns a default value if no such variable exists
- See Also:
IVariables.getVariable(String, String)
-
getVariable
public String getVariable(String variableName)
Gets the value of the specified variable, or returns a default value if no such variable exists.- Specified by:
getVariablein interfaceIVariables- Parameters:
variableName- the variable name- Returns:
- the value of the specified variable, or returns a default value if no such variable exists
- See Also:
IVariables.getVariable(String)
-
getVariableBoolean
public boolean getVariableBoolean(String variableName, boolean defaultValue)
Returns a boolean representation of the specified variable after performing any necessary substitution. Truth values include case-insensitive versions of "Y", "YES", "TRUE" or "1".- Specified by:
getVariableBooleanin interfaceIVariables- Parameters:
variableName- the variable namedefaultValue- the default value- Returns:
- a boolean representation of the specified variable after performing any necessary substitution
- See Also:
IVariables.getVariableBoolean(String, boolean)
-
initializeFrom
public void initializeFrom(IVariables parent)
Sets the values of the pipeline's variables to the values from the parent variables.- Specified by:
initializeFromin interfaceIVariables- Parameters:
parent- the parent- See Also:
IVariables.initializeFrom( IVariables)
-
getVariableNames
public String[] getVariableNames()
Gets a list of variable names for the pipeline.- Specified by:
getVariableNamesin interfaceIVariables- Returns:
- a list of variable names
- See Also:
IVariables.getVariableNames()
-
setVariable
public void setVariable(String variableName, String variableValue)
Sets the value of the specified variable to the specified value.- Specified by:
setVariablein interfaceIVariables- Parameters:
variableName- the variable namevariableValue- the variable value- See Also:
IVariables.setVariable(String, String)
-
shareWith
public void shareWith(IVariables variables)
Shares a variable variables from another variable variables. This means that the object should take over the variables used as argument.- Specified by:
shareWithin interfaceIVariables- Parameters:
variables- the variable variables- See Also:
IVariables.shareWith(IVariables)
-
setVariables
public void setVariables(Map<String,String> map)
Injects variables using the given Map. The behavior should be that the properties object will be stored and at the time the IVariables is initialized (or upon calling this method if the variables is already initialized). After injecting the link of the properties object should be removed.- Specified by:
setVariablesin interfaceIVariables- Parameters:
map- the property map- See Also:
IVariables.setVariables(Map)
-
pauseExecution
public void pauseExecution()
Pauses the pipeline (pause all transforms).- Specified by:
pauseExecutionin interfaceIPipelineEngine<PipelineMeta>
-
resumeExecution
public void resumeExecution()
Resumes running the pipeline after a pause (resume all transforms).- Specified by:
resumeExecutionin interfaceIPipelineEngine<PipelineMeta>
-
isPreview
public boolean isPreview()
Checks whether the pipeline is being previewed.- Specified by:
isPreviewin interfaceIPipelineEngine<PipelineMeta>- Returns:
- true if the pipeline is being previewed, false otherwise
-
setPreview
public void setPreview(boolean preview)
Sets whether the pipeline is being previewed.- Specified by:
setPreviewin interfaceIPipelineEngine<PipelineMeta>- Parameters:
preview- true if the pipeline is being previewed, false otherwise
-
getTransformPerformanceSnapShots
public Map<String,List<PerformanceSnapShot>> getTransformPerformanceSnapShots()
Gets a named list (map) of transform performance snapshots.- Returns:
- a named list (map) of transform performance snapshots
-
setTransformPerformanceSnapShots
public void setTransformPerformanceSnapShots(Map<String,List<PerformanceSnapShot>> transformPerformanceSnapShots)
Sets the named list (map) of transform performance snapshots.- Parameters:
transformPerformanceSnapShots- a named list (map) of transform performance snapshots to set
-
addExecutionStartedListener
public void addExecutionStartedListener(IExecutionStartedListener executionStartedListener)
Adds a pipeline started listener.- Specified by:
addExecutionStartedListenerin interfaceIPipelineEngine<PipelineMeta>- Parameters:
executionStartedListener- the pipeline started listener
-
addExecutionFinishedListener
public void addExecutionFinishedListener(IExecutionFinishedListener executionFinishedListener)
Adds a pipeline finished listener.- Specified by:
addExecutionFinishedListenerin interfaceIPipelineEngine<PipelineMeta>- Parameters:
executionFinishedListener- the pipeline finished listener
-
addExecutionStoppedListener
public void addExecutionStoppedListener(IExecutionStoppedListener executionStoppedListener)
Adds a pipeline stopped listener.- Specified by:
addExecutionStoppedListenerin interfaceIPipelineEngine<PipelineMeta>- Parameters:
executionStoppedListener- the pipeline stopped listener
-
setExecutionStoppedListeners
public void setExecutionStoppedListeners(List<IExecutionStoppedListener<IPipelineEngine<PipelineMeta>>> executionStoppedListeners)
Sets the list of stop-event listeners for the pipeline.- Parameters:
executionStoppedListeners- the list of stop-event listeners to set
-
getExecutionStoppedListeners
public List<IExecutionStoppedListener<IPipelineEngine<PipelineMeta>>> getExecutionStoppedListeners()
Gets the list of stop-event listeners for the pipeline. This is not concurrent safe. Please note this is mutable implementation only for backward compatibility reasons.- Returns:
- the list of stop-event listeners
-
addPipelineStoppedListener
public void addPipelineStoppedListener(IExecutionStoppedListener<IPipelineEngine<PipelineMeta>> pipelineStoppedListener)
Adds a stop-event listener to the pipeline.- Parameters:
pipelineStoppedListener- the stop-event listener to add
-
isPaused
public boolean isPaused()
Checks if the pipeline is paused.- Specified by:
isPausedin interfaceIPipelineEngine<PipelineMeta>- Returns:
- true if the pipeline is paused, false otherwise
-
setPaused
public void setPaused(boolean paused)
-
isStopped
public boolean isStopped()
Checks if the pipeline is stopped.- Specified by:
isStoppedin interfaceIPipelineEngine<PipelineMeta>- Returns:
- true if the pipeline is stopped, false otherwise
-
setStopped
public void setStopped(boolean stopped)
-
addParameterDefinition
public void addParameterDefinition(String key, String defValue, String description) throws DuplicateParamException
Adds a parameter definition to this pipeline.- Specified by:
addParameterDefinitionin interfaceINamedParameterDefinitions- Parameters:
key- the name of the parameterdefValue- the default value for the parameterdescription- the description of the parameter- Throws:
DuplicateParamException- the duplicate param exception- See Also:
INamedParameterDefinitions.addParameterDefinition(String, String, String)
-
getParameterDefault
public String getParameterDefault(String key) throws UnknownParamException
Gets the default value of the specified parameter.- Specified by:
getParameterDefaultin interfaceINamedParameterDefinitions- Parameters:
key- the name of the parameter- Returns:
- the default value of the parameter
- Throws:
UnknownParamException- if the parameter does not exist- See Also:
INamedParameterDefinitions.getParameterDefault(String)
-
getParameterDescription
public String getParameterDescription(String key) throws UnknownParamException
Gets the description of the specified parameter.- Specified by:
getParameterDescriptionin interfaceINamedParameterDefinitions- Parameters:
key- the name of the parameter- Returns:
- the parameter description
- Throws:
UnknownParamException- if the parameter does not exist- See Also:
INamedParameterDefinitions.getParameterDescription(String)
-
getParameterValue
public String getParameterValue(String key) throws UnknownParamException
Gets the value of the specified parameter.- Specified by:
getParameterValuein interfaceINamedParameters- Parameters:
key- the name of the parameter- Returns:
- the parameter value
- Throws:
UnknownParamException- if the parameter does not exist- See Also:
INamedParameters.getParameterValue(String)
-
listParameters
public String[] listParameters()
Gets a list of the parameters for the pipeline.- Specified by:
listParametersin interfaceINamedParameterDefinitions- Returns:
- an array of strings containing the names of all parameters for the pipeline
- See Also:
INamedParameterDefinitions.listParameters()
-
setParameterValue
public void setParameterValue(String key, String value) throws UnknownParamException
Sets the value for the specified parameter.- Specified by:
setParameterValuein interfaceINamedParameters- Parameters:
key- the name of the parametervalue- the name of the value- Throws:
UnknownParamException- if the parameter does not exist- See Also:
INamedParameters.setParameterValue(String, String)
-
removeAllParameters
public void removeAllParameters()
Remove all parameters.- Specified by:
removeAllParametersin interfaceINamedParameterDefinitions- Specified by:
removeAllParametersin interfaceINamedParameters- See Also:
INamedParameters.removeAllParameters()
-
clearParameterValues
public void clearParameterValues()
Clear the values of all parameters.- Specified by:
clearParameterValuesin interfaceINamedParameters- See Also:
INamedParameters.clearParameterValues()
-
activateParameters
public void activateParameters(IVariables variables)
Activates all parameters by setting their values. If no values already exist, the method will attempt to set the parameter to the default value. If no default value exists, the method will set the value of the parameter to the empty string ("").- Specified by:
activateParametersin interfaceINamedParameters
-
copyParametersFromDefinitions
public void copyParametersFromDefinitions(INamedParameterDefinitions definitions)
- Specified by:
copyParametersFromDefinitionsin interfaceINamedParameters
-
getParentPipeline
public IPipelineEngine getParentPipeline()
Gets the parent pipeline, which is null if no parent pipeline exists.- Specified by:
getParentPipelinein interfaceIPipelineEngine<PipelineMeta>- Returns:
- a reference to the parent pipeline's Pipeline object, or null if no parent pipeline exists
-
setParentPipeline
public void setParentPipeline(IPipelineEngine parentPipeline)
Sets the parent pipeline.- Specified by:
setParentPipelinein interfaceIPipelineEngine<PipelineMeta>- Parameters:
parentPipeline- the parentPipeline to set
-
getObjectName
public String getObjectName()
Gets the object name.- Specified by:
getObjectNamein interfaceILoggingObject- Returns:
- the object name
- See Also:
ILoggingObject.getObjectName()
-
getObjectCopy
public String getObjectCopy()
Gets the object copy. For Pipeline, this always returns null- Specified by:
getObjectCopyin interfaceILoggingObject- Returns:
- null
- See Also:
ILoggingObject.getObjectCopy()
-
getFilename
public String getFilename()
Gets the filename of the pipeline, or null if no filename exists- Specified by:
getFilenamein interfaceILoggingObject- Returns:
- the filename
- See Also:
ILoggingObject.getFilename()
-
getLogChannelId
public String getLogChannelId()
Gets the log channel ID.- Specified by:
getLogChannelIdin interfaceILoggingObject- Specified by:
getLogChannelIdin interfaceIPipelineEngine<PipelineMeta>- Returns:
- the log channel ID
- See Also:
ILoggingObject.getLogChannelId()
-
getObjectType
public LoggingObjectType getObjectType()
Gets the object type. For Pipeline, this always returns LoggingObjectType.PIPELINE- Specified by:
getObjectTypein interfaceILoggingObject- Returns:
- the object type
- See Also:
ILoggingObject.getObjectType()
-
getParent
public ILoggingObject getParent()
Gets the parent logging object interface.- Specified by:
getParentin interfaceILoggingObject- Returns:
- the parent
- See Also:
ILoggingObject.getParent()
-
getLogLevel
public LogLevel getLogLevel()
Gets the log level.- Specified by:
getLogLevelin interfaceILoggingObject- Specified by:
getLogLevelin interfaceIPipelineEngine<PipelineMeta>- Returns:
- the log level
- See Also:
ILoggingObject.getLogLevel()
-
setLogLevel
public void setLogLevel(LogLevel logLevel)
Sets the log level.- Specified by:
setLogLevelin interfaceIPipelineEngine<PipelineMeta>- Parameters:
logLevel- the new log level
-
getLoggingHierarchy
public List<LoggingHierarchy> getLoggingHierarchy()
Gets the logging hierarchy.- Returns:
- the logging hierarchy
-
addActiveSubPipeline
public void addActiveSubPipeline(String subPipelineName, IPipelineEngine subPipeline)
Description copied from interface:IPipelineEngineAdd an active sub-pipeline to allow drill-down in the GUI- Specified by:
addActiveSubPipelinein interfaceIPipelineEngine<PipelineMeta>
-
getActiveSubPipeline
public IPipelineEngine getActiveSubPipeline(String subPipelineName)
Description copied from interface:IPipelineEngineGet the active sub-pipeline with the given name- Specified by:
getActiveSubPipelinein interfaceIPipelineEngine<PipelineMeta>- Returns:
- The active pipeline engine or null if it was not found
-
addActiveSubWorkflow
public void addActiveSubWorkflow(String subWorkflowName, IWorkflowEngine<WorkflowMeta> subWorkflow)
Description copied from interface:IPipelineEngineAdd an active sub-workflow to allow drill-down in the GUI- Specified by:
addActiveSubWorkflowin interfaceIPipelineEngine<PipelineMeta>
-
getActiveSubWorkflow
public IWorkflowEngine<WorkflowMeta> getActiveSubWorkflow(String subWorkflowName)
Description copied from interface:IPipelineEngineGet the active sub-workflow with the given name- Specified by:
getActiveSubWorkflowin interfaceIPipelineEngine<PipelineMeta>- Returns:
- The active workflow or null if nothing was found
-
getActiveSubWorkflows
public Map<String,IWorkflowEngine<WorkflowMeta>> getActiveSubWorkflows()
Gets the active sub-workflows.- Returns:
- a map (by name) of the active sub-workflows
-
getContainerId
public String getContainerId()
Gets the container object ID.- Specified by:
getContainerIdin interfaceILoggingObject- Returns:
- the HopServer object ID
-
setContainerId
public void setContainerId(String containerId)
Sets the container object ID.- Specified by:
setContainerIdin interfaceIPipelineEngine<PipelineMeta>- Parameters:
containerId- the HopServer object ID to set
-
getRegistrationDate
public Date getRegistrationDate()
Gets the registration date. For Pipeline, this always returns null- Specified by:
getRegistrationDatein interfaceILoggingObject- Returns:
- null
-
getExecutingServer
public String getExecutingServer()
Gets the name of the executing server.- Specified by:
getExecutingServerin interfaceIExecutor- Returns:
- the executingServer
-
setExecutingServer
public void setExecutingServer(String executingServer)
Sets the name of the executing server.- Specified by:
setExecutingServerin interfaceIExecutor- Parameters:
executingServer- the executingServer to set
-
getExecutingUser
public String getExecutingUser()
Gets the name of the executing user.- Specified by:
getExecutingUserin interfaceIExecutor- Returns:
- the executingUser
-
setExecutingUser
public void setExecutingUser(String executingUser)
Sets the name of the executing user.- Specified by:
setExecutingUserin interfaceIExecutor- Parameters:
executingUser- the executingUser to set
-
isGatheringMetrics
public boolean isGatheringMetrics()
- Specified by:
isGatheringMetricsin interfaceILoggingObject
-
setGatheringMetrics
public void setGatheringMetrics(boolean gatheringMetrics)
- Specified by:
setGatheringMetricsin interfaceILoggingObject
-
isForcingSeparateLogging
public boolean isForcingSeparateLogging()
- Specified by:
isForcingSeparateLoggingin interfaceILoggingObject
-
setForcingSeparateLogging
public void setForcingSeparateLogging(boolean forcingSeparateLogging)
- Specified by:
setForcingSeparateLoggingin interfaceILoggingObject
-
getResultFiles
public List<ResultFile> getResultFiles()
-
setResultFiles
public void setResultFiles(List<ResultFile> resultFiles)
-
getResultRows
public List<RowMetaAndData> getResultRows()
-
setResultRows
public void setResultRows(List<RowMetaAndData> resultRows)
-
getPreviousResult
public Result getPreviousResult()
Description copied from interface:IPipelineEngineGet the execution result of a previous execution in a workflow- Specified by:
getPreviousResultin interfaceIPipelineEngine<PipelineMeta>- Returns:
- the previous execution result
-
setPreviousResult
public void setPreviousResult(Result previousResult)
Description copied from interface:IPipelineEngineInform the pipeline about a previous execution result in a workflow or pipeline- Specified by:
setPreviousResultin interfaceIPipelineEngine<PipelineMeta>
-
clearError
public void clearError()
Clear the error in the pipeline, clear all the rows from all the row sets, to make sure the pipeline can continue with other data. This is intended for use when running single threaded.
-
getMetadataProvider
public IHopMetadataProvider getMetadataProvider()
- Specified by:
getMetadataProviderin interfaceIPipelineEngine<PipelineMeta>
-
setMetadataProvider
public void setMetadataProvider(IHopMetadataProvider metadataProvider)
- Specified by:
setMetadataProviderin interfaceIPipelineEngine<PipelineMeta>
-
setServletReponse
public void setServletReponse(javax.servlet.http.HttpServletResponse response)
Sets encoding of HttpServletResponse according to System encoding.Check if system encoding is null or an empty and set it to HttpServletResponse when not and writes error to log if null. Throw IllegalArgumentException if input parameter is null.- Parameters:
response- the HttpServletResponse to set encoding, mayn't be null
-
getServletResponse
public javax.servlet.http.HttpServletResponse getServletResponse()
-
setServletRequest
public void setServletRequest(javax.servlet.http.HttpServletRequest request)
-
getServletRequest
public javax.servlet.http.HttpServletRequest getServletRequest()
-
doTopologySortOfTransforms
public void doTopologySortOfTransforms()
-
getExecutionStartDate
public Date getExecutionStartDate()
Gets executionStartDate- Specified by:
getExecutionStartDatein interfaceIPipelineEngine<PipelineMeta>- Returns:
- value of executionStartDate
-
setExecutionStartDate
public void setExecutionStartDate(Date executionStartDate)
- Parameters:
executionStartDate- The executionStartDate to set
-
getExecutionEndDate
public Date getExecutionEndDate()
Gets executionEndDate- Specified by:
getExecutionEndDatein interfaceIPipelineEngine<PipelineMeta>- Returns:
- value of executionEndDate
-
setExecutionEndDate
public void setExecutionEndDate(Date executionEndDate)
- Parameters:
executionEndDate- The executionEndDate to set
-
getExtensionDataMap
public Map<String,Object> getExtensionDataMap()
Description copied from interface:IPipelineEngineA map in which the various transforms and plugins can store objects to remember during the lifecycle of the pipeline. TODO This is obviously harder to do in a clustered or remote execution environment. As such it's not clear that it makes sense to leave this method here over time- Specified by:
getExtensionDataMapin interfaceIExtensionData- Specified by:
getExtensionDataMapin interfaceIPipelineEngine<PipelineMeta>- Returns:
- The extension data map to use
-
getEngineMetrics
public EngineMetrics getEngineMetrics()
Description copied from interface:IPipelineEngineCollect metrics from the various components in the engine.- Specified by:
getEngineMetricsin interfaceIPipelineEngine<PipelineMeta>- Returns:
- The engine metrics
-
getEngineMetrics
public EngineMetrics getEngineMetrics(String componentName, int copyNr)
Description copied from interface:IPipelineEngineGet the engine metrics for a specific component name and/or copy nr- Specified by:
getEngineMetricsin interfaceIPipelineEngine<PipelineMeta>- Parameters:
componentName- the name of the component or null for all componentscopyNr- The copy nr to select or a negative number for all components- Returns:
- The engine metrics for the given
-
getComponentLogText
public String getComponentLogText(String componentName, int copyNr)
Description copied from interface:IPipelineEngineRetrieve the logging text of a particular component in the engine- Specified by:
getComponentLogTextin interfaceIPipelineEngine<PipelineMeta>- Parameters:
componentName- The name of the component (transform)copyNr- The copy number if multiple components are running in parallel (0 based).- Returns:
- The logging text
-
getComponents
public List<IEngineComponent> getComponents()
Description copied from interface:IPipelineEngineGet a list of components- Specified by:
getComponentsin interfaceIPipelineEngine<PipelineMeta>- Returns:
- The list of components in the engine
-
findComponent
public IEngineComponent findComponent(String name, int copyNr)
Description copied from interface:IPipelineEngineFind a component by name and copy nr- Specified by:
findComponentin interfaceIPipelineEngine<PipelineMeta>- Parameters:
name- The name of the component to look for (transform)copyNr- The copy number to match- Returns:
- The component or null if nothing was found
-
getComponentCopies
public List<IEngineComponent> getComponentCopies(String name)
Description copied from interface:IPipelineEngineFind all component copies by name- Specified by:
getComponentCopiesin interfaceIPipelineEngine<PipelineMeta>- Parameters:
name- The name of the component to look for (transform)- Returns:
- The list of components found
-
getPluginId
public String getPluginId()
Gets pluginId- Specified by:
getPluginIdin interfaceIPipelineEngine<PipelineMeta>- Returns:
- value of pluginId
-
setPluginId
public void setPluginId(String pluginId)
- Specified by:
setPluginIdin interfaceIPipelineEngine<PipelineMeta>- Parameters:
pluginId- The pluginId to set
-
createDefaultPipelineEngineRunConfiguration
public IPipelineEngineRunConfiguration createDefaultPipelineEngineRunConfiguration()
Description copied from interface:IPipelineEngineAsk the engine to generate a default pipeline engine configuration for this engine- Specified by:
createDefaultPipelineEngineRunConfigurationin interfaceIPipelineEngine<PipelineMeta>- Returns:
- a new pipeline engine run configuration
-
retrieveComponentOutput
public void retrieveComponentOutput(IVariables variables, String componentName, int copyNr, int nrRows, IPipelineComponentRowsReceived rowsReceived) throws HopException
Description copied from interface:IPipelineEngineRetrieve output rows from a component copy. Pass the rows to the rows received lambda.- Specified by:
retrieveComponentOutputin interfaceIPipelineEngine<PipelineMeta>- Throws:
HopException
-
addStartedListener
public void addStartedListener(IExecutionStartedListener<IPipelineEngine<PipelineMeta>> listener) throws HopException
- Throws:
HopException
-
addFinishedListener
public void addFinishedListener(IExecutionFinishedListener<IPipelineEngine<PipelineMeta>> listener) throws HopException
- Throws:
HopException
-
getRowSetSize
public int getRowSetSize()
Gets rowSetSize- Returns:
- value of rowSetSize
-
setRowSetSize
public void setRowSetSize(int rowSetSize)
- Parameters:
rowSetSize- The rowSetSize to set
-
isFeedbackShown
public boolean isFeedbackShown()
Gets feedbackShown- Specified by:
isFeedbackShownin interfaceIPipelineEngine<PipelineMeta>- Returns:
- value of feedbackShown
-
setFeedbackShown
public void setFeedbackShown(boolean feedbackShown)
- Parameters:
feedbackShown- The feedbackShown to set
-
getFeedbackSize
public int getFeedbackSize()
Gets feedbackSize- Specified by:
getFeedbackSizein interfaceIPipelineEngine<PipelineMeta>- Returns:
- value of feedbackSize
-
setFeedbackSize
public void setFeedbackSize(int feedbackSize)
- Parameters:
feedbackSize- The feedbackSize to set
-
getExecutionStartedListeners
public List<IExecutionStartedListener<IPipelineEngine<PipelineMeta>>> getExecutionStartedListeners()
Gets executionStartedListeners- Returns:
- value of executionStartedListeners
-
setExecutionStartedListeners
public void setExecutionStartedListeners(List<IExecutionStartedListener<IPipelineEngine<PipelineMeta>>> executionStartedListeners)
- Parameters:
executionStartedListeners- The executionStartedListeners to set
-
getExecutionFinishedListeners
public List<IExecutionFinishedListener<IPipelineEngine<PipelineMeta>>> getExecutionFinishedListeners()
Gets executionFinishedListeners- Returns:
- value of executionFinishedListeners
-
setExecutionFinishedListeners
public void setExecutionFinishedListeners(List<IExecutionFinishedListener<IPipelineEngine<PipelineMeta>>> executionFinishedListeners)
- Parameters:
executionFinishedListeners- The executionFinishedListeners to set
-
getPipelineRunConfiguration
public PipelineRunConfiguration getPipelineRunConfiguration()
Gets pipelineRunConfiguration- Specified by:
getPipelineRunConfigurationin interfaceIPipelineEngine<PipelineMeta>- Returns:
- value of pipelineRunConfiguration
-
setPipelineRunConfiguration
public void setPipelineRunConfiguration(PipelineRunConfiguration pipelineRunConfiguration)
- Specified by:
setPipelineRunConfigurationin interfaceIPipelineEngine<PipelineMeta>- Parameters:
pipelineRunConfiguration- The pipelineRunConfiguration to set
-
getActiveSubPipelines
public Map<String,IPipelineEngine> getActiveSubPipelines()
Gets activeSubPipelines- Returns:
- value of activeSubPipelines
-
setActiveSubPipelines
public void setActiveSubPipelines(Map<String,IPipelineEngine> activeSubPipelines)
- Parameters:
activeSubPipelines- The activeSubPipelines to set
-
setActiveSubWorkflows
public void setActiveSubWorkflows(Map<String,IWorkflowEngine<WorkflowMeta>> activeSubWorkflows)
- Parameters:
activeSubWorkflows- The activeSubWorkflows to set
-
getVariables
public IVariables getVariables()
Gets variables- Returns:
- value of variables
-
setVariables
public void setVariables(IVariables variables)
- Parameters:
variables- The variables to set
-
getDataSamplers
public List<IExecutionDataSampler<? extends IExecutionDataSamplerStore>> getDataSamplers()
Gets dataSamplers- Returns:
- value of dataSamplers
-
setDataSamplers
public void setDataSamplers(List<IExecutionDataSampler<? extends IExecutionDataSamplerStore>> dataSamplers)
Sets dataSamplers- Parameters:
dataSamplers- value of dataSamplers
-
addExecutionDataSampler
public <Store extends IExecutionDataSamplerStore,Sampler extends IExecutionDataSampler<Store>> void addExecutionDataSampler(Sampler sampler)
Description copied from interface:IPipelineEnginePlugins can add individual data samplers which will act upon the rows passing through the transforms in the pipeline. The execution information collected at this level will be sent to the execution information location if any is defined in the run configuration used to execute this pipeline.- Specified by:
addExecutionDataSamplerin interfaceIPipelineEngine<PipelineMeta>- Parameters:
sampler- The sampler to use
-
-