Package org.apache.hop.pipeline.transforms.file

Class BaseFileInputTransform<Meta extends BaseFileInputMeta,Data extends BaseFileInputTransformData>

java.lang.Object

org.apache.hop.pipeline.transform.BaseTransform<Meta,Data>

org.apache.hop.pipeline.transforms.file.BaseFileInputTransform<Meta,Data>

All Implemented Interfaces:

IExtensionData, IHasLogChannel, ILoggingObject, IVariables, IEngineComponent, ITransform, IBaseFileInputTransformControl

public abstract class BaseFileInputTransform<Meta extends BaseFileInputMeta,Data extends BaseFileInputTransformData>
extends BaseTransform<Meta,Data>
implements IBaseFileInputTransformControl

This class contains base functionality for file-based input transforms.

Field Summary

Fields inherited from class org.apache.hop.pipeline.transform.BaseTransform

data, deadLockCounter, executionEndDate, executionStartDate, extensionDataMap, first, firstRowReadDate, initStartDate, lastRowWrittenDate, log, meta, metadataProvider, rowListeners, safeStopped, terminator, terminatorRows, variables

Constructor Summary

Constructors

Constructor	Description
BaseFileInputTransform(TransformMeta transformMeta, Meta meta, Data data, int copyNr, PipelineMeta pipelineMeta, Pipeline pipeline)

Method Summary

Modifier and Type	Method	Description
boolean	checkFeedback(long lines)	Check feedback.
protected void	closeLastFile()	Close last opened file/
protected abstract IBaseFileInputReader	createReader(Meta meta, Data data, org.apache.commons.vfs2.FileObject file)	Create reader for specific file.
void	dispose()	Dispose transform.
boolean	failAfterBadFile(String errorMsg)
protected void	fillFileAdditionalFields(Data data, org.apache.commons.vfs2.FileObject file)	Prepare file-dependent data for fill additional fields.
protected boolean	handleOpenFileException(Exception e)
boolean	init()	Initialize transform before execute.
protected boolean	openNextFile()	Open next VFS file for processing.
protected void	prepareToRowProcessing()	Prepare to process.
boolean	processRow()	Process next row.

Methods inherited from class org.apache.hop.pipeline.transform.BaseTransform

addResultFile, addRowListener, addRowSetToInputRowSets, addRowSetToOutputRowSets, addTransformFinishedListener, addTransformStartedListener, batchComplete, canProcessOneRow, cleanup, clearInputRowSets, clearOutputRowSets, closeQuietly, copyFrom, decrementLinesRead, decrementLinesWritten, dispatch, findInputRowSet, findInputRowSet, findOutputRowSet, findOutputRowSet, finishBundle, getContainerId, getCopy, getCopyNr, getCurrentInputRowSetNr, getCurrentOutputRowSetNr, getData, getDispatcher, getErrorRowMeta, getErrors, getExecutionDuration, getExecutionEndDate, getExecutionStartDate, getExtensionDataMap, getFilename, getFirstInputRowSet, getFirstRowReadDate, getInitStartDate, getInputBufferSize, getInputRowMeta, getInputRowSets, getLastRowWrittenDate, getLinesInput, getLinesOutput, getLinesRead, getLinesRejected, getLinesSkipped, getLinesUpdated, getLinesWritten, getLogChannel, getLogChannelId, getLogLevel, getLogText, getMeta, getMetadataProvider, getName, getObjectCopy, getObjectName, getObjectType, getOutputBufferSize, getOutputRowSets, getParent, getParentVariables, getPartitionId, getPartitionTargets, getPipeline, getPipelineMeta, getPreviewRowMeta, getProcessed, getRegistrationDate, getRepartitioning, getResultFiles, getRow, getRowFrom, getRowHandler, getRowListeners, getStatus, getStatusDescription, getTransformFinishedListeners, getTransformMeta, getTransformName, getTransformPluginId, getTransformStartedListeners, getVariable, getVariable, getVariableBoolean, getVariableNames, handleGetRowFrom, handlePutError, handlePutRowTo, identifyErrorOutput, incrementLinesInput, incrementLinesOutput, incrementLinesRead, incrementLinesRejected, incrementLinesSkipped, incrementLinesUpdated, incrementLinesWritten, initBeforeStart, initializeFrom, isAllowEmptyFieldNamesAndTypes, isBasic, isDebug, isDetailed, isDistributed, isForcingSeparateLogging, isGatheringMetrics, isInitialising, isMapping, isPartitioned, isPaused, isRowLevel, isRunning, isSafeStopped, isSelected, isStopped, logBasic, logBasic, logDebug, logDebug, logDetailed, logDetailed, logError, logError, logError, logMinimal, logMinimal, logRowlevel, logRowlevel, logSummary, markStart, markStop, outputIsDone, pauseRunning, putError, putRow, putRowTo, removeRowListener, resolve, resolve, resolve, resumeRunning, rowsetInputSize, rowsetOutputSize, safeModeChecking, safeModeChecking, setAllowEmptyFieldNamesAndTypes, setCarteObjectId, setCopy, setCopyNr, setCurrentInputRowSetNr, setCurrentOutputRowSetNr, setDistributed, setErrorRowMeta, setErrors, setExecutionEndDate, setExecutionStartDate, setFirstRowReadDate, setForcingSeparateLogging, setGatheringMetrics, setInitStartDate, setInputRowMeta, setInputRowSets, setInternalVariables, setLastRowWrittenDate, setLinesInput, setLinesOutput, setLinesRead, setLinesRejected, setLinesSkipped, setLinesUpdated, setLinesWritten, setLogLevel, setMetadataProvider, setOutputDone, setOutputRowSets, setParentVariables, setPartitioned, setPartitionId, setPartitionTargets, setPaused, setPaused, setPreviewRowMeta, setRepartitioning, setRowHandler, setRunning, setSafeStopped, setStopped, setTransformFinishedListeners, setTransformName, setTransformStartedListeners, setVariable, setVariables, shareWith, startBundle, stopAll, stopRunning, swapFirstInputRowSetIfExists, toString, waitUntilPipelineIsStarted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hop.pipeline.transforms.file.IBaseFileInputTransformControl

getErrors, getLinesInput, getLinesWritten, incrementLinesInput, incrementLinesUpdated, putRow, setErrors, stopAll

Methods inherited from interface org.apache.hop.pipeline.transform.ITransform

subStatuses

Constructor Detail

BaseFileInputTransform

public BaseFileInputTransform(TransformMeta transformMeta,
                              Meta meta,
                              Data data,
                              int copyNr,
                              PipelineMeta pipelineMeta,
                              Pipeline pipeline)

Method Detail

createReader

protected abstract IBaseFileInputReader createReader(Meta meta,
                                                     Data data,
                                                     org.apache.commons.vfs2.FileObject file)
                                              throws Exception

Create reader for specific file.

Throws:

Exception

init

public boolean init()

Initialize transform before execute.

Specified by:

init in interface ITransform

Overrides:

init in class BaseTransform<Meta extends BaseFileInputMeta,Data extends BaseFileInputTransformData>

openNextFile

protected boolean openNextFile()

Open next VFS file for processing.

This method will support different parallelization methods later.

handleOpenFileException

protected boolean handleOpenFileException(Exception e)

processRow

public boolean processRow()
                   throws HopException

Process next row. This methods opens next file automatically.

Specified by:

processRow in interface ITransform

Overrides:

processRow in class BaseTransform<Meta extends BaseFileInputMeta,Data extends BaseFileInputTransformData>

Returns:

false if no more rows can be processed or an error occurred.

Throws:

HopException

prepareToRowProcessing

protected void prepareToRowProcessing()
                               throws HopException

Prepare to process. Executed only first time row processing. It can't be possible to prepare to process in the init() phrase, because files can be in fields from previous transform.

Throws:

HopException

checkFeedback

public boolean checkFeedback(long lines)

Description copied from class: BaseTransform

Check feedback.

Specified by:

checkFeedback in interface IBaseFileInputTransformControl

Overrides:

checkFeedback in class BaseTransform<Meta extends BaseFileInputMeta,Data extends BaseFileInputTransformData>

Parameters:

lines - the lines

Returns:

true, if successful

closeLastFile

protected void closeLastFile()

Close last opened file/

dispose

public void dispose()

Dispose transform.

Specified by:

dispose in interface ITransform

Overrides:

dispose in class BaseTransform<Meta extends BaseFileInputMeta,Data extends BaseFileInputTransformData>

failAfterBadFile

public boolean failAfterBadFile(String errorMsg)

Specified by:

failAfterBadFile in interface IBaseFileInputTransformControl

Parameters:

errorMsg - Message to send to rejected row if enabled

Returns:

If should stop processing after having problems with a file

fillFileAdditionalFields

protected void fillFileAdditionalFields(Data data,
                                        org.apache.commons.vfs2.FileObject file)
                                 throws org.apache.commons.vfs2.FileSystemException

Prepare file-dependent data for fill additional fields.

Throws:

org.apache.commons.vfs2.FileSystemException