org.apache.gobblin.source

Interface Source<S,D>

Type Parameters:

S - output schema type

D - output record type

All Known Subinterfaces:

WorkUnitStreamSource<S,D>

public interface Source<S,D>

An interface for classes that the end users implement to work with a data source from which schema and data records can be extracted.

An implementation of this interface should contain all the logic required to work with a specific data source. This usually includes work determination and partitioning, and details of the connection protocol to work with the data source.

Method Summary

Modifier and Type	Method and Description
Extractor<S,D>	getExtractor(WorkUnitState state) Get an Extractor based on a given WorkUnitState.
List<WorkUnit>	getWorkunits(SourceState state) Get a list of WorkUnits, each of which is for extracting a portion of the data.
default boolean	isEarlyStopped() Instead of handling all WorkUnits in one run, some Source may choose to stop early in order to handle the proper workload, which can cause multiple runs after the initial run.
void	shutdown(SourceState state) Shutdown this Source instance.

Method Detail

getWorkunits

List<WorkUnit> getWorkunits(SourceState state)

Get a list of WorkUnits, each of which is for extracting a portion of the data.

Each WorkUnit will be used instantiate a WorkUnitState that gets passed to the getExtractor(org.apache.gobblin.configuration.WorkUnitState) method to get an Extractor for extracting schema and data records from the source. The WorkUnit instance should have all the properties needed for the Extractor to work.

Typically the list of WorkUnits for the current run is determined by taking into account the list of WorkUnits from the previous run so data gets extracted incrementally. The method SourceState.getPreviousWorkUnitStates() can be used to get the list of WorkUnits from the previous run.

Parameters:

state - see SourceState

Returns:

a list of WorkUnits

getExtractor

Extractor<S,D> getExtractor(WorkUnitState state)
                     throws IOException

Get an Extractor based on a given WorkUnitState.

The Extractor returned can use WorkUnitState to store arbitrary key-value pairs that will be persisted to the state store and loaded in the next scheduled job run.

Parameters:

state - a WorkUnitState carrying properties needed by the returned Extractor

Returns:

an Extractor used to extract schema and data records from the data source

Throws:

IOException - if it fails to create an Extractor

shutdown

void shutdown(SourceState state)

Shutdown this Source instance.

This method is called once when the job completes. Properties (key-value pairs) added to the input SourceState instance will be persisted and available to the next scheduled job run through the method getWorkunits(SourceState). If there is no cleanup or reporting required for a particular implementation of this interface, then it is acceptable to have a default implementation of this method.

Parameters:

state - see SourceState

isEarlyStopped

default boolean isEarlyStopped()

Instead of handling all WorkUnits in one run, some Source may choose to stop early in order to handle the proper workload, which can cause multiple runs after the initial run.

Returns:

If the same job has early stopped