org.apache.gobblin.configuration

Class SourceState

java.lang.Object

org.apache.gobblin.configuration.State

org.apache.gobblin.configuration.SourceState

All Implemented Interfaces:

WritableShim

Direct Known Subclasses:

SourceState

public class SourceState
extends State

A container for all meta data related to a particular source. This includes all properties defined in job configuration files and all properties from tasks of the previous run.

Properties can be overwritten at runtime and persisted upon job completion. Persisted properties will be loaded in the next run and made available to use by the Source.

Constructor Summary

Constructors

Constructor and Description
SourceState() Default constructor.
SourceState(State properties) Constructor.
SourceState(State properties, Iterable<WorkUnitState> previousWorkUnitStates) Constructor.
SourceState(State properties, Map<String,? extends SourceState> previousDatasetStatesByUrns, Iterable<WorkUnitState> previousWorkUnitStates) Constructor.

Method Summary

Modifier and Type	Method and Description
Extract	createExtract(Extract.TableType type, String namespace, String table) Deprecated.
WorkUnit	createWorkUnit(Extract extract) Deprecated. Properties in SourceState should not added to a WorkUnit. Having each WorkUnit contain a copy of SourceState is a waste of memory. Use WorkUnit.create(Extract).
boolean	equals(Object object)
SourceState	getPreviousDatasetState(String datasetUrn) Get the state (in the form of a SourceState) of a dataset identified by a dataset URN of the previous job run.
Map<String,SourceState>	getPreviousDatasetStatesByUrns()
SourceState	getPreviousSourceState() Get the SourceState of the previous job run.
List<WorkUnitState>	getPreviousWorkUnitStates() Get a List of previous WorkUnitStates.
List<WorkUnitState>	getPreviousWorkUnitStates(String datasetUrn) Get a List of previous WorkUnitStates for a given datasetUrn.
Map<String,Iterable<WorkUnitState>>	getPreviousWorkUnitStatesByDatasetUrns() Get a Map from dataset URNs (as being specified by ConfigurationKeys.DATASET_URN_KEY to the WorkUnitState with the dataset URNs.
int	hashCode()
void	readFields(DataInput in)
void	write(DataOutput out)
void	write(DataOutput out, boolean writePreviousWorkUnitStates)

Methods inherited from class org.apache.gobblin.configuration.State

addAll, addAll, addAllIfNotExist, addAllIfNotExist, appendToListProp, appendToSetProp, contains, getId, getProp, getProp, getPropAsBoolean, getPropAsBoolean, getPropAsCaseInsensitiveSet, getPropAsCaseInsensitiveSet, getPropAsDouble, getPropAsDouble, getPropAsInt, getPropAsInt, getPropAsJsonArray, getPropAsList, getPropAsList, getPropAsLong, getPropAsLong, getPropAsSet, getPropAsSet, getPropAsShort, getPropAsShort, getPropAsShortWithRadix, getPropAsShortWithRadix, getProperties, getProperty, getProperty, getPropertyNames, overrideWith, overrideWith, removeProp, removePropsWithPrefix, setId, setProp, setProps, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

SourceState

public SourceState()

Default constructor.

SourceState

public SourceState(State properties)

Constructor.

Parameters:

properties - job configuration properties

SourceState

public SourceState(State properties,
                   Iterable<WorkUnitState> previousWorkUnitStates)

Constructor.

Parameters:

properties - job configuration properties

previousWorkUnitStates - an Iterable of WorkUnitStates of the previous job run

SourceState

public SourceState(State properties,
                   Map<String,? extends SourceState> previousDatasetStatesByUrns,
                   Iterable<WorkUnitState> previousWorkUnitStates)

Constructor.

Parameters:

properties - job configuration properties

previousDatasetStatesByUrns - SourceState of the previous job run

previousWorkUnitStates - an Iterable of WorkUnitStates of the previous job run

Method Detail

getPreviousSourceState

public SourceState getPreviousSourceState()

Get the SourceState of the previous job run.

This is a convenient method for existing jobs that do not use the new feature that allows output data to be committed on a per-dataset basis. Use of this method assumes that the job deals with a single dataset, which uses the default data URN defined by ConfigurationKeys.DEFAULT_DATASET_URN.

Returns:

SourceState of the previous job run or null if no previous SourceState is found

getPreviousDatasetState

public SourceState getPreviousDatasetState(String datasetUrn)

Get the state (in the form of a SourceState) of a dataset identified by a dataset URN of the previous job run. Useful when dataset state store is enabled and we want to load the latest state of a global watermark dataset.

Parameters:

datasetUrn - the dataset URN

Returns:

the dataset state (in the form of a SourceState) of the previous job run or null if no previous dataset state is found for the given dataset URN

getPreviousDatasetStatesByUrns

public Map<String,SourceState> getPreviousDatasetStatesByUrns()

Returns:

a Map from dataset URNs (as being specified by ConfigurationKeys.DATASET_URN_KEY to the SourceState with the dataset URNs. The map is materialized upon invocation of the method by the source. Subsequent calls to this method will return the previously materialized map.

SourceStates that do not have ConfigurationKeys.DATASET_URN_KEY set will be added to the dataset state belonging to ConfigurationKeys.DEFAULT_DATASET_URN.

getPreviousWorkUnitStates

public List<WorkUnitState> getPreviousWorkUnitStates()

Get a List of previous WorkUnitStates. The list is lazily materialized upon invocation of the method by the Source. Subsequent calls to this method will return the previously materialized map.

getPreviousWorkUnitStates

public List<WorkUnitState> getPreviousWorkUnitStates(String datasetUrn)

Get a List of previous WorkUnitStates for a given datasetUrn.

Parameters:

datasetUrn -

Returns:

List of WorkUnitStates.

getPreviousWorkUnitStatesByDatasetUrns

public Map<String,Iterable<WorkUnitState>> getPreviousWorkUnitStatesByDatasetUrns()

Get a Map from dataset URNs (as being specified by ConfigurationKeys.DATASET_URN_KEY to the WorkUnitState with the dataset URNs.

WorkUnitStates that do not have ConfigurationKeys.DATASET_URN_KEY set will be added to the dataset state belonging to ConfigurationKeys.DEFAULT_DATASET_URN.

Returns:

a Map from dataset URNs to the WorkUnitState with the dataset URNs

createExtract

@Deprecated
public Extract createExtract(Extract.TableType type,
                                         String namespace,
                                         String table)

Deprecated.

Create a new properly populated Extract instance.

This method should always return a new unique Extract instance.

Parameters:

type - Extract.TableType

namespace - namespace of the table this extract belongs to

table - name of the table this extract belongs to

Returns:

a new unique Extract instance

createWorkUnit

@Deprecated
public WorkUnit createWorkUnit(Extract extract)

Deprecated. Properties in SourceState should not added to a WorkUnit. Having each WorkUnit contain a copy of SourceState is a waste of memory. Use WorkUnit.create(Extract).

Create a new WorkUnit instance from a given Extract.

Parameters:

extract - given Extract

Returns:

a new WorkUnit instance

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface WritableShim

Overrides:

write in class State

Throws:

IOException

write

public void write(DataOutput out,
                  boolean writePreviousWorkUnitStates)
           throws IOException

Throws:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface WritableShim

Overrides:

readFields in class State

Throws:

IOException

equals

public boolean equals(Object object)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object