org.apache.gobblin.source.workunit

Class WorkUnit

java.lang.Object

org.apache.gobblin.configuration.State

org.apache.gobblin.source.workunit.WorkUnit

All Implemented Interfaces:

WritableShim

Direct Known Subclasses:

ImmutableWorkUnit, MultiWorkUnit, WorkUnit

public class WorkUnit
extends State

A logic concept that defines a unit of work or task for extracting a portion of the data to be pulled in a job run.

An instance of this class should contain all the properties an Extractor needs to extract the schema and data records.

Constructor Summary

Constructors

Constructor and Description
WorkUnit() Deprecated. Use createEmpty()
WorkUnit(Extract extract) Constructor.
WorkUnit(SourceState state, Extract extract) Deprecated. Properties in SourceState should not be added to a WorkUnit. Having each WorkUnit contain a copy of SourceState is a waste of memory. Use create(Extract).
WorkUnit(SourceState state, Extract extract, WatermarkInterval watermarkInterval) Deprecated. Properties in SourceState should not be added to a WorkUnit. Having each WorkUnit contain a copy of SourceState is a waste of memory. Use create(Extract, WatermarkInterval).
WorkUnit(WorkUnit other) Deprecated. Use copyOf(WorkUnit)

Method Summary

Modifier and Type	Method and Description
boolean	contains(String key) Check if a property is set.
static WorkUnit	copyOf(WorkUnit other) Factory method.
static WorkUnit	create(Extract extract) Factory method.
static WorkUnit	create(Extract extract, WatermarkInterval watermarkInterval) Factory method.
static WorkUnit	createEmpty() Factory method.
boolean	equals(Object object)
com.google.gson.JsonElement	getExpectedHighWatermark() Get the expected high Watermark as a JsonElement.
<T extends Watermark> T	getExpectedHighWatermark(Class<T> watermarkClass) Get the expected high Watermark.
<T extends Watermark> T	getExpectedHighWatermark(Class<T> watermarkClass, com.google.gson.Gson gson) Get the expected high Watermark.
Extract	getExtract() Get the Extract associated with this WorkUnit.
long	getHighWaterMark() Deprecated. use the getExpectedHighWatermark() method.
com.google.gson.JsonElement	getLowWatermark() Get the low Watermark as a JsonElement.
long	getLowWaterMark() Deprecated. use the getLowWatermark() method.
<T extends Watermark> T	getLowWatermark(Class<T> watermarkClass) Get the low Watermark.
<T extends Watermark> T	getLowWatermark(Class<T> watermarkClass, com.google.gson.Gson gson) Get the low Watermark.
String	getOutputFilePath()
String	getProp(String key) Get the value of a property.
String	getProp(String key, String def) Get the value of a property, using the given default value if the property is not set.
int	hashCode()
void	readFields(DataInput in)
void	setHighWaterMark(long highWaterMark) Deprecated. use setWatermarkInterval(WatermarkInterval).
void	setLowWaterMark(long lowWaterMark) Deprecated. use setWatermarkInterval(WatermarkInterval).
void	setWatermarkInterval(WatermarkInterval watermarkInterval) Set WatermarkInterval for a WorkUnit.
void	skip() This method will allow a work unit to be skipped if needed.
void	write(DataOutput out)

Methods inherited from class org.apache.gobblin.configuration.State

addAll, addAll, addAllIfNotExist, addAllIfNotExist, appendToListProp, appendToSetProp, getId, getPropAsBoolean, getPropAsBoolean, getPropAsCaseInsensitiveSet, getPropAsCaseInsensitiveSet, getPropAsDouble, getPropAsDouble, getPropAsInt, getPropAsInt, getPropAsJsonArray, getPropAsList, getPropAsList, getPropAsLong, getPropAsLong, getPropAsSet, getPropAsSet, getPropAsShort, getPropAsShort, getPropAsShortWithRadix, getPropAsShortWithRadix, getProperties, getProperty, getProperty, getPropertyNames, overrideWith, overrideWith, removeProp, removePropsWithPrefix, setId, setProp, setProps, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

WorkUnit

@Deprecated
public WorkUnit()

Deprecated. Use createEmpty()

Default constructor.

WorkUnit

@Deprecated
public WorkUnit(SourceState state,
                            Extract extract)

Deprecated. Properties in SourceState should not be added to a WorkUnit. Having each WorkUnit contain a copy of SourceState is a waste of memory. Use create(Extract).

Constructor.

Parameters:

state - a SourceState the properties of which will be copied into this WorkUnit instance

extract - an Extract

WorkUnit

@Deprecated
public WorkUnit(SourceState state,
                            Extract extract,
                            WatermarkInterval watermarkInterval)

Deprecated. Properties in SourceState should not be added to a WorkUnit. Having each WorkUnit contain a copy of SourceState is a waste of memory. Use create(Extract, WatermarkInterval).

Constructor for a WorkUnit given a SourceState, Extract, and a WatermarkInterval.

Parameters:

state - a SourceState the properties of which will be copied into this WorkUnit instance.

extract - an Extract.

watermarkInterval - a WatermarkInterval which defines the range of data this WorkUnit will process.

WorkUnit

public WorkUnit(Extract extract)

Constructor.

Parameters:

extract - a Extract object

WorkUnit

@Deprecated
public WorkUnit(WorkUnit other)

Deprecated. Use copyOf(WorkUnit)

Copy constructor.

Parameters:

other - the other WorkUnit instance

Method Detail

createEmpty

public static WorkUnit createEmpty()

Factory method.

Returns:

An empty WorkUnit.

create

public static WorkUnit create(Extract extract)

Factory method.

Parameters:

extract - Extract

Returns:

A WorkUnit with the given Extract

create

public static WorkUnit create(Extract extract,
                              WatermarkInterval watermarkInterval)

Factory method.

Parameters:

extract - Extract

watermarkInterval - WatermarkInterval

Returns:

A WorkUnit with the given Extract and WatermarkInterval

copyOf

public static WorkUnit copyOf(WorkUnit other)

Factory method.

Parameters:

other - a WorkUnit instance

Returns:

A copy of the given WorkUnit instance

getExtract

public Extract getExtract()

Get the Extract associated with this WorkUnit.

Returns:

the Extract associated with this WorkUnit

skip

public void skip()

This method will allow a work unit to be skipped if needed.

getLowWatermark

public com.google.gson.JsonElement getLowWatermark()

Get the low Watermark as a JsonElement.

Returns:

a JsonElement representing the low Watermark or null if the low Watermark is not set.

getLowWatermark

public <T extends Watermark> T getLowWatermark(Class<T> watermarkClass,
                                               com.google.gson.Gson gson)

Get the low Watermark.

Parameters:

watermarkClass - the watermark class for this WorkUnit.

gson - a Gson object used to deserialize the watermark.

Returns:

the low watermark in this WorkUnit.

getLowWatermark

public <T extends Watermark> T getLowWatermark(Class<T> watermarkClass)

Get the low Watermark. A default Gson object will be used to deserialize the watermark.

Parameters:

watermarkClass - the watermark class for this WorkUnit.

Returns:

the low watermark in this WorkUnit.

getExpectedHighWatermark

public com.google.gson.JsonElement getExpectedHighWatermark()

Get the expected high Watermark as a JsonElement.

Returns:

a JsonElement representing the expected high Watermark.

getExpectedHighWatermark

public <T extends Watermark> T getExpectedHighWatermark(Class<T> watermarkClass,
                                                        com.google.gson.Gson gson)

Get the expected high Watermark.

Parameters:

watermarkClass - the watermark class for this WorkUnit.

gson - a Gson object used to deserialize the watermark.

Returns:

the expected high watermark in this WorkUnit.

getExpectedHighWatermark

public <T extends Watermark> T getExpectedHighWatermark(Class<T> watermarkClass)

Get the expected high Watermark. A default Gson object will be used to deserialize the watermark.

Parameters:

watermarkClass - the watermark class for this WorkUnit.

Returns:

the expected high watermark in this WorkUnit.

getHighWaterMark

@Deprecated
public long getHighWaterMark()

Deprecated. use the getExpectedHighWatermark() method.

Get the high watermark of this WorkUnit.

Returns:

high watermark

setWatermarkInterval

public void setWatermarkInterval(WatermarkInterval watermarkInterval)

Set WatermarkInterval for a WorkUnit.

setHighWaterMark

@Deprecated
public void setHighWaterMark(long highWaterMark)

Deprecated. use setWatermarkInterval(WatermarkInterval).

Set the high watermark of this WorkUnit.

Parameters:

highWaterMark - high watermark

getLowWaterMark

@Deprecated
public long getLowWaterMark()

Deprecated. use the getLowWatermark() method.

Get the low watermark of this WorkUnit.

Returns:

low watermark

contains

public boolean contains(String key)

Description copied from class: State

Check if a property is set.

Overrides:

contains in class State

Parameters:

key - property key

Returns:

true if the property is set or false otherwise

getProp

public String getProp(String key)

Description copied from class: State

Get the value of a property.

Overrides:

getProp in class State

Parameters:

key - property key

Returns:

value associated with the key as a string or null if the property is not set

getProp

public String getProp(String key,
                      String def)

Description copied from class: State

Get the value of a property, using the given default value if the property is not set.

Overrides:

getProp in class State

Parameters:

key - property key

def - default value

Returns:

value associated with the key or the default value if the property is not set

setLowWaterMark

@Deprecated
public void setLowWaterMark(long lowWaterMark)

Deprecated. use setWatermarkInterval(WatermarkInterval).

Set the low watermark of this WorkUnit.

Parameters:

lowWaterMark - low watermark

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface WritableShim

Overrides:

readFields in class State

Throws:

IOException

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface WritableShim

Overrides:

write in class State

Throws:

IOException

equals

public boolean equals(Object object)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

getOutputFilePath

public String getOutputFilePath()