public class WorkUnitState extends State
WorkUnit instance and additionally holds all the
task runtime state of that WorkUnit.
Properties set in the encapsulated WorkUnit can be overridden at runtime,
with the original values available through the getWorkunit() method.
Getters will return values set at task runtime if available, or the corresponding
values from encapsulated WorkUnit if they are not set at task runtime.
| Modifier and Type | Class and Description |
|---|---|
static class |
WorkUnitState.WorkingState
Runtime state of the
WorkUnit. |
| Constructor and Description |
|---|
WorkUnitState()
Default constructor used for deserialization.
|
WorkUnitState(WorkUnit workUnit)
Deprecated.
It is recommended to use
WorkUnitState(WorkUnit, State) rather than combining properties
in the job state into the workunit. |
WorkUnitState(WorkUnit workUnit,
State jobState)
If creating a
WorkUnitState for use by a task, use WorkUnitState(WorkUnit, State, SharedResourcesBroker)
instead. |
WorkUnitState(WorkUnit workUnit,
State jobState,
SharedResourcesBroker<GobblinScopeTypes> taskBroker) |
WorkUnitState(WorkUnit workUnit,
State jobState,
SubscopedBrokerBuilder<GobblinScopeTypes,?> taskBrokerBuilder) |
| Modifier and Type | Method and Description |
|---|---|
void |
addFinalConstructState(String infix,
State finalConstructState)
Adds all properties from
State to this WorkUnitState. |
void |
backoffActualHighWatermark()
Backoff the actual high watermark to the low watermark returned by
WorkUnit.getLowWatermark(). |
boolean |
contains(String key)
Check if a property is set.
|
boolean |
equals(Object object) |
com.google.gson.JsonElement |
getActualHighWatermark()
Get the actual high
Watermark as a JsonElement. |
<T extends Watermark> |
getActualHighWatermark(Class<T> watermarkClass)
Get the actual high
Watermark. |
<T extends Watermark> |
getActualHighWatermark(Class<T> watermarkClass,
com.google.gson.Gson gson)
Get the actual high
Watermark. |
Extract |
getExtract()
|
State |
getFinalConstructStates()
Builds a State containing all properties added with
addFinalConstructState(java.lang.String, org.apache.gobblin.configuration.State)
to this WorkUnitState. |
long |
getHighWaterMark()
Deprecated.
|
String |
getOutputFilePath() |
State |
getPreviousTableState()
Get properties set in the previous run for the same table as the
WorkUnit. |
String |
getProp(String key)
Get the value of a property.
|
String |
getProp(String key,
String def)
Get the value of a property, using the given default value if the property is not set.
|
Properties |
getProperties()
Return a copy of the underlying
Properties object. |
protected String |
getProperty(String key)
Deprecated.
Use
getProp(String) |
protected String |
getProperty(String key,
String def)
Deprecated.
|
Set<String> |
getPropertyNames()
Get the names of all the properties set in a
Set. |
SharedResourcesBroker<GobblinScopeTypes> |
getTaskBroker()
Get a
SharedResourcesBroker scoped for this task. |
SharedResourcesBroker<GobblinScopeTypes> |
getTaskBrokerNullable()
Get a
SharedResourcesBroker scoped for this task or null if it doesn't exist. |
WorkUnitState.WorkingState |
getWorkingState()
Get the current runtime state of the
WorkUnit. |
WorkUnit |
getWorkunit()
Get an
ImmutableWorkUnit that wraps the internal WorkUnit. |
int |
hashCode() |
void |
readFields(DataInput in) |
void |
removeProp(String key)
Remove a property if it exists.
|
void |
removePropsWithPrefix(String prefix)
Remove all properties with a certain keyPrefix
|
void |
setActualHighWatermark(Watermark watermark)
This method should set the actual, runtime high
Watermark for this WorkUnitState. |
void |
setHighWaterMark(long value)
Deprecated.
|
void |
setJobState(State jobState) |
void |
setWorkingState(WorkUnitState.WorkingState state)
Set the current runtime state of the
WorkUnit. |
void |
setWuProperties(Properties commonProps,
Properties specProps)
Override
workUnit's properties with new commonProps and specProps. |
String |
toString() |
void |
write(DataOutput out) |
addAll, addAll, addAllIfNotExist, addAllIfNotExist, appendToListProp, appendToSetProp, getId, getPropAsBoolean, getPropAsBoolean, getPropAsCaseInsensitiveSet, getPropAsCaseInsensitiveSet, getPropAsDouble, getPropAsDouble, getPropAsInt, getPropAsInt, getPropAsJsonArray, getPropAsList, getPropAsList, getPropAsLong, getPropAsLong, getPropAsSet, getPropAsSet, getPropAsShort, getPropAsShort, getPropAsShortWithRadix, getPropAsShortWithRadix, overrideWith, overrideWith, setId, setProp, setPropspublic WorkUnitState()
@Deprecated public WorkUnitState(WorkUnit workUnit)
WorkUnitState(WorkUnit, State) rather than combining properties
in the job state into the workunit.workUnit - a WorkUnit instance based on which a WorkUnitState instance is constructedpublic WorkUnitState(WorkUnit workUnit, State jobState)
WorkUnitState for use by a task, use WorkUnitState(WorkUnit, State, SharedResourcesBroker)
instead.public WorkUnitState(WorkUnit workUnit, State jobState, SubscopedBrokerBuilder<GobblinScopeTypes,?> taskBrokerBuilder)
public WorkUnitState(WorkUnit workUnit, State jobState, SharedResourcesBroker<GobblinScopeTypes> taskBroker)
public String getOutputFilePath()
public SharedResourcesBroker<GobblinScopeTypes> getTaskBroker()
SharedResourcesBroker scoped for this task.@Nullable public SharedResourcesBroker<GobblinScopeTypes> getTaskBrokerNullable()
SharedResourcesBroker scoped for this task or null if it doesn't exist. This is used for internal calls.public WorkUnit getWorkunit()
ImmutableWorkUnit that wraps the internal WorkUnit.ImmutableWorkUnit that wraps the internal WorkUnitpublic void setWuProperties(Properties commonProps, Properties specProps)
workUnit's properties with new commonProps and specProps.public WorkUnitState.WorkingState getWorkingState()
WorkUnit.WorkUnitState.WorkingState of the WorkUnitpublic void setWorkingState(WorkUnitState.WorkingState state)
WorkUnit.state - WorkUnitState.WorkingState of the WorkUnitpublic com.google.gson.JsonElement getActualHighWatermark()
Watermark as a JsonElement.public <T extends Watermark> T getActualHighWatermark(Class<T> watermarkClass, com.google.gson.Gson gson)
Watermark. If the WorkUnitState does not contain the actual high watermark
(which may be caused by task failures), the low watermark in the corresponding WorkUnit will be returned.watermarkClass - the watermark class for this WorkUnitState.gson - a Gson object used to deserialize the watermark.WorkUnitState. null is returned if this WorkUnitState
does not contain an actual high watermark, and the corresponding WorkUnit does not contain a low
watermark.public <T extends Watermark> T getActualHighWatermark(Class<T> watermarkClass)
Watermark. If the WorkUnitState does not contain the actual high watermark
(which may be caused by task failures), the low watermark in the corresponding WorkUnit will be returned.
A default Gson object will be used to deserialize the watermark.
watermarkClass - the watermark class for this WorkUnitState.WorkUnitState. null is returned if this WorkUnitState
does not contain an actual high watermark, and the corresponding WorkUnit does not contain a low
watermark.public void setActualHighWatermark(Watermark watermark)
Watermark for this WorkUnitState. A high
Watermark indicates that all data for the source has been pulled up to a specific point.
This method should be called inside the Extractor class, during the initialization
of the class, before any calls to Extractor.readRecord(Object) are executed. This
method keeps a local point to the given Watermark and expects the following invariant to always be upheld.
The invariant for this Watermark is that it should cover all records up to and including the most recent
record returned by Extractor.readRecord(Object).
The Watermark set in this method may be polled by the framework multiple times, in order to track the
progress of how the Watermark changes. This is important for reporting percent completion of a
WorkUnit.
Extractor
interface, this method should become part of the Extractor interface. For example,
a method such as getCurrentHighWatermark() should be added.public void backoffActualHighWatermark()
WorkUnit.getLowWatermark().@Deprecated public long getHighWaterMark()
getActualHighWatermark().Extractor.@Deprecated public void setHighWaterMark(long value)
setActualHighWatermark(Watermark).value - high watermarkpublic Properties getProperties()
StateProperties object.getProperties in class StateProperties object.public String getProp(String key)
Statepublic String getProp(String key, String def)
State@Deprecated protected String getProperty(String key)
getProp(String)getProperty in class State@Deprecated protected String getProperty(String key, String def)
getProp(String, String)getProperty in class Statepublic Set<String> getPropertyNames()
StateSet.getPropertyNames in class StateSetpublic boolean contains(String key)
Statepublic void removeProp(String key)
StateremoveProp in class Statekey - property keypublic void removePropsWithPrefix(String prefix)
StateremovePropsWithPrefix in class Stateprefix - key prefixpublic State getPreviousTableState()
WorkUnit.State objectpublic void setJobState(State jobState)
public void readFields(DataInput in) throws IOException
readFields in interface WritableShimreadFields in class StateIOExceptionpublic void write(DataOutput out) throws IOException
write in interface WritableShimwrite in class StateIOExceptionpublic void addFinalConstructState(String infix, State finalConstructState)
State to this WorkUnitState.
A property with name "property" will be added to this object with the key
"FINAL_CONSTRUCT_STATE_PREFIX[.
infix - Optional infix used for the name of the property in the WorkUnitState.finalConstructState - State for which all properties should be added to this
object.public State getFinalConstructStates()
addFinalConstructState(java.lang.String, org.apache.gobblin.configuration.State)
to this WorkUnitState. All such properties will be stripped of
FINAL_CONSTRUCT_STATE_PREFIX but not of any infixes.
For example, if state={sample.property: sampleValue} then
this.addFinalConstructState("infix",state);
this.getFinalConstructState();
will return state={infix.sample.property: sampleValue}
addFinalConstructState(java.lang.String, org.apache.gobblin.configuration.State).