org.apache.gobblin.publisher

Class DataPublisher

java.lang.Object

org.apache.gobblin.publisher.DataPublisher

All Implemented Interfaces:

Closeable, AutoCloseable, CapabilityAware

Direct Known Subclasses:

SingleTaskDataPublisher

public abstract class DataPublisher
extends Object
implements Closeable, CapabilityAware

Defines how to publish data and its corresponding metadata. Can be used for either task level or job level publishing.

Field Summary

Fields

Modifier and Type	Field and Description
static Capability	REUSABLE Reusable capability.
protected State	state

Constructor Summary

Constructors

Constructor and Description
DataPublisher(State state)

Method Summary

Modifier and Type	Method and Description
boolean	canBeSkipped() Return true if the current publisher can be skipped.
static DataPublisher	getInstance(Class<? extends DataPublisher> dataPublisherClass, State state) Get an instance of DataPublisher.
State	getState()
abstract void	initialize() Deprecated. DataPublisher initialization should be done in the constructor.
boolean	isThreadSafe() Returns true if the implementation of DataPublisher is thread-safe.
protected void	markCommit(Collection<? extends WorkUnitState> states)
void	publish(Collection<? extends WorkUnitState> states) First publish the metadata via publishMetadata(Collection), and then publish the output data via the publishData(Collection) method.
abstract void	publishData(Collection<? extends WorkUnitState> states) Publish the data for the given tasks.
abstract void	publishMetadata(Collection<? extends WorkUnitState> states) Publish the metadata (e.g., schema) for the given tasks.
protected boolean	shouldPublishMetadataFirst() Generally metadata should be published before the data it represents, but this allows subclasses to override if they are dependent on data getting published first.
boolean	supportsCapability(Capability c, Map<String,Object> properties) Checks if this object supports the given Capability with the given properties.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.io.Closeable

close

Field Detail

REUSABLE

public static final Capability REUSABLE

Reusable capability.

state

protected final State state

Constructor Detail

DataPublisher

public DataPublisher(State state)

Method Detail

initialize

@Deprecated
public abstract void initialize()
                                     throws IOException

Deprecated. DataPublisher initialization should be done in the constructor.

Throws:

IOException

publishData

public abstract void publishData(Collection<? extends WorkUnitState> states)
                          throws IOException

Publish the data for the given tasks.

Throws:

IOException

publishMetadata

public abstract void publishMetadata(Collection<? extends WorkUnitState> states)
                              throws IOException

Publish the metadata (e.g., schema) for the given tasks. Checkpoints should not be published as part of metadata. They are published by Gobblin runtime after the metadata and data are published.

Throws:

IOException

publish

public void publish(Collection<? extends WorkUnitState> states)
             throws IOException

First publish the metadata via publishMetadata(Collection), and then publish the output data via the publishData(Collection) method.

Parameters:

states - is a Collection of WorkUnitStates.

Throws:

IOException - if there is a problem with publishing the metadata or the data.

markCommit

protected void markCommit(Collection<? extends WorkUnitState> states)

getState

public State getState()

getInstance

public static DataPublisher getInstance(Class<? extends DataPublisher> dataPublisherClass,
                                        State state)
                                 throws ReflectiveOperationException

Get an instance of DataPublisher.

Parameters:

dataPublisherClass - A concrete class that extends DataPublisher.

state - A State used to instantiate the DataPublisher.

Returns:

A DataPublisher instance.

Throws:

ReflectiveOperationException

isThreadSafe

public boolean isThreadSafe()

Returns true if the implementation of DataPublisher is thread-safe.

For a thread-safe DataPublisher, this method should return this.getClass() == .class to ensure that any extensions must explicitly be marked as thread safe.

canBeSkipped

public boolean canBeSkipped()

Return true if the current publisher can be skipped.

For a publisher that can be skipped, it should not have any effect on state persistence. It will be skipped when a job is cancelled, and all finished tasks are configured to be committed.

shouldPublishMetadataFirst

protected boolean shouldPublishMetadataFirst()

Generally metadata should be published before the data it represents, but this allows subclasses to override if they are dependent on data getting published first.

supportsCapability

public boolean supportsCapability(Capability c,
                                  Map<String,Object> properties)

Description copied from interface: CapabilityAware

Checks if this object supports the given Capability with the given properties. Implementers of this should always check if their super-class may happen to support a capability before returning false!

Specified by:

supportsCapability in interface CapabilityAware

Parameters:

c - Capability being queried

properties - Properties specific to the capability. Properties are capability specific.

Returns:

True if this object supports the given capability + property settings, false if not