org.apache.gobblin.publisher

Class SingleTaskDataPublisher

java.lang.Object

org.apache.gobblin.publisher.DataPublisher

org.apache.gobblin.publisher.SingleTaskDataPublisher

All Implemented Interfaces:

Closeable, AutoCloseable, CapabilityAware

public abstract class SingleTaskDataPublisher
extends DataPublisher

A subclass of DataPublisher that can publish data of a single Task.

Field Summary

Fields inherited from class org.apache.gobblin.publisher.DataPublisher

REUSABLE, state

Constructor Summary

Constructors

Constructor and Description
SingleTaskDataPublisher(State state)

Method Summary

Modifier and Type	Method and Description
static SingleTaskDataPublisher	getInstance(Class<? extends DataPublisher> dataPublisherClass, State state) Get an instance of SingleTaskDataPublisher.
void	publish(WorkUnitState state) First publish the metadata via DataPublisher#publishMetadata(WorkUnitState), and then publish the output data via the DataPublisher#publishData(WorkUnitState) method.
abstract void	publishData(WorkUnitState state) Publish the data for the given task.
abstract void	publishMetadata(WorkUnitState state) Publish the metadata (e.g., schema) for the given task.

Methods inherited from class org.apache.gobblin.publisher.DataPublisher

canBeSkipped, getState, initialize, isThreadSafe, markCommit, publish, publishData, publishMetadata, shouldPublishMetadataFirst, supportsCapability

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.io.Closeable

close

Constructor Detail

SingleTaskDataPublisher

public SingleTaskDataPublisher(State state)

Method Detail

publishData

public abstract void publishData(WorkUnitState state)
                          throws IOException

Publish the data for the given task. The state must contains property "writer.final.output.file.paths" (or "writer.final.output.file.paths.[branchId]" if there are multiple branches), so that the publisher knows which files to publish.

Throws:

IOException

publishMetadata

public abstract void publishMetadata(WorkUnitState state)
                              throws IOException

Publish the metadata (e.g., schema) for the given task. Checkpoints should not be published as part of metadata. They are published by Gobblin runtime after the metadata and data are published.

Throws:

IOException

publish

public void publish(WorkUnitState state)
             throws IOException

First publish the metadata via DataPublisher#publishMetadata(WorkUnitState), and then publish the output data via the DataPublisher#publishData(WorkUnitState) method.

Parameters:

state - is a WorkUnitState.

Throws:

IOException - if there is a problem with publishing the metadata or the data.

getInstance

public static SingleTaskDataPublisher getInstance(Class<? extends DataPublisher> dataPublisherClass,
                                                  State state)
                                           throws ReflectiveOperationException

Get an instance of SingleTaskDataPublisher.

Parameters:

dataPublisherClass - A concrete class that extends SingleTaskDataPublisher.

state - A State used to instantiate the SingleTaskDataPublisher.

Returns:

A SingleTaskDataPublisher instance.

Throws:

ReflectiveOperationException