org.apache.gobblin.writer

Interface DataWriter<D>

Type Parameters:

D - data record type

All Superinterfaces:

AutoCloseable, Closeable, Flushable

All Known Implementing Classes:

WriterWrapper

public interface DataWriter<D>
extends Closeable, Flushable

An interface for data writers

Generally, one work unit has a dedicated DataWriter instance, which processes only one dataset

Method Summary

Modifier and Type	Method and Description
long	bytesWritten() Get the number of bytes written.
void	cleanup() Cleanup context/resources.
void	commit() Commit the data written.
default void	flush() Flush data written by the writer.
default Descriptor	getDataDescriptor() The method should return a Descriptor that represents what the writer is writing
default ControlMessageHandler	getMessageHandler() Default handler calls flush on this object when a FlushControlMessage is received
long	recordsWritten() Get the number of records written.
default void	write(D record) Write a data record.
default void	writeEnvelope(RecordEnvelope<D> recordEnvelope) Write the input RecordEnvelope.

Methods inherited from interface java.io.Closeable

close

Method Detail

write

default void write(D record)
            throws IOException

Write a data record.

Parameters:

record - data record to write

Throws:

IOException - if there is anything wrong writing the record

commit

void commit()
     throws IOException

Commit the data written. This method is expected to be called at most once during the lifetime of a writer.

Throws:

IOException - if there is anything wrong committing the output

cleanup

void cleanup()
      throws IOException

Cleanup context/resources.

Throws:

IOException - if there is anything wrong doing cleanup.

recordsWritten

long recordsWritten()

Get the number of records written.

Returns:

number of records written

bytesWritten

long bytesWritten()
           throws IOException

Get the number of bytes written.

Returns:

number of bytes written

Throws:

IOException

getDataDescriptor

default Descriptor getDataDescriptor()

The method should return a Descriptor that represents what the writer is writing

Note that, this information might be useless and discarded by a DataPublisher, which determines the final form of dataset or partition

Returns:

a DatasetDescriptor if it writes files of a dataset or a PartitionDescriptor if it writes files of a dataset partition or null if it is useless

writeEnvelope

default void writeEnvelope(RecordEnvelope<D> recordEnvelope)
                    throws IOException

Write the input RecordEnvelope. By default, just call write(Object). DataWriters that implement this method must acknowledge the recordEnvelope once the write has been acknowledged by the destination system.

Throws:

IOException

getMessageHandler

default ControlMessageHandler getMessageHandler()

Default handler calls flush on this object when a FlushControlMessage is received

Returns:

A ControlMessageHandler.

flush

default void flush()
            throws IOException

Flush data written by the writer. By default, does nothing. This method is expected to be called multiple times during the lifetime of a writer.

Specified by:

flush in interface Flushable

Throws:

IOException