org.apache.gobblin.source.extractor

Interface Extractor<S,D>

Type Parameters:

S - output schema type

D - output record type

All Superinterfaces:

AutoCloseable, Closeable

All Known Subinterfaces:

StreamingExtractor<S,D>

public interface Extractor<S,D>
extends Closeable

An interface for classes that are responsible for extracting data from a data source.

All source specific logic for a data source should be encapsulated in an implementation of this interface and Source.

Method Summary

Modifier and Type	Method and Description
long	getExpectedRecordCount() Get the expected source record count.
long	getHighWatermark() Deprecated. there is no longer support for reporting the high watermark via this method, please see Watermarks for more information.
S	getSchema() Get the schema (metadata) of the extracted data records.
default D	readRecord(D reuse) Read the next data record from the data source.
default RecordEnvelope<D>	readRecordEnvelope() Read an RecordEnvelope.
default StreamEntity<D>	readStreamEntity() Read an StreamEntity.
default RecordStreamWithMetadata<D,S>	recordStream(AtomicBoolean shutdownRequest)
default void	shutdown() Called to notify the Extractor it should shut down as soon as possible.

Methods inherited from interface java.io.Closeable

close

Method Detail

getSchema

S getSchema()
     throws IOException

Get the schema (metadata) of the extracted data records.

Returns:

schema of the extracted data records

Throws:

IOException - if there is problem getting the schema

readRecord

@Nullable
default D readRecord(@Deprecated
                               D reuse)
                        throws DataRecordException,
                               IOException

Read the next data record from the data source.

Reuse of data records has been deprecated and is not executed internally.

Parameters:

reuse - the data record object to be reused

Returns:

the next data record extracted from the data source

Throws:

DataRecordException - if there is problem with the extracted data record

IOException - if there is problem extracting the next data record from the source

getExpectedRecordCount

long getExpectedRecordCount()

Get the expected source record count.

Returns:

the expected source record count

getHighWatermark

@Deprecated
long getHighWatermark()

Deprecated. there is no longer support for reporting the high watermark via this method, please see Watermarks for more information.

Get the calculated high watermark up to which data records are to be extracted.

Returns:

the calculated high watermark

shutdown

default void shutdown()
               throws JobShutdownException

Called to notify the Extractor it should shut down as soon as possible. If this call returns successfully, the task will continue consuming records from the Extractor and continue execution normally. The extractor should only emit those records necessary to stop at a graceful committable state. Most job executors will eventually kill the task if the Extractor does not stop emitting records after a few seconds.

Throws:

JobShutdownException - if the extractor does not support early termination. This will cause the task to fail.

readRecordEnvelope

default RecordEnvelope<D> readRecordEnvelope()
                                      throws DataRecordException,
                                             IOException

Read an RecordEnvelope. By default, just wrap readRecord(Object) in a RecordEnvelope.

Throws:

DataRecordException

IOException

readStreamEntity

default StreamEntity<D> readStreamEntity()
                                  throws DataRecordException,
                                         IOException

Read an StreamEntity. By default, just return result of readRecordEnvelope().

Throws:

DataRecordException

IOException

recordStream

default RecordStreamWithMetadata<D,S> recordStream(AtomicBoolean shutdownRequest)
                                            throws IOException

Parameters:

shutdownRequest - an AtomicBoolean that becomes true when a shutdown has been requested.

Returns:

a Flowable with the records from this source. Note the flowable should honor downstream backpressure.

Throws:

IOException