org.apache.gobblin.converter

Class Converter<SI,SO,DI,DO>

java.lang.Object

org.apache.gobblin.converter.Converter<SI,SO,DI,DO>

Type Parameters:

SI - input schema type

SO - output schema type

DI - input data type

DO - output data type

All Implemented Interfaces:

Closeable, AutoCloseable, RecordStreamProcessor<SI,SO,DI,DO>, FinalState

public abstract class Converter<SI,SO,DI,DO>
extends Object
implements Closeable, FinalState, RecordStreamProcessor<SI,SO,DI,DO>

An interface for classes that implement data transformations, e.g., data type conversions, schema projections, data manipulations, data filtering, etc.

This interface is responsible for converting both schema and data records. Classes implementing this interface are composible and can be chained together to achieve more complex data transformations.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.gobblin.records.RecordStreamProcessor

RecordStreamProcessor.StreamProcessingException

Constructor Summary

Constructors

Constructor and Description
Converter()

Method Summary

Modifier and Type	Method and Description
void	close()
abstract Iterable<DO>	convertRecord(SO outputSchema, DI inputRecord, WorkUnitState workUnit) Convert an input data record to an Iterable of output records conforming to the output schema of convertSchema(SI, org.apache.gobblin.configuration.WorkUnitState).
protected io.reactivex.Flowable<RecordEnvelope<DO>>	convertRecordEnvelope(SO outputSchema, RecordEnvelope<DI> inputRecordEnvelope, WorkUnitState workUnitState) Converts a RecordEnvelope.
abstract SO	convertSchema(SI inputSchema, WorkUnitState workUnit) Convert an input schema to an output schema.
State	getFinalState() Get final state for this object.
ConverterInitializer	getInitializer(State state, WorkUnitStream workUnits, int branches, int branchId)
ControlMessageHandler	getMessageHandler()
Converter<SI,SO,DI,DO>	init(WorkUnitState workUnit) Initialize this Converter.
RecordStreamWithMetadata<DO,SO>	processStream(RecordStreamWithMetadata<DI,SI> inputStream, WorkUnitState workUnitState) Apply conversions to the input RecordStreamWithMetadata.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Converter

public Converter()

Method Detail

init

public Converter<SI,SO,DI,DO> init(WorkUnitState workUnit)

Initialize this Converter.

Parameters:

workUnit - a WorkUnitState object carrying configuration properties

Returns:

an initialized Converter instance

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

convertSchema

public abstract SO convertSchema(SI inputSchema,
                                 WorkUnitState workUnit)
                          throws SchemaConversionException

Convert an input schema to an output schema.

Schema conversion is limited to have a 1-to-1 mapping between the input and output schema. When try to convert avro schema, please call to preserve schame creationTime

Parameters:

inputSchema - input schema to be converted

workUnit - a WorkUnitState object carrying configuration properties

Returns:

converted output schema

Throws:

SchemaConversionException - if it fails to convert the input schema

convertRecord

public abstract Iterable<DO> convertRecord(SO outputSchema,
                                           DI inputRecord,
                                           WorkUnitState workUnit)
                                    throws DataConversionException

Convert an input data record to an Iterable of output records conforming to the output schema of convertSchema(SI, org.apache.gobblin.configuration.WorkUnitState).

Record conversion can have a 1-to-0 mapping, 1-to-1 mapping, or 1-to-many mapping between the input and output records as long as all the output records conforms to the same converted schema.

Converter for data record, both record type conversion, and record manipulation conversion.

Parameters:

outputSchema - output schema converted using the convertSchema(SI, org.apache.gobblin.configuration.WorkUnitState) method

inputRecord - input data record to be converted

workUnit - a WorkUnitState object carrying configuration properties

Returns:

converted data record

Throws:

DataConversionException - if it fails to convert the input data record

convertRecordEnvelope

protected io.reactivex.Flowable<RecordEnvelope<DO>> convertRecordEnvelope(SO outputSchema,
                                                                          RecordEnvelope<DI> inputRecordEnvelope,
                                                                          WorkUnitState workUnitState)
                                                                   throws DataConversionException

Converts a RecordEnvelope. This method can be overridden by implementations that need to manipulate the RecordEnvelope, such as to set watermarks or metadata.

Parameters:

outputSchema - output schema converted using the convertSchema(SI, org.apache.gobblin.configuration.WorkUnitState) method

inputRecordEnvelope - input record envelope with data record to be converted

workUnitState - a WorkUnitState object carrying configuration properties

Returns:

a Flowable emitting the converted RecordEnvelopes

Throws:

DataConversionException

getFinalState

public State getFinalState()

Get final state for this object. By default this returns an empty State, but concrete subclasses can add information that will be added to the task state.

Specified by:

getFinalState in interface FinalState

Returns:

Empty State.

processStream

public RecordStreamWithMetadata<DO,SO> processStream(RecordStreamWithMetadata<DI,SI> inputStream,
                                                     WorkUnitState workUnitState)
                                              throws SchemaConversionException

Apply conversions to the input RecordStreamWithMetadata.

Specified by:

processStream in interface RecordStreamProcessor<SI,SO,DI,DO>

Throws:

SchemaConversionException

getMessageHandler

public ControlMessageHandler getMessageHandler()

Returns:

ControlMessageHandler to call for each ControlMessage received.

getInitializer

public ConverterInitializer getInitializer(State state,
                                           WorkUnitStream workUnits,
                                           int branches,
                                           int branchId)