Interface Exchange
- All Superinterfaces:
AutoCloseable,Closeable
-
Nested Class Summary
Nested Classes -
Method Summary
Modifier and TypeMethodDescriptionaddSink(int taskPartitionId) Registers a new sinkvoidCalled by the engine when all required sinks finished successfully.voidclose()getId()Get id of this exchangeReturns anExchangeSourceHandleSourceinstance to be used to enumerateExchangeSourceHandles.instantiateSink(ExchangeSinkHandle sinkHandle, int taskAttemptId) Registers a sink instance for an attempt.voidCalled when no more sinks will be added withaddSink(int).default voidsetSourceHandlesDeliveryMode(Exchange.SourceHandlesDeliveryMode sourceHandlesDeliveryMode) ChangeExchangeSourceHandleSourcedelivery mode.voidsinkFinished(ExchangeSinkHandle sinkHandle, int taskAttemptId) Called by the engine when an attempt finishes successfully.updateSinkInstanceHandle(ExchangeSinkHandle sinkHandle, int taskAttemptId) UpdateExchangeSinkInstanceHandle.
-
Method Details
-
getId
ExchangeId getId()Get id of this exchange -
addSink
Registers a new sink- Parameters:
taskPartitionId- uniquely identifies a dataset written to a sink- Returns:
ExchangeSinkHandleassociated with thetaskPartitionId. Must be passed toinstantiateSink(ExchangeSinkHandle, int)along with an attempt id to create a sink instance
-
noMoreSinks
void noMoreSinks()Called when no more sinks will be added withaddSink(int). New sink instances for an existing sink may still be added withinstantiateSink(ExchangeSinkHandle, int). -
instantiateSink
CompletableFuture<ExchangeSinkInstanceHandle> instantiateSink(ExchangeSinkHandle sinkHandle, int taskAttemptId) Registers a sink instance for an attempt.Attempts are expected to produce the same data.
The implementation must ensure the data written by unsuccessful attempts is safely discarded. An attempt is considered successful when an
ExchangeSinkis closed with theExchangeSink.finish()method. When a node dies beforeExchangeSink.finish()is called or when anExchangeSinkis closed with theExchangeSink.abort()method an attempt should be considered as unsuccessful.When more than a single attempt is successful the implementation must pick one and discard the output of the other attempts
- Parameters:
sinkHandle- - handle returned byaddSinktaskAttemptId- - attempt id- Returns:
- ExchangeSinkInstanceHandle to be sent to a worker that is needed to create an
ExchangeSinkinstance usingExchangeManager.createSink(ExchangeSinkInstanceHandle)
-
updateSinkInstanceHandle
CompletableFuture<ExchangeSinkInstanceHandle> updateSinkInstanceHandle(ExchangeSinkHandle sinkHandle, int taskAttemptId) UpdateExchangeSinkInstanceHandle. Update is requested byExchangeSink. The updatedExchangeSinkInstanceHandleis expected to be set byExchangeSink.updateHandle(ExchangeSinkInstanceHandle).- Parameters:
sinkHandle- - handle returned byaddSinktaskAttemptId- - attempt id- Returns:
- updated handle
-
sinkFinished
Called by the engine when an attempt finishes successfully.This method is expected to be lightweight. An implementation shouldn't perform any long running blocking operations within this method.
-
allRequiredSinksFinished
void allRequiredSinksFinished()Called by the engine when all required sinks finished successfully. While some source tasks may still be running and writing to their sinks the data written to these sinks could be safely ignored after this method is invoked. -
getSourceHandles
ExchangeSourceHandleSource getSourceHandles()Returns anExchangeSourceHandleSourceinstance to be used to enumerateExchangeSourceHandles.- Returns:
- Future containing a list of
ExchangeSourceHandleto be sent to a worker that is needed to create anExchangeSourceusingExchangeManager.createSource()
-
setSourceHandlesDeliveryMode
default void setSourceHandlesDeliveryMode(Exchange.SourceHandlesDeliveryMode sourceHandlesDeliveryMode) ChangeExchangeSourceHandleSourcedelivery mode.In
Exchange.SourceHandlesDeliveryMode.STANDARDmode the handles are delivered at pace optimized for throughput.In
Exchange.SourceHandlesDeliveryMode.EAGERthe handles are delivered as soon as possible even if that would mean each handle corresponds to smaller amount of data, which may be not optimal from throughput.There are no strict constraints regarding when this method can be called. When called, the newly selected delivery mode will apply to all
ExchangeSourceHandleSourceinstances already obtained viagetSourceHandles()method. As well as to those yet to be obtained.Support for this method is optional and best-effort.
-
close
void close()- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceCloseable
-