Interface Exchange

All Superinterfaces:

AutoCloseable, Closeable

@ThreadSafe
@Experimental(eta="2024-03-01")
public interface Exchange
extends Closeable

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static enum	Exchange.SourceHandlesDeliveryMode

Method Summary

Modifier and Type	Method	Description
ExchangeSinkHandle	addSink(int taskPartitionId)	Registers a new sink
void	allRequiredSinksFinished()	Called by the engine when all required sinks finished successfully.
void	close()
ExchangeId	getId()	Get id of this exchange
ExchangeSourceHandleSource	getSourceHandles()	Returns an ExchangeSourceHandleSource instance to be used to enumerate ExchangeSourceHandles.
CompletableFuture<ExchangeSinkInstanceHandle>	instantiateSink(ExchangeSinkHandle sinkHandle, int taskAttemptId)	Registers a sink instance for an attempt.
void	noMoreSinks()	Called when no more sinks will be added with addSink(int).
default void	setSourceHandlesDeliveryMode(Exchange.SourceHandlesDeliveryMode sourceHandlesDeliveryMode)	Change ExchangeSourceHandleSource delivery mode.
void	sinkFinished(ExchangeSinkHandle sinkHandle, int taskAttemptId)	Called by the engine when an attempt finishes successfully.
CompletableFuture<ExchangeSinkInstanceHandle>	updateSinkInstanceHandle(ExchangeSinkHandle sinkHandle, int taskAttemptId)	Update ExchangeSinkInstanceHandle.

Method Details

getId

ExchangeId getId()

Get id of this exchange

addSink

ExchangeSinkHandle addSink(int taskPartitionId)

Registers a new sink

Parameters:

taskPartitionId - uniquely identifies a dataset written to a sink

Returns:

ExchangeSinkHandle associated with the taskPartitionId. Must be passed to instantiateSink(ExchangeSinkHandle, int) along with an attempt id to create a sink instance

noMoreSinks

void noMoreSinks()

Called when no more sinks will be added with addSink(int). New sink instances for an existing sink may still be added with instantiateSink(ExchangeSinkHandle, int).

instantiateSink

CompletableFuture<ExchangeSinkInstanceHandle> instantiateSink(ExchangeSinkHandle sinkHandle,
 int taskAttemptId)

Registers a sink instance for an attempt.

Attempts are expected to produce the same data.

The implementation must ensure the data written by unsuccessful attempts is safely discarded. An attempt is considered successful when an ExchangeSink is closed with the ExchangeSink.finish() method. When a node dies before ExchangeSink.finish() is called or when an ExchangeSink is closed with the ExchangeSink.abort() method an attempt should be considered as unsuccessful.

When more than a single attempt is successful the implementation must pick one and discard the output of the other attempts

Parameters:

sinkHandle - - handle returned by addSink

taskAttemptId - - attempt id

Returns:

ExchangeSinkInstanceHandle to be sent to a worker that is needed to create an ExchangeSink instance using ExchangeManager.createSink(ExchangeSinkInstanceHandle)

updateSinkInstanceHandle

CompletableFuture<ExchangeSinkInstanceHandle> updateSinkInstanceHandle(ExchangeSinkHandle sinkHandle,
 int taskAttemptId)

Update ExchangeSinkInstanceHandle. Update is requested by ExchangeSink. The updated ExchangeSinkInstanceHandle is expected to be set by ExchangeSink.updateHandle(ExchangeSinkInstanceHandle).

Parameters:

sinkHandle - - handle returned by addSink

taskAttemptId - - attempt id

Returns:

updated handle

sinkFinished

void sinkFinished(ExchangeSinkHandle sinkHandle,
 int taskAttemptId)

Called by the engine when an attempt finishes successfully.

This method is expected to be lightweight. An implementation shouldn't perform any long running blocking operations within this method.

allRequiredSinksFinished

void allRequiredSinksFinished()

Called by the engine when all required sinks finished successfully. While some source tasks may still be running and writing to their sinks the data written to these sinks could be safely ignored after this method is invoked.

getSourceHandles

ExchangeSourceHandleSource getSourceHandles()

Returns an ExchangeSourceHandleSource instance to be used to enumerate ExchangeSourceHandles.

Returns:

Future containing a list of ExchangeSourceHandle to be sent to a worker that is needed to create an ExchangeSource using ExchangeManager.createSource()

setSourceHandlesDeliveryMode

default void setSourceHandlesDeliveryMode(Exchange.SourceHandlesDeliveryMode sourceHandlesDeliveryMode)

Change ExchangeSourceHandleSource delivery mode.

In Exchange.SourceHandlesDeliveryMode.STANDARD mode the handles are delivered at pace optimized for throughput.

In Exchange.SourceHandlesDeliveryMode.EAGER the handles are delivered as soon as possible even if that would mean each handle corresponds to smaller amount of data, which may be not optimal from throughput.

There are no strict constraints regarding when this method can be called. When called, the newly selected delivery mode will apply to all ExchangeSourceHandleSource instances already obtained via getSourceHandles() method. As well as to those yet to be obtained.

Support for this method is optional and best-effort.

close

void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable