org.openucx.jucx.ucp

Class UcpWorker

java.lang.Object

org.openucx.jucx.UcxNativeStruct

org.openucx.jucx.ucp.UcpWorker

All Implemented Interfaces:

Closeable, AutoCloseable

public class UcpWorker
extends UcxNativeStruct
implements Closeable

UCP worker is an opaque object representing the communication context. The worker represents an instance of a local communication resource and the progress engine associated with it. The progress engine is a construct that is responsible for asynchronous and independent progress of communication directives. The progress engine could be implemented in hardware or software. The worker object abstracts an instance of network resources such as a host channel adapter port, network interface, or multiple resources such as multiple network interfaces or communication ports. It could also represent virtual communication resources that are defined across multiple devices. Although the worker can represent multiple network resources, it is associated with a single UcpContext "UCX application context". All communication functions require a context to perform the operation on the dedicated hardware resource(s) and an "endpoint" to address the destination.

Worker are parallel "threading points" that an upper layer may use to optimize concurrent communications.

Constructor Summary

Constructors

Constructor and Description
UcpWorker(UcpContext context, UcpWorkerParams params)

Method Summary

Modifier and Type	Method and Description
void	amDataRelease(long address) This routine releases data that persisted through an Active Message callback because that callback returned UCS_INPROGRESS.
void	arm() This routine needs to be called before waiting on each notification on this worker, so will typically be called once the processing of the previous event is over, as part of the wake-up mechanism.
void	cancelRequest(UcpRequest request) This routine tries to cancels an outstanding communication request.
void	close()
UcpRequest	flushNonBlocking(UcxCallback callback) This routine flushes all outstanding AMO and RMA communications on the this worker.
ByteBuffer	getAddress() This routine returns the address of the worker object.
int	getEventFD() This routine returns a valid file descriptor for polling functions.
long	getMaxAmHeaderSize() Maximal allowed header size for UcpEndpoint.sendAmNonBlocking(int, long, long, long, long, long, org.openucx.jucx.UcxCallback, org.openucx.jucx.ucp.UcpRequestParams) routine.
UcpEndpoint	newEndpoint(UcpEndpointParams params) Creates new UcpEndpoint on current worker.
UcpListener	newListener(UcpListenerParams params) Creates new UcpListener on current worker.
int	progress() This routine explicitly progresses all communication operations on a worker.
void	progressRequest(UcpRequest request) Blocking progress for request until it's not completed.
UcpRequest	recvAmDataNonBlocking(long dataDesc, long address, long size, UcxCallback callback, int memoryType)
UcpRequest	recvAmDataNonBlocking(long dataDesc, long address, long size, UcxCallback callback, UcpRequestParams params) This routine receives a message that is described by the data descriptor dataDesc, local address address and size size on a worker.
UcpRequest	recvTaggedMessageNonBlocking(ByteBuffer buffer, UcpTagMessage message, UcxCallback callback)
UcpRequest	recvTaggedMessageNonBlocking(long address, long size, UcpTagMessage message, UcxCallback callback)
UcpRequest	recvTaggedMessageNonBlocking(long address, long size, UcpTagMessage message, UcxCallback callback, int memoryType)
UcpRequest	recvTaggedMessageNonBlocking(long address, long size, UcpTagMessage message, UcxCallback callback, UcpRequestParams params) Non-blocking receive operation for a probed message.
UcpRequest	recvTaggedNonBlocking(ByteBuffer recvBuffer, long tag, long tagMask, UcxCallback callback) Non-blocking tagged-receive operation.
UcpRequest	recvTaggedNonBlocking(ByteBuffer recvBuffer, UcxCallback callback) Non-blocking receive operation.
UcpRequest	recvTaggedNonBlocking(long[] localAddresses, long[] sizes, long tag, long tagMask, UcxCallback callback)
UcpRequest	recvTaggedNonBlocking(long[] localAddresses, long[] sizes, long tag, long tagMask, UcxCallback callback, int memoryType)
UcpRequest	recvTaggedNonBlocking(long[] localAddresses, long[] sizes, long tag, long tagMask, UcxCallback callback, UcpRequestParams params)
UcpRequest	recvTaggedNonBlocking(long localAddress, long size, long tag, long tagMask, UcxCallback callback)
UcpRequest	recvTaggedNonBlocking(long localAddress, long size, long tag, long tagMask, UcxCallback callback, int memoryType)
UcpRequest	recvTaggedNonBlocking(long localAddress, long size, long tag, long tagMask, UcxCallback callback, UcpRequestParams params)
void	removeAmRecvHandler(int amId) Clears Active Message callback.
void	setAmRecvHandler(int amId, UcpAmRecvCallback callback)
void	setAmRecvHandler(int amId, UcpAmRecvCallback callback, long flags) This routine installs a user defined callback to handle incoming Active Messages with a specific id.
void	signal() This routine signals that the event has happened, as part of the wake-up mechanism.
UcpTagMessage	tagProbeNonBlocking(long tag, long tagMask, boolean remove) Non-blocking probe and return a message.
void	waitForEvents() This routine waits (blocking) until an event has happened, as part of the wake-up mechanism.

Methods inherited from class org.openucx.jucx.UcxNativeStruct

equals, getNativeId, hashCode, setNativeId

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UcpWorker

public UcpWorker(UcpContext context,
                 UcpWorkerParams params)

Method Detail

newEndpoint

public UcpEndpoint newEndpoint(UcpEndpointParams params)

Creates new UcpEndpoint on current worker.

newListener

public UcpListener newListener(UcpListenerParams params)

Creates new UcpListener on current worker.

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

getMaxAmHeaderSize

public long getMaxAmHeaderSize()

Maximal allowed header size for UcpEndpoint.sendAmNonBlocking(int, long, long, long, long, long, org.openucx.jucx.UcxCallback, org.openucx.jucx.ucp.UcpRequestParams) routine.

setAmRecvHandler

public void setAmRecvHandler(int amId,
                             UcpAmRecvCallback callback,
                             long flags)

This routine installs a user defined callback to handle incoming Active Messages with a specific id. This callback is called whenever an Active Message that was sent from the remote peer by @ref ucp_am_send_nbx is received on this worker.

Parameters:

callback - - Active Message callback. To clear the already set callback, this value should be set to null.

setAmRecvHandler

public void setAmRecvHandler(int amId,
                             UcpAmRecvCallback callback)

removeAmRecvHandler

public void removeAmRecvHandler(int amId)

Clears Active Message callback.

amDataRelease

public void amDataRelease(long address)

This routine releases data that persisted through an Active Message callback because that callback returned UCS_INPROGRESS.

recvAmDataNonBlocking

public UcpRequest recvAmDataNonBlocking(long dataDesc,
                                        long address,
                                        long size,
                                        UcxCallback callback,
                                        UcpRequestParams params)

This routine receives a message that is described by the data descriptor dataDesc, local address address and size size on a worker. The routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer.

recvAmDataNonBlocking

public UcpRequest recvAmDataNonBlocking(long dataDesc,
                                        long address,
                                        long size,
                                        UcxCallback callback,
                                        int memoryType)

progress

public int progress()
             throws Exception

This routine explicitly progresses all communication operations on a worker.

Returns:

Non-zero if any communication was progressed, zero otherwise.

Throws:

Exception

progressRequest

public void progressRequest(UcpRequest request)
                     throws Exception

Blocking progress for request until it's not completed.

Throws:

Exception

flushNonBlocking

public UcpRequest flushNonBlocking(UcxCallback callback)

This routine flushes all outstanding AMO and RMA communications on the this worker. All the AMO and RMA operations issued on this worker prior to this call are completed both at the origin and at the target when this call returns.

waitForEvents

public void waitForEvents()

This routine waits (blocking) until an event has happened, as part of the wake-up mechanism. This function is guaranteed to return only if new communication events occur on the worker. Therefore one must drain all existing events before waiting on the file descriptor. This can be achieved by calling progress() repeatedly until it returns 0.

signal

public void signal()

This routine signals that the event has happened, as part of the wake-up mechanism. This function causes a blocking call to waitForEvents() to return, even if no event from the underlying interfaces has taken place. It’s safe to use this routine from any thread, even if UCX is compiled without multi-threading support and/or initialized without UcpWorkerParams.requestThreadSafety(). However UcpContext has to be created with UcpParams.requestWakeupFeature().

arm

public void arm()

This routine needs to be called before waiting on each notification on this worker, so will typically be called once the processing of the previous event is over, as part of the wake-up mechanism. The worker must be armed before waiting on an event (must be re-armed after it has been signaled for re-use) with arm(). The events triggering a signal of the file descriptor from getEventFD() depend on the interfaces used by the worker and defined in the transport layer, and typically represent a request completion or newly available resources. It can also be triggered by calling signal(). The file descriptor is guaranteed to become signaled only if new communication events occur on the worker. Therefore one must drain all existing events before waiting on the file descriptor. This can be achieved by calling progress() repeatedly until it returns 0.

getEventFD

public int getEventFD()

This routine returns a valid file descriptor for polling functions. The file descriptor will get signaled when an event occurs, as part of the wake-up mechanism. Signaling means a call to poll() or select() with this file descriptor will return at this point, with this descriptor marked as the reason (or one of the reasons) the function has returned. The user does not need to release the obtained file descriptor. The wake-up mechanism exists to allow for the user process to register for notifications on events of the underlying interfaces, and wait until such occur. This is an alternative to repeated polling for request completion. The goal is to allow for waiting while consuming minimal resources from the system. This is recommended for cases where traffic is infrequent, and latency can be traded for lower resource consumption while waiting for it. There are two alternative ways to use the wakeup mechanism: the first is the file descriptor obtained per worker (this function) and the second is the waitForEvents() function for waiting on the next event internally.

Returns:

The internal file descriptor, which this worker reports to.

recvTaggedNonBlocking

public UcpRequest recvTaggedNonBlocking(ByteBuffer recvBuffer,
                                        long tag,
                                        long tagMask,
                                        UcxCallback callback)

Non-blocking tagged-receive operation. This routine receives a messages that is described by the local recvBuffer buffer on the current worker. The tag value of the receive message has to match the tag of sent message. The routine is a non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the recvBuffer at position recvBuffer.position() and size recvBuffer.remaining(). In order to notify the application about completion of the receive operation the UCP library will invoke the call-back callback when the received message is in the receive buffer and ready for application access.

Parameters:

tagMask - - bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.

recvTaggedNonBlocking

public UcpRequest recvTaggedNonBlocking(long localAddress,
                                        long size,
                                        long tag,
                                        long tagMask,
                                        UcxCallback callback)

recvTaggedNonBlocking

public UcpRequest recvTaggedNonBlocking(long localAddress,
                                        long size,
                                        long tag,
                                        long tagMask,
                                        UcxCallback callback,
                                        int memoryType)

recvTaggedNonBlocking

public UcpRequest recvTaggedNonBlocking(long localAddress,
                                        long size,
                                        long tag,
                                        long tagMask,
                                        UcxCallback callback,
                                        UcpRequestParams params)

recvTaggedNonBlocking

public UcpRequest recvTaggedNonBlocking(ByteBuffer recvBuffer,
                                        UcxCallback callback)

Non-blocking receive operation. Invokes recvTaggedNonBlocking(ByteBuffer, long, long, UcxCallback) with default tag=0 and tagMask=0.

recvTaggedNonBlocking

public UcpRequest recvTaggedNonBlocking(long[] localAddresses,
                                        long[] sizes,
                                        long tag,
                                        long tagMask,
                                        UcxCallback callback)

recvTaggedNonBlocking

public UcpRequest recvTaggedNonBlocking(long[] localAddresses,
                                        long[] sizes,
                                        long tag,
                                        long tagMask,
                                        UcxCallback callback,
                                        UcpRequestParams params)

recvTaggedNonBlocking

public UcpRequest recvTaggedNonBlocking(long[] localAddresses,
                                        long[] sizes,
                                        long tag,
                                        long tagMask,
                                        UcxCallback callback,
                                        int memoryType)

tagProbeNonBlocking

public UcpTagMessage tagProbeNonBlocking(long tag,
                                         long tagMask,
                                         boolean remove)

Non-blocking probe and return a message. This routine probes (checks) if a messages described by the tag and tagMask was received (fully or partially) on the worker. The tag value of the received message has to match the tag and tagMask values, where the tagMask indicates what bits of the tag have to be matched. The function returns immediately and if the message is matched it returns a handle for the message. This function does not advance the communication state of the network. If this routine is used in busy-poll mode, need to make sure progress() is called periodically to extract messages from the transport.

Parameters:

remove - - The flag indicates if the matched message has to be removed from UCP library. If true, the message handle is removed from the UCP library and the application is responsible to call recvTaggedMessageNonBlocking(long, long, UcpTagMessage, UcxCallback) in order to receive the data and release the resources associated with the message handle. If false, the return value is merely an indication to whether a matching message is present, and it cannot be used in any other way, and in particular it cannot be passed to recvTaggedMessageNonBlocking(long, long, UcpTagMessage, UcxCallback)

Returns:

NULL - No match found. Message handle (not NULL) - If message is matched the message handle is returned.

recvTaggedMessageNonBlocking

public UcpRequest recvTaggedMessageNonBlocking(long address,
                                               long size,
                                               UcpTagMessage message,
                                               UcxCallback callback,
                                               UcpRequestParams params)

Non-blocking receive operation for a probed message. This routine receives a messages that is described by the local address, size, and a message handle. The message handle can be obtain by calling the tagProbeNonBlocking(long, long, boolean). This routine is a non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer, described by address and size. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back callback when the received message is in the receive buffer and ready for application access. If the receive operation cannot be stated the routine returns an error.

recvTaggedMessageNonBlocking

public UcpRequest recvTaggedMessageNonBlocking(long address,
                                               long size,
                                               UcpTagMessage message,
                                               UcxCallback callback,
                                               int memoryType)

recvTaggedMessageNonBlocking

public UcpRequest recvTaggedMessageNonBlocking(long address,
                                               long size,
                                               UcpTagMessage message,
                                               UcxCallback callback)

recvTaggedMessageNonBlocking

public UcpRequest recvTaggedMessageNonBlocking(ByteBuffer buffer,
                                               UcpTagMessage message,
                                               UcxCallback callback)

cancelRequest

public void cancelRequest(UcpRequest request)

This routine tries to cancels an outstanding communication request. After calling this routine, the request will be in completed or canceled (but not both) state regardless of the status of the target endpoint associated with the communication request. If the request is completed successfully, the "send" or the "receive" completion callbacks (based on the type of the request) will be called with the status argument of the callback set to UCS_OK, and in a case it is canceled the status argument is set to UCS_ERR_CANCELED.

getAddress

public ByteBuffer getAddress()

This routine returns the address of the worker object. This address can be passed to remote instances of the UCP library in order to connect to this worker. Ucp worker address - is an opaque object that is used as an identifier for a UcpWorker instance.