org.nd4j.parameterserver.distributed.v2.transport.impl

Class BaseTransport

java.lang.Object

org.nd4j.parameterserver.distributed.v2.transport.impl.BaseTransport

All Implemented Interfaces:

Transport

Direct Known Subclasses:

AeronUdpTransport, DummyTransport

public abstract class BaseTransport
extends Object
implements Transport

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	BaseTransport.HeartbeatThread
static class	BaseTransport.MessageFlow<T> Generic Publisher/Consumer implementation for interconnect

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,io.reactivex.functions.Consumer>	consumers
protected ThreadPoolExecutor	executorService
protected org.nd4j.common.primitives.AtomicBoolean	handshakeFlag
protected MessagesHistoryHolder<String>	historyHolder
protected String	id
protected BaseTransport.MessageFlow<INDArrayMessage>	incomingFlow
protected boolean	masterMode
protected org.nd4j.common.primitives.Atomic<MeshOrganizer>	mesh
protected MeshBuildMode	meshBuildMode
protected TransferQueue<VoidMessage>	messageQueue
protected AtomicInteger	numerOfNodes
protected BaseTransport.MessageFlow<VoidMessage>	outgoingFlow
protected Map<String,ResponseMessage>	replies
protected RestartCallback	restartCallback
protected String	rootId
protected MessageSplitter	splitter
protected VoidConfiguration	voidConfiguration

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseTransport()
protected	BaseTransport(@NonNull String rootId)
protected	BaseTransport(@NonNull String ownId, @NonNull String rootId, @NonNull VoidConfiguration voidConfiguration)
protected	BaseTransport(@NonNull String rootId, @NonNull VoidConfiguration voidConfiguration)

Method Summary

Modifier and Type	Method and Description
<T extends RequestMessage> void	addRequestConsumer(@NonNull Class<T> cls, io.reactivex.functions.Consumer<T> consumer) This methd allows to set callback instance for various
void	ensureConnection(String id) This method checks connection to the given node ID, and if it's not connected - establishes connection
String	getRandomDownstreamFrom(@NonNull String id, String exclude) This method returns random
String	getRootId() This method returns ID of the root node
String	getUpstreamId() This methos returns Id of the upstream node
org.reactivestreams.Publisher<INDArrayMessage>	incomingPublisher() This method returns flow of messages for parameter server
protected void	internalProcessMessage(VoidMessage message)
boolean	isConnected() This method checks if all connections required for work are established
boolean	isIntroduced() This method checks if this node was properly introduced to driver
protected boolean	isLoopedNode(MeshOrganizer.Node node, @NonNull String originatorId, @NonNull String relayId)
void	launch() This method starts this Transport instance
void	launchAsMaster() This method will start this Transport instance
void	onMeshUpdate(MeshOrganizer mesh) This method will be called if mesh update was received PLEASE NOTE: This method will be called ONLY if new mesh differs from current one
void	onRemap(String id) This method will be called upon remap request
io.reactivex.functions.Consumer<VoidMessage>	outgoingConsumer() This method returns consumer that accepts messages for delivery
void	processMessage(VoidMessage message) This method will be invoked for all incoming messages PLEASE NOTE: this method is mostly suited for tests
protected void	propagateArrayMessage(INDArrayMessage message, PropagationMode mode)
protected void	propagateBroadcastableMessage(@NonNull BroadcastableMessage voidMessage, PropagationMode mode)
void	propagateMessage(@NonNull VoidMessage voidMessage, PropagationMode mode) This method will send message to the network, using tree structure
void	propagateMessageDirect(@NonNull BroadcastableMessage message)
<T extends ResponseMessage> T	sendMessageBlocking(@NonNull RequestMessage message, @NonNull String id) This method will send message to specified node, and will return its response
<T extends ResponseMessage> T	sendMessageBlocking(@NonNull RequestMessage message, @NonNull String id, long timeWait, @NonNull TimeUnit timeUnit) This method will send message to specified node, and will return its response
void	setRestartCallback(RestartCallback callback) This method allows to set callback instance, which will be called upon restart event
void	shutdown() This method shuts down this Transport instance
int	totalNumberOfNodes() This method returns total number of nodes known to this Transport

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.parameterserver.distributed.v2.transport.Transport

id, sendMessage

Field Detail

outgoingFlow

protected final BaseTransport.MessageFlow<VoidMessage> outgoingFlow

incomingFlow

protected final BaseTransport.MessageFlow<INDArrayMessage> incomingFlow

mesh

protected final org.nd4j.common.primitives.Atomic<MeshOrganizer> mesh

id

protected String id

rootId

protected String rootId

masterMode

protected boolean masterMode

replies

protected final Map<String,ResponseMessage> replies

restartCallback

protected RestartCallback restartCallback

consumers

protected Map<String,io.reactivex.functions.Consumer> consumers

voidConfiguration

protected final VoidConfiguration voidConfiguration

meshBuildMode

protected final MeshBuildMode meshBuildMode

numerOfNodes

protected final AtomicInteger numerOfNodes

messageQueue

protected final TransferQueue<VoidMessage> messageQueue

splitter

protected MessageSplitter splitter

historyHolder

protected MessagesHistoryHolder<String> historyHolder

handshakeFlag

protected org.nd4j.common.primitives.AtomicBoolean handshakeFlag

executorService

protected final ThreadPoolExecutor executorService

Constructor Detail

BaseTransport

protected BaseTransport()

BaseTransport

protected BaseTransport(@NonNull
                        @NonNull String rootId)

BaseTransport

protected BaseTransport(@NonNull
                        @NonNull String rootId,
                        @NonNull
                        @NonNull VoidConfiguration voidConfiguration)

BaseTransport

protected BaseTransport(@NonNull
                        @NonNull String ownId,
                        @NonNull
                        @NonNull String rootId,
                        @NonNull
                        @NonNull VoidConfiguration voidConfiguration)

Method Detail

outgoingConsumer

public io.reactivex.functions.Consumer<VoidMessage> outgoingConsumer()

Description copied from interface: Transport

This method returns consumer that accepts messages for delivery

Specified by:

outgoingConsumer in interface Transport

Returns:

incomingPublisher

public org.reactivestreams.Publisher<INDArrayMessage> incomingPublisher()

Description copied from interface: Transport

This method returns flow of messages for parameter server

Specified by:

incomingPublisher in interface Transport

Returns:

getUpstreamId

public String getUpstreamId()

Description copied from interface: Transport

This methos returns Id of the upstream node

Specified by:

getUpstreamId in interface Transport

Returns:

launch

public void launch()

Description copied from interface: Transport

This method starts this Transport instance

Specified by:

launch in interface Transport

launchAsMaster

public void launchAsMaster()

Description copied from interface: Transport

This method will start this Transport instance

Specified by:

launchAsMaster in interface Transport

shutdown

public void shutdown()

Description copied from interface: Transport

This method shuts down this Transport instance

Specified by:

shutdown in interface Transport

propagateArrayMessage

protected void propagateArrayMessage(INDArrayMessage message,
                                     PropagationMode mode)
                              throws IOException

Throws:

IOException

propagateMessage

public void propagateMessage(@NonNull
                             @NonNull VoidMessage voidMessage,
                             PropagationMode mode)
                      throws IOException

Description copied from interface: Transport

This method will send message to the network, using tree structure

Specified by:

propagateMessage in interface Transport

Throws:

IOException

propagateBroadcastableMessage

protected void propagateBroadcastableMessage(@NonNull
                                             @NonNull BroadcastableMessage voidMessage,
                                             PropagationMode mode)

isLoopedNode

protected boolean isLoopedNode(@NonNull
                               MeshOrganizer.Node node,
                               @NonNull
                               @NonNull String originatorId,
                               @NonNull
                               @NonNull String relayId)

internalProcessMessage

protected void internalProcessMessage(VoidMessage message)

propagateMessageDirect

public void propagateMessageDirect(@NonNull
                                   @NonNull BroadcastableMessage message)

processMessage

public void processMessage(VoidMessage message)

Description copied from interface: Transport

This method will be invoked for all incoming messages PLEASE NOTE: this method is mostly suited for tests

Specified by:

processMessage in interface Transport

getRandomDownstreamFrom

public String getRandomDownstreamFrom(@NonNull
                                      @NonNull String id,
                                      String exclude)

Description copied from interface: Transport

This method returns random

Specified by:

getRandomDownstreamFrom in interface Transport

Returns:

sendMessageBlocking

public <T extends ResponseMessage> T sendMessageBlocking(@NonNull
                                                         @NonNull RequestMessage message,
                                                         @NonNull
                                                         @NonNull String id)
                                                  throws InterruptedException

Description copied from interface: Transport

This method will send message to specified node, and will return its response

Specified by:

sendMessageBlocking in interface Transport

Returns:

Throws:

InterruptedException

sendMessageBlocking

public <T extends ResponseMessage> T sendMessageBlocking(@NonNull
                                                         @NonNull RequestMessage message,
                                                         @NonNull
                                                         @NonNull String id,
                                                         long timeWait,
                                                         @NonNull
                                                         @NonNull TimeUnit timeUnit)
                                                  throws InterruptedException

Description copied from interface: Transport

This method will send message to specified node, and will return its response

Specified by:

sendMessageBlocking in interface Transport

Returns:

Throws:

InterruptedException

setRestartCallback

public void setRestartCallback(RestartCallback callback)

Description copied from interface: Transport

This method allows to set callback instance, which will be called upon restart event

Specified by:

setRestartCallback in interface Transport

addRequestConsumer

public <T extends RequestMessage> void addRequestConsumer(@NonNull
                                                          @NonNull Class<T> cls,
                                                          io.reactivex.functions.Consumer<T> consumer)

Description copied from interface: Transport

This methd allows to set callback instance for various

Specified by:

addRequestConsumer in interface Transport

onMeshUpdate

public void onMeshUpdate(MeshOrganizer mesh)

Description copied from interface: Transport

This method will be called if mesh update was received PLEASE NOTE: This method will be called ONLY if new mesh differs from current one

Specified by:

onMeshUpdate in interface Transport

onRemap

public void onRemap(String id)

Description copied from interface: Transport

This method will be called upon remap request

Specified by:

onRemap in interface Transport

getRootId

public String getRootId()

Description copied from interface: Transport

This method returns ID of the root node

Specified by:

getRootId in interface Transport

Returns:

totalNumberOfNodes

public int totalNumberOfNodes()

Description copied from interface: Transport

This method returns total number of nodes known to this Transport

Specified by:

totalNumberOfNodes in interface Transport

Returns:

isConnected

public boolean isConnected()

Description copied from interface: Transport

This method checks if all connections required for work are established

Specified by:

isConnected in interface Transport

Returns:

true

isIntroduced

public boolean isIntroduced()

Description copied from interface: Transport

This method checks if this node was properly introduced to driver

Specified by:

isIntroduced in interface Transport

Returns:

ensureConnection

public void ensureConnection(String id)

Description copied from interface: Transport

This method checks connection to the given node ID, and if it's not connected - establishes connection

Specified by:

ensureConnection in interface Transport