ec.exchange

Class IslandExchange

java.lang.Object

ec.Exchanger

ec.exchange.IslandExchange

All Implemented Interfaces:

Setup, Singleton, Serializable

public class IslandExchange
extends Exchanger

IslandExchange is an Exchanger which implements a simple but quite functional asynchronous island model for doing massive parallel distribution of evolution across beowulf clusters. One of its really nice features is that because everything is in Java, your cluster can have mixed platforms in it (MacOS, Unix, Windoze, whatever you like). You can also have multiple processes running on the same machine, as long as they're given different client ports. IslandExchange operates over TCP/IP with Java sockets, and is compatible with checkpointing.

IslandExchange uses an arbitrary graph topology for migrating individuals from island (EC process) to island over the network. There are a few restrictions for simplicity, however:

• Every island must have the same kind of subpopulations and species.
• Every subpopulation will send the same number of migrants as any other subpopulation.
• Migrants from a subpopulation will go to the same subpopulation.

Every island is a client. Additionally one island is designated a server. Note that, just like in the Hair Club for Men, the server is also a client. The purpose of the server is to synchronize the clients so that they all get set up properly and hook up to each other, then to send them small signal messages (like informing them that another client has discovered the ideal individual), and help them gracefully shut down. Other than these few signals which are routed through the server to the clients, all other information -- namely the migrants themselves -- are sent directly from client to client in a peer-to-peer fashion.

The topology of the network is stored solely in the server's parameter database. When the clients fire up, they first set up "Mailboxes" (where immigrants from other clients will appear), then they go to the server and ask it who they should connect to to send migrants. The server tells them, and then they then hook up. When a client has finished hooking up, it reports this to the server. After everyone has hooked up, the server tells the clients to begin evolution, and they're off and running.

Islands send immigrants to other islands by copying good individuals (selected with a SelectionMethod) and sending the good individuals to the mailboxes of receiving clients. Once an individual has been received, it is considered to be unevaluated by the receiving island, even though it had been previously evaluated by the sending island.

The IslandExchange model is typically asynchronous because migrants may appear in your mailbox at any time; islands do not wait for each other to complete the next generation. This is a more efficient usage of network bandwidth. When an island completes its breeding, it looks inside its mailbox for new migrants. It then replaces some of its newly-bred individuals (chosen entirely at random) with the migrants (we could have increased the population size so we didn't waste that breeding time, but we were lazy). It then flushes the mailbox, which patiently sits waiting for more individuals.

Clients may also be given different start times and modulos for migrating. For example, client A might be told that he begins sending immigrants only after generation 6, and then sends immigrants on every 4 generations beyond that. The purpose for the start times and modulos is so that not every client sends immigrants at the same time; this also makes better use of network bandwidth.

When a client goes down, the other clients deal with it gracefully; they simply stop trying to send to it. But if the server goes down, the clients do not continue operation; they will shut themselves down. This means that in general you can shut down an entire island model network just by killing the server process. However, if the server quits because it runs out of generations, it will wait for the clients to all quit before it finally stops.

IslandExchange works correctly with checkpointing. If you restart from a checkpoint, the IslandExchange will start up the clients and servers again and reconnect. Processes can start from different checkpoints, of course. However, realize that if you restart from a checkpoint, some migrants may have been lost in transit from island to island. That's the nature of networking without heavy-duty transaction management! This means that we cannot guarantee that restarting from checkpoint will yield the same results as the first run yielded.

Islands are not described in the topology parameters by their IP addresses; instead, they are described by "ids", strings which uniquely identify each island. For example, "gilligans-island" might be an id. :-) This allows you to move your topology to different IP addresses without having to change all your parameter files! You can even move your topology to totally different machines, and restart from previous checkpoints, and everything should still work correctly.

There are times, especially to experiment with dynamics, that you need a synchronous island model. If you specify synchronicity, the server's stated modulo and offset override any modulii or offsets specified by clients. Everyone will use the server's modulo and offset. This means that everyone will trade individuals on the same generation. Additionally, clients will wait until everyone else has traded, before they are permitted to continue evolving. This has the effect of locking all the clients together generation-wise; no clients can run faster than any other clients.

One last item: normally in this model, the server is also a client. But if for some reason you need the server to be a process all by itself, without creating a client as well, you can do that. You spawn such a server differently than the main execution of ECJ. To spawn a server on a given server params file (let's say it's server.params) but NOT spawn a client, you do:

 java ec.exchange.IslandExchange -file server.params
 

...this sets up a special process which just spawns a server, and doesn't do all the setup of an evolutionary run. Of course as usual, for each of the clients, you'll run java ec.Evolve ... instead.

Parameters

Note: some of these parameters are only necessary for creating clients. Others are necessary for creating the server.

base.chatty boolean, default = true	Should we be verbose or silent about our exchanges?
base.select classname, inherits and != ec.SelectionMethod	client: The selection method used for picking migrants to emigrate to other islands
base.select-to-die classname, inherits and != ec.SelectionMethod, default is ec.select.RandomSelection	client: The selection method used for picking individuals to be replaced by incoming migrants. IMPORTANT Note. This selection method must not pick an individual based on fitness. The selection method will be called just after breeding but before evaluation; many individuals will not have had a fitness assigned at that point. You might want to design a SelectionMethod other than RandomSelection, however, to do things like not picking elites to die.
base.server-addr String	client: The IP address of the server
base.server-port int >= 1	client: The port number of the server
base.client-port int >= 1	client: The port number of the client (where it will receive migrants) -- this should be different from the server port.
base.id String	client: The "name" the client is giving itself. Each client should have a unique name. For example, "gilligans-island".
base.compressed bool = true (default) or false	client: Whether the communication with other islands should be compressed or not. Compressing uses more CPU, but it may also significantly reduce communication.
base.i-am-server bool = true or false (default)	client: Is this client also the server? If so, it'll read the server parameters and set up a server as well.
base.sync bool = true or false (default)	server: Are we doing a synchronous island model? If so, the server's modulo and offset override any client's stated modulo and offset.
base.num-islands int >= 1	server: The number of islands in the topology.
base.island.n.id String	server: The ID of island #n in the topology.
base.island.n.num-mig int >= 1	server: The number of islands that island #n sends immigrants to.
base.island.n.mig.m int >= 1	server: The ID of island #m that island #n sends immigrants to.
base.island.n.size int >= 1	server: The number of immigrants (per subpopulation) that island #n sends to other islands. If not set, uses the default parameter below.
base.size int >= 1	server: Default parameter: number of immigrants (per subpopulation) that a given island sends to other islands.
base.island.n.start int >= 0	server: The generation when island #n begins sending immigrants. If not set, uses the default parameter below.
base.start bool = true or false (default)	server: Default parameter: the generation when an island begins sending immigrants.
base.island.n.mod int >= 1	server: The number of generations that island #n waits between sending immigrants. If not set, uses the default parameter below.
base.mod bool = true or false (default)	server: Default parameter: The number of generations an island waits between sending immigrants.
base.island.n.mailbox-capacity int >= 1	server: The maximum size (per subpopulation) of the mailbox for island #n. If not set, uses the default parameter below.
base.mailbox-capacity int >= 1	server: Default parameter: the maximum size (per subpopulation) of the mailbox for a given island.

Parameter bases

base.select

selection method for the client's migrants

Version:

2.0

Author:

Liviu Panait and Sean Luke

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
(package private) boolean	alreadyReadGoodBye
Parameter	base My parameter base -- I need to keep this in order to help the server reinitialize contacts
(package private) boolean	chatty Our chattiness
int	clientPort The port of the client mailbox
boolean	compressedCommunication whether the communication is compressed or not
static String	FOUND Found signal
static int	FOUND_TIMEOUT How long we sleep between checking for FOUND messages
(package private) DataInputStream	fromServer
boolean	iAmServer whether the server should be running on the current island or not
SelectionMethod	immigrantsSelectionMethod the selection method for immigrants
SelectionMethod	indsToDieSelectionMethod the selection method for individuals to be replaced by immigrants
(package private) static boolean	just_server
(package private) IslandExchangeMailbox	mailbox
(package private) Thread	mailboxThread
(package private) String	message
int	modulo how often to send individuals
(package private) int	number_of_destination_islands
int	offset after how many generations to start sending individuals
static String	OKAY Okay signal
String[]	outgoingIds
(package private) Socket[]	outSockets
(package private) DataOutputStream[]	outWriters
String	ownId the id of the current island
static String	P_CHATTY Whether or not we're chatty
static String	P_CLIENT_PORT The client port
static String	P_COMPRESSED_COMMUNICATION Whether the communication is compressed or not
static String	P_IS_SERVER Whether the server is also on this island
static String	P_OWN_ID The id of the island
static String	P_SELECT_METHOD The selection method for sending individuals to other islands
static String	P_SELECT_TO_DIE_METHOD The selection method for deciding individuals to be replaced by immigrants
static String	P_SERVER_ADDRESS The server address
static String	P_SERVER_PORT The server port
(package private) boolean[]	running
private static long	serialVersionUID
String	serverAddress The address of the server
int	serverPort The port of the server
(package private) Socket	serverSocket
Thread	serverThread The thread of the server (is different than null only for the island with the server)
int	size how many individuals to send each time
static int	SLEEP_TIME How long we sleep in between attempts to connect or look for signals
static String	SYNC Synchronize signal
boolean	synchronous synchronous or asynchronous communication
(package private) DataOutputStream	toServer

Constructor Summary

Constructors

Constructor and Description
IslandExchange()

Method Summary

Methods

Modifier and Type	Method and Description
void	closeContacts(EvolutionState state, int result) Closes contacts with other processes, if that's what you're doing.
int	evolve()
void	finish(int result)
void	fireUpServer(EvolutionState state, Parameter serverBase) Fires up the server.
void	initializeContacts(EvolutionState state) Initializes contacts with other processes, if that's what you're doing.
static void	main(String[] args)
Population	postBreedingExchangePopulation(EvolutionState state) Performs exchanges after the population has been bred but before it has been evaluated, once every generation (or pseudogeneration).
Population	preBreedingExchangePopulation(EvolutionState state) Performs exchanges after the population has been evaluated but before it has been bred, once every generation (or pseudogeneration).
private void	readObject(ObjectInputStream in) Custom serialization
void	reinitializeContacts(EvolutionState state) Initializes contacts with other processes, if that's what you're doing.
String	runComplete(EvolutionState state) Called after preBreedingExchangePopulation(...) to evaluate whether or not the exchanger wishes the run to shut down (with ec.EvolutionState.R_FAILURE).
void	setup(EvolutionState state, Parameter _base) Sets up the object by reading it from the parameters stored in state, built off of the parameter base base.
void	startFresh()
void	startFromCheckpoint()
private void	writeObject(ObjectOutputStream out) Custom serialization

Methods inherited from class ec.Exchanger

process

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

serialVersionUID

private static final long serialVersionUID

See Also:

Constant Field Values

P_SERVER_ADDRESS

public static final String P_SERVER_ADDRESS

The server address

See Also:

Constant Field Values

P_SERVER_PORT

public static final String P_SERVER_PORT

The server port

See Also:

Constant Field Values

P_CLIENT_PORT

public static final String P_CLIENT_PORT

The client port

See Also:

Constant Field Values

P_IS_SERVER

public static final String P_IS_SERVER

Whether the server is also on this island

See Also:

Constant Field Values

P_OWN_ID

public static final String P_OWN_ID

The id of the island

See Also:

Constant Field Values

P_COMPRESSED_COMMUNICATION

public static final String P_COMPRESSED_COMMUNICATION

Whether the communication is compressed or not

See Also:

Constant Field Values

P_SELECT_METHOD

public static final String P_SELECT_METHOD

The selection method for sending individuals to other islands

See Also:

Constant Field Values

P_SELECT_TO_DIE_METHOD

public static final String P_SELECT_TO_DIE_METHOD

The selection method for deciding individuals to be replaced by immigrants

See Also:

Constant Field Values

SLEEP_TIME

public static final int SLEEP_TIME

How long we sleep in between attempts to connect or look for signals

See Also:

Constant Field Values

FOUND_TIMEOUT

public static final int FOUND_TIMEOUT

How long we sleep between checking for FOUND messages

See Also:

Constant Field Values

P_CHATTY

public static final String P_CHATTY

Whether or not we're chatty

See Also:

Constant Field Values

OKAY

public static final String OKAY

Okay signal

See Also:

Constant Field Values

SYNC

public static final String SYNC

Synchronize signal

See Also:

Constant Field Values

FOUND

public static final String FOUND

Found signal

See Also:

Constant Field Values

chatty

boolean chatty

Our chattiness

serverThread

public Thread serverThread

The thread of the server (is different than null only for the island with the server)

base

public Parameter base

My parameter base -- I need to keep this in order to help the server reinitialize contacts

serverAddress

public String serverAddress

The address of the server

serverPort

public int serverPort

The port of the server

clientPort

public int clientPort

The port of the client mailbox

iAmServer

public boolean iAmServer

whether the server should be running on the current island or not

ownId

public String ownId

the id of the current island

compressedCommunication

public boolean compressedCommunication

whether the communication is compressed or not

immigrantsSelectionMethod

public SelectionMethod immigrantsSelectionMethod

the selection method for immigrants

indsToDieSelectionMethod

public SelectionMethod indsToDieSelectionMethod

the selection method for individuals to be replaced by immigrants

mailbox

IslandExchangeMailbox mailbox

mailboxThread

Thread mailboxThread

number_of_destination_islands

int number_of_destination_islands

synchronous

public boolean synchronous

synchronous or asynchronous communication

modulo

public int modulo

how often to send individuals

offset

public int offset

after how many generations to start sending individuals

size

public int size

how many individuals to send each time

outSockets

Socket[] outSockets

outWriters

DataOutputStream[] outWriters

outgoingIds

public String[] outgoingIds

running

boolean[] running

serverSocket

Socket serverSocket

toServer

DataOutputStream toServer

fromServer

DataInputStream fromServer

just_server

static boolean just_server

alreadyReadGoodBye

boolean alreadyReadGoodBye

message

String message

Constructor Detail

IslandExchange

public IslandExchange()

Method Detail

main

public static void main(String[] args)
                 throws InterruptedException

Throws:

InterruptedException

setup

public void setup(EvolutionState state,
         Parameter _base)

Description copied from interface: Setup

Sets up the object by reading it from the parameters stored in state, built off of the parameter base base. If an ancestor implements this method, be sure to call super.setup(state,base); before you do anything else.

writeObject

private void writeObject(ObjectOutputStream out)
                  throws IOException

Custom serialization

Throws:

IOException

readObject

private void readObject(ObjectInputStream in)
                 throws IOException,
                        ClassNotFoundException

Custom serialization

Throws:

IOException

ClassNotFoundException

fireUpServer

public void fireUpServer(EvolutionState state,
                Parameter serverBase)

Fires up the server.

initializeContacts

public void initializeContacts(EvolutionState state)

Initializes contacts with other processes, if that's what you're doing. Called at the beginning of an evolutionary run, before a population is set up.

Overrides:

initializeContacts in class Exchanger

reinitializeContacts

public void reinitializeContacts(EvolutionState state)

Initializes contacts with other processes, if that's what you're doing. Called after restarting from a checkpoint.

Overrides:

reinitializeContacts in class Exchanger

preBreedingExchangePopulation

public Population preBreedingExchangePopulation(EvolutionState state)

Description copied from class: Exchanger

Performs exchanges after the population has been evaluated but before it has been bred, once every generation (or pseudogeneration).

Specified by:

preBreedingExchangePopulation in class Exchanger

postBreedingExchangePopulation

public Population postBreedingExchangePopulation(EvolutionState state)

Description copied from class: Exchanger

Performs exchanges after the population has been bred but before it has been evaluated, once every generation (or pseudogeneration).

Specified by:

postBreedingExchangePopulation in class Exchanger

runComplete

public String runComplete(EvolutionState state)

Called after preBreedingExchangePopulation(...) to evaluate whether or not the exchanger wishes the run to shut down (with ec.EvolutionState.R_FAILURE). This would happen for two reasons. First, another process might have found an ideal individual and the global run is now over. Second, some network or operating system error may have occurred and the system needs to be shut down gracefully. This function does not return a String as soon as it wants to exit (another island found the perfect individual, or couldn't connect to the server). Instead, it sets a flag, called message, to remember next time to exit. This is due to a need for a graceful shutdown, where checkpoints are working properly and save all needed information.

Specified by:

runComplete in class Exchanger

closeContacts

public void closeContacts(EvolutionState state,
                 int result)

Closes contacts with other processes, if that's what you're doing. Called at the end of an evolutionary run. result is either ec.EvolutionState.R_SUCCESS or ec.EvolutionState.R_FAILURE, indicating whether or not an ideal individual was found.

Overrides:

closeContacts in class Exchanger

finish

public void finish(int result)

startFromCheckpoint

public void startFromCheckpoint()

startFresh

public void startFresh()

evolve

public int evolve()
           throws InternalError

Throws:

InternalError