com.staros.worker

Class WorkerManager

java.lang.Object

com.staros.worker.WorkerManager

public class WorkerManager
extends java.lang.Object

Worker Manager handles all operations that are related to worker, such as add worker and remove worker.

Assumptions: - WorkerGroupId is globally unique Running Mode: WorkerManager can run in two different modes based on whether supporting zero worker group id. - Strict Mode: Zero group id is not supported. Client must create a worker group before adding a worker - Compatible Mode: Zero group id is allowed, and is created ahead by starManager, client can add worker to group-0 without creating it first. This mode is mostly used when doing process level integration with starRocks FE and support manual management of BE with `ALTER SYSTEM ADD/DROP BACKEND XXX`.

Constructor Summary

Constructors

Constructor and Description
WorkerManager(JournalSystem journalSystem, IdGenerator idGenerator)

Method Summary

Modifier and Type	Method and Description
long	addWorker(java.lang.String serviceId, long groupId, java.lang.String ipPort) Add a worker for worker group in service
java.util.List<java.lang.Long>	addWorkers(java.lang.String serviceId, long groupId, java.util.List<java.lang.String> ipPorts)
void	bootstrapService(java.lang.String serviceId)
long	createWorkerGroup(java.lang.String serviceId, java.lang.String owner, WorkerGroupSpec spec, java.util.Map<java.lang.String,java.lang.String> labels, java.util.Map<java.lang.String,java.lang.String> properties)
static WorkerManager	createWorkerManagerForTest(JournalSystem journalSystem)
void	deleteWorkerGroup(java.lang.String serviceId, long groupId)
void	doWorkerHeartbeat(long workerId)
void	dump(java.io.DataOutputStream out)
void	dumpMeta(java.io.OutputStream out) dump workerManager's meta
java.util.List<java.lang.Long>	getAllWorkerIds() Get a snapshot of all worker ids
WorkerGroup	getDefaultWorkerGroup(java.lang.String serviceId) Returns a default worker group of the service.
Worker	getWorker(long workerId)
int	getWorkerCount()
WorkerGroup	getWorkerGroup(java.lang.String serviceId, long groupId)
WorkerGroup	getWorkerGroupNoException(java.lang.String serviceId, long groupId)
WorkerInfo	getWorkerInfo(long workerId) Get worker info by id
WorkerInfo	getWorkerInfo(java.lang.String ipPort) Get worker info by ip port
java.util.List<WorkerGroupDetailInfo>	listWorkerGroups(java.lang.String serviceId, java.util.Map<java.lang.String,java.lang.String> filterLabels, boolean includeWorkersInfo)
java.util.List<WorkerGroupDetailInfo>	listWorkerGroupsById(java.lang.String serviceId, java.util.List<java.lang.Long> groupIds, boolean includeWorkersInfo)
void	loadMeta(java.io.InputStream in)
boolean	processWorkerHeartbeat(java.lang.String serviceId, long workerId, long startTime, long numOfShards, java.util.Map<java.lang.String,java.lang.String> workerProperties, long lastSeenTime) process worker heartbeat, update worker's info
void	removeWorker(java.lang.String serviceId, long groupId, long workerId) Remove a worker for worker group in service
void	replayAddWorker(java.lang.String serviceId, Worker worker)
void	replayCreateWorkerGroup(java.lang.String serviceId, WorkerGroup group)
void	replayDeleteWorkerGroup(java.lang.String serviceId, long groupId)
void	replayRemoveWorker(java.lang.String serviceId, long groupId, long workerId)
void	replayUpdateWorker(java.lang.String serviceId, java.util.List<Worker> workers)
void	replayUpdateWorkerGroup(java.lang.String serviceId, UpdateWorkerGroupInfo info)
void	start()
void	stop()
void	updateWorkerGroup(java.lang.String serviceId, long groupId, WorkerGroupSpec spec, java.util.Map<java.lang.String,java.lang.String> labels, java.util.Map<java.lang.String,java.lang.String> props)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WorkerManager

public WorkerManager(JournalSystem journalSystem,
                     IdGenerator idGenerator)

Method Detail

bootstrapService

public void bootstrapService(java.lang.String serviceId)

createWorkerGroup

public long createWorkerGroup(java.lang.String serviceId,
                              java.lang.String owner,
                              WorkerGroupSpec spec,
                              java.util.Map<java.lang.String,java.lang.String> labels,
                              java.util.Map<java.lang.String,java.lang.String> properties)
                       throws StarException

Throws:

StarException

addWorker

public long addWorker(java.lang.String serviceId,
                      long groupId,
                      java.lang.String ipPort)
               throws StarException

Add a worker for worker group in service

Returns:

worker id

Throws:

StarException - - ALREADY_EXIST if the ipPort already used by other worker - NOT_EXIST if the given groupId or serviceId not exist yet

addWorkers

public java.util.List<java.lang.Long> addWorkers(java.lang.String serviceId,
                                                 long groupId,
                                                 java.util.List<java.lang.String> ipPorts)
                                          throws StarException

Throws:

StarException

doWorkerHeartbeat

public void doWorkerHeartbeat(long workerId)

removeWorker

public void removeWorker(java.lang.String serviceId,
                         long groupId,
                         long workerId)
                  throws StarException

Remove a worker for worker group in service

Throws:

StarException - if group id or worker id not exist

getWorkerInfo

public WorkerInfo getWorkerInfo(long workerId)
                         throws StarException

Get worker info by id

Throws:

StarException

getWorkerInfo

public WorkerInfo getWorkerInfo(java.lang.String ipPort)
                         throws StarException

Get worker info by ip port

Throws:

StarException

getDefaultWorkerGroup

public WorkerGroup getDefaultWorkerGroup(java.lang.String serviceId)
                                  throws StarException

Returns a default worker group of the service.

Parameters:

serviceId - service identity

Returns:

default WorkerGroup. the workerGroup with the smallest id returns or exception if no worker group available.

Throws:

StarException - NOT_EXIST if the service doesn't have any workerGroup yet.

getWorkerGroup

public WorkerGroup getWorkerGroup(java.lang.String serviceId,
                                  long groupId)
                           throws StarException

Throws:

StarException

getWorkerGroupNoException

public WorkerGroup getWorkerGroupNoException(java.lang.String serviceId,
                                             long groupId)

getWorker

public Worker getWorker(long workerId)
                 throws StarException

Throws:

StarException

getAllWorkerIds

public java.util.List<java.lang.Long> getAllWorkerIds()

Get a snapshot of all worker ids

Returns:

list of worker ids

processWorkerHeartbeat

public boolean processWorkerHeartbeat(java.lang.String serviceId,
                                      long workerId,
                                      long startTime,
                                      long numOfShards,
                                      java.util.Map<java.lang.String,java.lang.String> workerProperties,
                                      long lastSeenTime)
                               throws StarException

process worker heartbeat, update worker's info

Throws:

StarException

replayCreateWorkerGroup

public void replayCreateWorkerGroup(java.lang.String serviceId,
                                    WorkerGroup group)

replayAddWorker

public void replayAddWorker(java.lang.String serviceId,
                            Worker worker)

replayRemoveWorker

public void replayRemoveWorker(java.lang.String serviceId,
                               long groupId,
                               long workerId)

replayUpdateWorker

public void replayUpdateWorker(java.lang.String serviceId,
                               java.util.List<Worker> workers)

getWorkerCount

public int getWorkerCount()

listWorkerGroupsById

public java.util.List<WorkerGroupDetailInfo> listWorkerGroupsById(java.lang.String serviceId,
                                                                  java.util.List<java.lang.Long> groupIds,
                                                                  boolean includeWorkersInfo)

listWorkerGroups

public java.util.List<WorkerGroupDetailInfo> listWorkerGroups(java.lang.String serviceId,
                                                              java.util.Map<java.lang.String,java.lang.String> filterLabels,
                                                              boolean includeWorkersInfo)

updateWorkerGroup

public void updateWorkerGroup(java.lang.String serviceId,
                              long groupId,
                              WorkerGroupSpec spec,
                              java.util.Map<java.lang.String,java.lang.String> labels,
                              java.util.Map<java.lang.String,java.lang.String> props)
                       throws StarException

Throws:

StarException

replayUpdateWorkerGroup

public void replayUpdateWorkerGroup(java.lang.String serviceId,
                                    UpdateWorkerGroupInfo info)

deleteWorkerGroup

public void deleteWorkerGroup(java.lang.String serviceId,
                              long groupId)
                       throws StarException

Throws:

StarException

replayDeleteWorkerGroup

public void replayDeleteWorkerGroup(java.lang.String serviceId,
                                    long groupId)

dumpMeta

public void dumpMeta(java.io.OutputStream out)
              throws java.io.IOException

dump workerManager's meta

Parameters:

out - Output stream

Throws:

java.io.IOException - I/O exception

  +--------------------+
  | WORKER_MGR_HEADER  | (batch_size, number of workers, number of service worker groups)
  +--------------------+
  | WORKER_MANAGER_SWG |
  +------------------- +
  |      SWG-1         |
  |      SWG-2         |
  |       ...          |
  |      SWG-N         |
  +--------------------+
  | WORKER_MGR_WORKER  |
  +--------------------+
  |  WORKERS_BATCH_1   |
  |  WORKERS_BATCH_2   |
  |       ...          |
  |  WORKERS_BATCH_N   |
  +--------------------+
  | WORKER_MGR_FOOTER  | (checksum)
  +--------------------+
 

loadMeta

public void loadMeta(java.io.InputStream in)
              throws java.io.IOException

Throws:

java.io.IOException

start

public void start()

stop

public void stop()

dump

public void dump(java.io.DataOutputStream out)
          throws java.io.IOException

Throws:

java.io.IOException

createWorkerManagerForTest

public static WorkerManager createWorkerManagerForTest(JournalSystem journalSystem)