org.apache.hadoop.hbase.master.balancer

Class StochasticLoadBalancer

java.lang.Object

org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, ConfigurationObserver, LoadBalancer, Stoppable

Direct Known Subclasses:

FavoredStochasticBalancer

@InterfaceAudience.LimitedPrivate(value="Configuration")
public class StochasticLoadBalancer
extends BaseLoadBalancer

This is a best effort load balancer. Given a Cost function F(C) => x It will randomly try and mutate the cluster to Cprime. If F(Cprime) < F(C) then the new cluster state becomes the plan. It includes costs functions to compute the cost of:

• Region Load
• Table Load
• Data Locality
• Memstore Sizes
• Storefile Sizes

Every cost function returns a number between 0 and 1 inclusive; where 0 is the lowest cost best solution, and 1 is the highest possible cost and the worst solution. The computed costs are scaled by their respective multipliers:

• hbase.master.balancer.stochastic.regionLoadCost
• hbase.master.balancer.stochastic.moveCost
• hbase.master.balancer.stochastic.tableLoadCost
• hbase.master.balancer.stochastic.localityCost
• hbase.master.balancer.stochastic.memstoreSizeCost
• hbase.master.balancer.stochastic.storefileSizeCost

You can also add custom Cost function by setting the the following configuration value:

• hbase.master.balancer.stochastic.additionalCostFunctions

All custom Cost Functions needs to extends StochasticLoadBalancer.CostFunction

In addition to the above configurations, the balancer can be tuned by the following configuration values:

• hbase.master.balancer.stochastic.maxMoveRegions which controls what the max number of regions that can be moved in a single invocation of this balancer.
• hbase.master.balancer.stochastic.stepsPerRegion is the coefficient by which the number of regions is multiplied to try and get the number of times the balancer will mutate all servers.
• hbase.master.balancer.stochastic.maxSteps which controls the maximum number of times that the balancer will try and mutate all the servers. The balancer will use the minimum of this value and the above computation.

This balancer is best used with hbase.master.loadbalance.bytable set to false so that the balancer gets the full picture of all loads on the cluster.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StochasticLoadBalancer.CostFunction Base class of StochasticLoadBalancer's Cost Functions.
static class	StochasticLoadBalancer.GeneratorType

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

BaseLoadBalancer.Cluster

Field Summary

Fields

Modifier and Type	Field and Description
protected List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator>	candidateGenerators
static double	COST_EPSILON
protected static String	COST_FUNCTIONS_COST_FUNCTIONS_KEY
protected static String	KEEP_REGION_LOADS
protected static String	MAX_RUNNING_TIME_KEY
protected static String	MAX_STEPS_KEY
protected static String	MIN_COST_NEED_BALANCE_KEY
protected static Random	RANDOM
protected static String	RUN_MAX_STEPS_KEY
protected static String	STEPS_PER_REGION_KEY

Fields inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

BALANCER_DECISION_BUFFER_ENABLED, BALANCER_REJECTION_BUFFER_ENABLED, clusterStatus, config, DEFAULT_BALANCER_DECISION_BUFFER_ENABLED, DEFAULT_BALANCER_REJECTION_BUFFER_ENABLED, isByTable, masterServerName, metricsBalancer, MIN_SERVER_BALANCE, onlySystemTablesOnMaster, overallSlop, rackManager, regionFinder, services, slop, useRegionFinder

Fields inherited from interface org.apache.hadoop.hbase.master.LoadBalancer

BOGUS_SERVER_NAME, SYSTEM_TABLES_ON_MASTER, TABLES_ON_MASTER

Constructor Summary

Constructors

Constructor and Description
StochasticLoadBalancer() The constructor that pass a MetricsStochasticBalancer to BaseLoadBalancer to replace its default MetricsBalancer

Method Summary

Modifier and Type	Method and Description
protected boolean	areSomeRegionReplicasColocated(BaseLoadBalancer.Cluster c) Subclasses should implement this to return true if the cluster has nodes that hosts multiple replicas for the same region, or, if there are multiple racks and the same rack hosts replicas of the same region
List<RegionPlan>	balanceTable(TableName tableName, Map<ServerName,List<RegionInfo>> loadOfOneTable) Given the cluster state this will try and approach an optimal balance.
static String	composeAttributeName(String tableName, String costFunctionName) A helper function to compose the attribute name from tablename and costfunction name
List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator>	getCandidateGenerators() Exposed for Testing!
protected org.apache.hadoop.hbase.master.balancer.CandidateGenerator	getRandomGenerator() Select the candidate generator to use based on the cost of cost functions.
protected boolean	needsBalance(TableName tableName, BaseLoadBalancer.Cluster cluster)
void	onConfigurationChange(org.apache.hadoop.conf.Configuration conf)
protected void	setCandidateGenerators(List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator> customCandidateGenerators)
void	setClusterMetrics(ClusterMetrics st) Set the current cluster status.
void	setConf(org.apache.hadoop.conf.Configuration conf)
protected void	setSlop(org.apache.hadoop.conf.Configuration conf)
void	updateMetricsSize(int size) Update the number of metrics that are reported to JMX

Methods inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

assignMasterSystemRegions, balanceCluster, balanceMasterRegions, createCluster, getConf, getRegionAssignmentsByServer, idleRegionServerExist, initialize, isStopped, postMasterStartupInitialize, randomAssignment, regionOffline, regionOnline, retainAssignment, roundRobinAssignment, setMasterServices, setRackManager, shouldBeOnMaster, stop, updateBalancerStatus

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.master.LoadBalancer

isMasterCanHostUserRegions, isSystemTablesOnlyOnMaster, isTablesOnMaster

Field Detail

STEPS_PER_REGION_KEY

protected static final String STEPS_PER_REGION_KEY

See Also:

Constant Field Values

MAX_STEPS_KEY

protected static final String MAX_STEPS_KEY

See Also:

Constant Field Values

RUN_MAX_STEPS_KEY

protected static final String RUN_MAX_STEPS_KEY

See Also:

Constant Field Values

MAX_RUNNING_TIME_KEY

protected static final String MAX_RUNNING_TIME_KEY

See Also:

Constant Field Values

KEEP_REGION_LOADS

protected static final String KEEP_REGION_LOADS

See Also:

Constant Field Values

MIN_COST_NEED_BALANCE_KEY

protected static final String MIN_COST_NEED_BALANCE_KEY

See Also:

Constant Field Values

COST_FUNCTIONS_COST_FUNCTIONS_KEY

protected static final String COST_FUNCTIONS_COST_FUNCTIONS_KEY

See Also:

Constant Field Values

RANDOM

protected static final Random RANDOM

COST_EPSILON

public static final double COST_EPSILON

See Also:

Constant Field Values

candidateGenerators

protected List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator> candidateGenerators

Constructor Detail

StochasticLoadBalancer

public StochasticLoadBalancer()

The constructor that pass a MetricsStochasticBalancer to BaseLoadBalancer to replace its default MetricsBalancer

Method Detail

onConfigurationChange

public void onConfigurationChange(org.apache.hadoop.conf.Configuration conf)

Specified by:

onConfigurationChange in interface ConfigurationObserver

Specified by:

onConfigurationChange in interface LoadBalancer

Overrides:

onConfigurationChange in class BaseLoadBalancer

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

Overrides:

setConf in class BaseLoadBalancer

setCandidateGenerators

protected void setCandidateGenerators(List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator> customCandidateGenerators)

getCandidateGenerators

public List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator> getCandidateGenerators()

Exposed for Testing!

setSlop

protected void setSlop(org.apache.hadoop.conf.Configuration conf)

Overrides:

setSlop in class BaseLoadBalancer

setClusterMetrics

public void setClusterMetrics(ClusterMetrics st)

Description copied from interface: LoadBalancer

Set the current cluster status. This allows a LoadBalancer to map host name to a server

Specified by:

setClusterMetrics in interface LoadBalancer

Overrides:

setClusterMetrics in class BaseLoadBalancer

updateMetricsSize

public void updateMetricsSize(int size)

Update the number of metrics that are reported to JMX

areSomeRegionReplicasColocated

protected boolean areSomeRegionReplicasColocated(BaseLoadBalancer.Cluster c)

Description copied from class: BaseLoadBalancer

Subclasses should implement this to return true if the cluster has nodes that hosts multiple replicas for the same region, or, if there are multiple racks and the same rack hosts replicas of the same region

Overrides:

areSomeRegionReplicasColocated in class BaseLoadBalancer

Parameters:

c - Cluster information

Returns:

whether region replicas are currently co-located

needsBalance

protected boolean needsBalance(TableName tableName,
                               BaseLoadBalancer.Cluster cluster)

Overrides:

needsBalance in class BaseLoadBalancer

getRandomGenerator

protected org.apache.hadoop.hbase.master.balancer.CandidateGenerator getRandomGenerator()

Select the candidate generator to use based on the cost of cost functions. The chance of selecting a candidate generator is propotional to the share of cost of all cost functions among all cost functions that benefit from it.

balanceTable

public List<RegionPlan> balanceTable(TableName tableName,
                                     Map<ServerName,List<RegionInfo>> loadOfOneTable)

Given the cluster state this will try and approach an optimal balance. This should always approach the optimal state given enough steps.

Specified by:

balanceTable in interface LoadBalancer

Specified by:

balanceTable in class BaseLoadBalancer

Parameters:

tableName - the table to be balanced

loadOfOneTable - region load of servers for the specific one table

Returns:

List of plans

composeAttributeName

public static String composeAttributeName(String tableName,
                                          String costFunctionName)

A helper function to compose the attribute name from tablename and costfunction name