org.apache.hadoop.hbase.replication

Class TableBasedReplicationQueuesImpl

java.lang.Object

org.apache.hadoop.hbase.replication.TableBasedReplicationQueuesImpl

All Implemented Interfaces:

ReplicationQueues

@InterfaceAudience.Private
public class TableBasedReplicationQueuesImpl
extends Object
implements ReplicationQueues

This class provides an implementation of the ReplicationQueues interface using an HBase table "Replication Table". It utilizes the ReplicationTableBase to access the Replication Table.

Field Summary

Fields

Modifier and Type	Field and Description
protected Abortable	abortable
static byte[]	CF_QUEUE
static byte[]	COL_QUEUE_OWNER
static byte[]	COL_QUEUE_OWNER_HISTORY
protected org.apache.hadoop.conf.Configuration	conf
static String	QUEUE_HISTORY_DELIMITER
static TableName	REPLICATION_TABLE_NAME Name of the HBase Table used for tracking replication
static String	ROW_KEY_DELIMITER

Constructor Summary

Constructors

Constructor and Description
TableBasedReplicationQueuesImpl(org.apache.hadoop.conf.Configuration conf, Abortable abort, ZooKeeperWatcher zkw)
TableBasedReplicationQueuesImpl(ReplicationQueuesArguments args)

Method Summary

Modifier and Type	Method and Description
void	addHFileRefs(String peerId, List<Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path>> pairs) Add new hfile references to the queue.
void	addLog(String queueId, String filename) Add a new WAL file to the given queue.
void	addPeerToHFileRefs(String peerId) Add a peer to hfile reference queue if peer does not exist.
protected String	buildClaimedQueueHistory(String originalHistory, String oldServer) Creates a "|" delimited record of the queue's past region server owners.
protected String	buildQueueRowKey(String serverName, String queueId) Build the row key for the given queueId.
Pair<String,SortedSet<String>>	claimQueue(String regionserver, String queueId) Take ownership for the queue identified by queueId and belongs to a dead region server.
List<String>	getAllQueues() Get a list of all queues for this region server.
protected List<String>	getAllQueues(String serverName)
boolean	getInitializationStatus() Get whether the Replication Table has been successfully initialized yet
List<String>	getListOfReplicators() Get a list of all region servers that have outstanding replication queues.
long	getLogPosition(String queueId, String filename) Get the current position for a specific WAL in a given queue.
protected List<String>	getLogsInQueue(byte[] rowKey)
List<String>	getLogsInQueue(String queueId) Get a list of all WALs in the given queue.
protected List<String>	getLogsInQueue(String serverName, String queueId)
protected Table	getOrBlockOnReplicationTable() Attempts to acquire the Replication Table.
protected ResultScanner	getQueuesBelongingToServer(String server) Get the queue id's and meta data (Owner and History) for the queues belonging to the named server
protected String	getRawQueueIdFromRowKey(String rowKey) Parse the original queueId from a row key
List<String>	getUnClaimedQueueIds(String regionserver) Get queueIds from a dead region server, whose queues has not been claimed by other region servers.
void	init(String serverName) Initialize the region server replication queue interface.
boolean	isThisOurRegionServer(String regionserver) Checks if the provided znode is the same as this region server's
protected byte[]	queueIdToRowKey(String serverName, String queueId) Returns a queue's row key given either its raw or reclaimed queueId
protected List<String>	readWALsFromResult(Result queue) Read all of the WAL's from a queue into a list
void	removeAllQueues() Remove all replication queues for this region server.
void	removeHFileRefs(String peerId, List<String> files) Remove hfile references from the queue.
void	removeLog(String queueId, String filename) Remove an WAL file from the given queue.
void	removePeerFromHFileRefs(String peerId) Remove a peer from hfile reference queue.
void	removeQueue(String queueId) Remove a replication queue.
void	removeReplicatorIfQueueIsEmpty(String regionserver) Remove the znode of region server if the queue is empty.
void	setLogPosition(String queueId, String filename, long position) Set the current position for a specific WAL in a given queue.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

REPLICATION_TABLE_NAME

public static final TableName REPLICATION_TABLE_NAME

Name of the HBase Table used for tracking replication

CF_QUEUE

public static final byte[] CF_QUEUE

COL_QUEUE_OWNER

public static final byte[] COL_QUEUE_OWNER

COL_QUEUE_OWNER_HISTORY

public static final byte[] COL_QUEUE_OWNER_HISTORY

ROW_KEY_DELIMITER

public static final String ROW_KEY_DELIMITER

See Also:

Constant Field Values

QUEUE_HISTORY_DELIMITER

public static final String QUEUE_HISTORY_DELIMITER

See Also:

Constant Field Values

conf

protected final org.apache.hadoop.conf.Configuration conf

abortable

protected final Abortable abortable

Constructor Detail

TableBasedReplicationQueuesImpl

public TableBasedReplicationQueuesImpl(ReplicationQueuesArguments args)
                                throws IOException

Throws:

IOException

TableBasedReplicationQueuesImpl

public TableBasedReplicationQueuesImpl(org.apache.hadoop.conf.Configuration conf,
                                       Abortable abort,
                                       ZooKeeperWatcher zkw)
                                throws IOException

Throws:

IOException

Method Detail

init

public void init(String serverName)
          throws ReplicationException

Description copied from interface: ReplicationQueues

Initialize the region server replication queue interface.

Specified by:

init in interface ReplicationQueues

Parameters:

serverName - The server name of the region server that owns the replication queues this interface manages.

Throws:

ReplicationException

getListOfReplicators

public List<String> getListOfReplicators()

Get a list of all region servers that have outstanding replication queues. These servers could be alive, dead or from a previous run of the cluster.

Specified by:

getListOfReplicators in interface ReplicationQueues

Returns:

a list of server names

removeQueue

public void removeQueue(String queueId)

Description copied from interface: ReplicationQueues

Remove a replication queue.

Specified by:

removeQueue in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue.

addLog

public void addLog(String queueId,
                   String filename)
            throws ReplicationException

Description copied from interface: ReplicationQueues

Add a new WAL file to the given queue. If the queue does not exist it is created.

Specified by:

addLog in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue.

filename - name of the WAL

Throws:

ReplicationException

removeLog

public void removeLog(String queueId,
                      String filename)

Description copied from interface: ReplicationQueues

Remove an WAL file from the given queue.

Specified by:

removeLog in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue.

filename - name of the WAL

setLogPosition

public void setLogPosition(String queueId,
                           String filename,
                           long position)

Description copied from interface: ReplicationQueues

Set the current position for a specific WAL in a given queue.

Specified by:

setLogPosition in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue

filename - name of the WAL

position - the current position in the file

getLogPosition

public long getLogPosition(String queueId,
                           String filename)
                    throws ReplicationException

Description copied from interface: ReplicationQueues

Get the current position for a specific WAL in a given queue.

Specified by:

getLogPosition in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue

filename - name of the WAL

Returns:

the current position in the file

Throws:

ReplicationException

removeAllQueues

public void removeAllQueues()

Description copied from interface: ReplicationQueues

Remove all replication queues for this region server.

Specified by:

removeAllQueues in interface ReplicationQueues

getLogsInQueue

public List<String> getLogsInQueue(String queueId)

Description copied from interface: ReplicationQueues

Get a list of all WALs in the given queue.

Specified by:

getLogsInQueue in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue

Returns:

a list of WALs, null if no such queue exists for this server

getAllQueues

public List<String> getAllQueues()

Description copied from interface: ReplicationQueues

Get a list of all queues for this region server.

Specified by:

getAllQueues in interface ReplicationQueues

Returns:

a list of queueIds, an empty list if this region server is dead and has no outstanding queues

getUnClaimedQueueIds

public List<String> getUnClaimedQueueIds(String regionserver)

Description copied from interface: ReplicationQueues

Get queueIds from a dead region server, whose queues has not been claimed by other region servers.

Specified by:

getUnClaimedQueueIds in interface ReplicationQueues

Returns:

empty if the queue exists but no children, null if the queue does not exist.

removeReplicatorIfQueueIsEmpty

public void removeReplicatorIfQueueIsEmpty(String regionserver)

Description copied from interface: ReplicationQueues

Remove the znode of region server if the queue is empty.

Specified by:

removeReplicatorIfQueueIsEmpty in interface ReplicationQueues

claimQueue

public Pair<String,SortedSet<String>> claimQueue(String regionserver,
                                                 String queueId)

Description copied from interface: ReplicationQueues

Take ownership for the queue identified by queueId and belongs to a dead region server.

Specified by:

claimQueue in interface ReplicationQueues

Parameters:

regionserver - the id of the dead region server

queueId - the id of the queue

Returns:

the new PeerId and A SortedSet of WALs in its queue, and null if no unclaimed queue.

isThisOurRegionServer

public boolean isThisOurRegionServer(String regionserver)

Description copied from interface: ReplicationQueues

Checks if the provided znode is the same as this region server's

Specified by:

isThisOurRegionServer in interface ReplicationQueues

Parameters:

regionserver - the id of the region server

Returns:

if this is this rs's znode

addPeerToHFileRefs

public void addPeerToHFileRefs(String peerId)
                        throws ReplicationException

Description copied from interface: ReplicationQueues

Add a peer to hfile reference queue if peer does not exist.

Specified by:

addPeerToHFileRefs in interface ReplicationQueues

Parameters:

peerId - peer cluster id to be added

Throws:

ReplicationException - if fails to add a peer id to hfile reference queue

removePeerFromHFileRefs

public void removePeerFromHFileRefs(String peerId)

Description copied from interface: ReplicationQueues

Remove a peer from hfile reference queue.

Specified by:

removePeerFromHFileRefs in interface ReplicationQueues

Parameters:

peerId - peer cluster id to be removed

addHFileRefs

public void addHFileRefs(String peerId,
                         List<Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path>> pairs)
                  throws ReplicationException

Description copied from interface: ReplicationQueues

Add new hfile references to the queue.

Specified by:

addHFileRefs in interface ReplicationQueues

Parameters:

peerId - peer cluster id to which the hfiles need to be replicated

pairs - list of pairs of { HFile location in staging dir, HFile path in region dir which will be added in the queue }

Throws:

ReplicationException - if fails to add a hfile reference

removeHFileRefs

public void removeHFileRefs(String peerId,
                            List<String> files)

Description copied from interface: ReplicationQueues

Remove hfile references from the queue.

Specified by:

removeHFileRefs in interface ReplicationQueues

Parameters:

peerId - peer cluster id from which this hfile references needs to be removed

files - list of hfile references to be removed

getInitializationStatus

public boolean getInitializationStatus()

Get whether the Replication Table has been successfully initialized yet

Returns:

whether the Replication Table is initialized

buildQueueRowKey

protected String buildQueueRowKey(String serverName,
                                  String queueId)

Build the row key for the given queueId. This will uniquely identify it from all other queues in the cluster.

Parameters:

serverName - The owner of the queue

queueId - String identifier of the queue

Returns:

String representation of the queue's row key

getRawQueueIdFromRowKey

protected String getRawQueueIdFromRowKey(String rowKey)

Parse the original queueId from a row key

Parameters:

rowKey - String representation of a queue's row key

Returns:

the original queueId

queueIdToRowKey

protected byte[] queueIdToRowKey(String serverName,
                                 String queueId)

Returns a queue's row key given either its raw or reclaimed queueId

Parameters:

queueId - queueId of the queue

Returns:

byte representation of the queue's row key

buildClaimedQueueHistory

protected String buildClaimedQueueHistory(String originalHistory,
                                          String oldServer)

Creates a "|" delimited record of the queue's past region server owners.

Parameters:

originalHistory - the queue's original owner history

oldServer - the name of the server that used to own the queue

Returns:

the queue's new owner history

getAllQueues

protected List<String> getAllQueues(String serverName)

getLogsInQueue

protected List<String> getLogsInQueue(String serverName,
                                      String queueId)

getLogsInQueue

protected List<String> getLogsInQueue(byte[] rowKey)

readWALsFromResult

protected List<String> readWALsFromResult(Result queue)

Read all of the WAL's from a queue into a list

Parameters:

queue - HBase query result containing the queue

Returns:

a list of all the WAL filenames

getQueuesBelongingToServer

protected ResultScanner getQueuesBelongingToServer(String server)
                                            throws IOException

Get the queue id's and meta data (Owner and History) for the queues belonging to the named server

Parameters:

server - name of the server

Returns:

a ResultScanner over the QueueIds belonging to the server

Throws:

IOException

getOrBlockOnReplicationTable

protected Table getOrBlockOnReplicationTable()
                                      throws IOException

Attempts to acquire the Replication Table. This operation will block until it is assigned by the CreateReplicationWorker thread. It is up to the caller of this method to close the returned Table

Returns:

the Replication Table when it is created

Throws:

IOException