org.apache.hadoop.hbase.replication

Interface ReplicationQueueStorage

All Known Implementing Classes:

TableReplicationQueueStorage

@InterfaceAudience.Private
public interface ReplicationQueueStorage

Perform read/write to the replication queue storage.

Method Summary

Modifier and Type	Method and Description
void	addHFileRefs(String peerId, List<Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path>> pairs) Add new hfile references to the queue.
void	batchUpdateHFileRefs(String peerId, List<String> hfileRefs) Add the given hfile refs to the given peer.
void	batchUpdateLastSequenceIds(List<ZKReplicationQueueStorageForMigration.ZkLastPushedSeqId> lastPushedSeqIds) Update last pushed sequence id for the given regions and peers.
void	batchUpdateQueues(ServerName serverName, List<ReplicationQueueData> datas) Update the replication queue datas for a given region server.
Map<String,ReplicationGroupOffset>	claimQueue(ReplicationQueueId queueId, ServerName targetServerName) Change ownership for the queue identified by queueId and belongs to a dead region server.
Set<String>	getAllHFileRefs() Load all hfile references in all replication queues.
List<String>	getAllPeersFromHFileRefsQueue() Get list of all peers from hfile reference queue.
long	getLastSequenceId(String encodedRegionName, String peerId) Read the max sequence id of the specific region for a given peer.
Map<String,ReplicationGroupOffset>	getOffsets(ReplicationQueueId queueId) Get the current offset of all the WAL groups for a queue
List<String>	getReplicableHFiles(String peerId) Get a list of all hfile references in the given peer.
boolean	hasData() Whether the replication queue table exists.
List<ReplicationQueueId>	listAllQueueIds(ServerName serverName) Get a list of all queues for the specific region server.
List<ReplicationQueueId>	listAllQueueIds(String peerId) Get a list of all queues for the specific peer.
List<ReplicationQueueId>	listAllQueueIds(String peerId, ServerName serverName) Get a list of all queues for the specific region server and the specific peer
List<ReplicationQueueData>	listAllQueues() Get a list of all queues and the offsets.
List<ServerName>	listAllReplicators() Get a list of all region servers that have outstanding replication queues.
void	removeAllQueues(String peerId) Remove all the replication queues for the given peer.
void	removeHFileRefs(String peerId, List<String> files) Remove hfile references from the queue.
void	removeLastSequenceIds(String peerId) Remove all the max sequence id record for the given peer.
void	removeLastSequenceIds(String peerId, List<String> encodedRegionNames) Remove the max sequence id record for the given peer and regions.
void	removeLastSequenceIdsAndHFileRefsBefore(long ts) Remove all the last sequence ids and hfile references data which are written before the given timestamp.
void	removePeerFromHFileRefs(String peerId) Remove a peer from hfile reference queue.
void	removeQueue(ReplicationQueueId queueId) Remove a replication queue
void	setLastSequenceIds(String peerId, Map<String,Long> lastSeqIds) Set the max sequence id of a bunch of regions for a given peer.
void	setOffset(ReplicationQueueId queueId, String walGroup, ReplicationGroupOffset offset, Map<String,Long> lastSeqIds) Set the current offset for a specific WAL group in a given queue.

Method Detail

setOffset

void setOffset(ReplicationQueueId queueId,
               String walGroup,
               ReplicationGroupOffset offset,
               Map<String,Long> lastSeqIds)
        throws ReplicationException

Set the current offset for a specific WAL group in a given queue.

Parameters:

queueId - the id of the queue

walGroup - the group of the WAL, can be empty if multi wal is not enabled

offset - the current offset of replication progress

lastSeqIds - map with {encodedRegionName, sequenceId} pairs for serial replication.

Throws:

ReplicationException

getOffsets

Map<String,ReplicationGroupOffset> getOffsets(ReplicationQueueId queueId)
                                       throws ReplicationException

Get the current offset of all the WAL groups for a queue

Parameters:

queueId - the id of the queue

Returns:

a map of all offsets of the WAL groups. The key the is WAL group and the value is the position.

Throws:

ReplicationException

listAllQueueIds

List<ReplicationQueueId> listAllQueueIds(String peerId)
                                  throws ReplicationException

Get a list of all queues for the specific peer.

Parameters:

peerId - the id of the peer

Returns:

a list of queueIds

Throws:

ReplicationException

listAllQueueIds

List<ReplicationQueueId> listAllQueueIds(ServerName serverName)
                                  throws ReplicationException

Get a list of all queues for the specific region server.

Parameters:

serverName - the server name of the region server that owns the set of queues

Returns:

a list of queueIds

Throws:

ReplicationException

listAllQueueIds

List<ReplicationQueueId> listAllQueueIds(String peerId,
                                         ServerName serverName)
                                  throws ReplicationException

Get a list of all queues for the specific region server and the specific peer

Parameters:

peerId - the id of the peer

serverName - the server name of the region server that owns the set of queues

Returns:

a list of queueIds

Throws:

ReplicationException

listAllQueues

List<ReplicationQueueData> listAllQueues()
                                  throws ReplicationException

Get a list of all queues and the offsets.

Throws:

ReplicationException

listAllReplicators

List<ServerName> listAllReplicators()
                             throws ReplicationException

Get a list of all region servers that have outstanding replication queues. These servers could be alive, dead or from a previous run of the cluster.

Returns:

a list of server names

Throws:

ReplicationException

claimQueue

Map<String,ReplicationGroupOffset> claimQueue(ReplicationQueueId queueId,
                                              ServerName targetServerName)
                                       throws ReplicationException

Change ownership for the queue identified by queueId and belongs to a dead region server.

Parameters:

queueId - the id of the queue

targetServerName - the name of the target region server

Returns:

the new PeerId and A SortedSet of WALs in its queue

Throws:

ReplicationException

removeQueue

void removeQueue(ReplicationQueueId queueId)
          throws ReplicationException

Remove a replication queue

Parameters:

queueId - the id of the queue to remove

Throws:

ReplicationException

removeAllQueues

void removeAllQueues(String peerId)
              throws ReplicationException

Remove all the replication queues for the given peer. Usually used when removing a peer.

Parameters:

peerId - the id of the peer

Throws:

ReplicationException

getLastSequenceId

long getLastSequenceId(String encodedRegionName,
                       String peerId)
                throws ReplicationException

Read the max sequence id of the specific region for a given peer. For serial replication, we need the max sequenced id to decide whether we can push the next entries.

Parameters:

encodedRegionName - the encoded region name

peerId - peer id

Returns:

the max sequence id of the specific region for a given peer.

Throws:

ReplicationException

setLastSequenceIds

void setLastSequenceIds(String peerId,
                        Map<String,Long> lastSeqIds)
                 throws ReplicationException

Set the max sequence id of a bunch of regions for a given peer. Will be called when setting up a serial replication peer.

Parameters:

peerId - peer id

lastSeqIds - map with {encodedRegionName, sequenceId} pairs for serial replication.

Throws:

ReplicationException

removeLastSequenceIds

void removeLastSequenceIds(String peerId)
                    throws ReplicationException

Remove all the max sequence id record for the given peer.

Parameters:

peerId - peer id

Throws:

ReplicationException

removeLastSequenceIds

void removeLastSequenceIds(String peerId,
                           List<String> encodedRegionNames)
                    throws ReplicationException

Remove the max sequence id record for the given peer and regions.

Parameters:

peerId - peer id

encodedRegionNames - the encoded region names

Throws:

ReplicationException

removePeerFromHFileRefs

void removePeerFromHFileRefs(String peerId)
                      throws ReplicationException

Remove a peer from hfile reference queue.

Parameters:

peerId - peer cluster id to be removed

Throws:

ReplicationException

addHFileRefs

void addHFileRefs(String peerId,
                  List<Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path>> pairs)
           throws ReplicationException

Add new hfile references to the queue.

Parameters:

peerId - peer cluster id to which the hfiles need to be replicated

pairs - list of pairs of { HFile location in staging dir, HFile path in region dir which will be added in the queue }

Throws:

ReplicationException - if fails to add a hfile reference

removeHFileRefs

void removeHFileRefs(String peerId,
                     List<String> files)
              throws ReplicationException

Remove hfile references from the queue.

Parameters:

peerId - peer cluster id from which this hfile references needs to be removed

files - list of hfile references to be removed

Throws:

ReplicationException

getAllPeersFromHFileRefsQueue

List<String> getAllPeersFromHFileRefsQueue()
                                    throws ReplicationException

Get list of all peers from hfile reference queue.

Returns:

a list of peer ids

Throws:

ReplicationException

getReplicableHFiles

List<String> getReplicableHFiles(String peerId)
                          throws ReplicationException

Get a list of all hfile references in the given peer.

Parameters:

peerId - a String that identifies the peer

Returns:

a list of hfile references

Throws:

ReplicationException

getAllHFileRefs

Set<String> getAllHFileRefs()
                     throws ReplicationException

Load all hfile references in all replication queues. This method guarantees to return a snapshot which contains all hfile references at the start of this call. However, some newly created hfile references during the call may not be included.

Throws:

ReplicationException

hasData

boolean hasData()
         throws ReplicationException

Whether the replication queue table exists.

Returns:

Whether the replication queue table exists

Throws:

ReplicationException

batchUpdateQueues

void batchUpdateQueues(ServerName serverName,
                       List<ReplicationQueueData> datas)
                throws ReplicationException

Update the replication queue datas for a given region server.

Throws:

ReplicationException

batchUpdateLastSequenceIds

void batchUpdateLastSequenceIds(List<ZKReplicationQueueStorageForMigration.ZkLastPushedSeqId> lastPushedSeqIds)
                         throws ReplicationException

Update last pushed sequence id for the given regions and peers.

Throws:

ReplicationException

batchUpdateHFileRefs

void batchUpdateHFileRefs(String peerId,
                          List<String> hfileRefs)
                   throws ReplicationException

Add the given hfile refs to the given peer.

Throws:

ReplicationException

removeLastSequenceIdsAndHFileRefsBefore

void removeLastSequenceIdsAndHFileRefsBefore(long ts)
                                      throws ReplicationException

Remove all the last sequence ids and hfile references data which are written before the given timestamp.

The data of these two types are not used by replication directly.

For last sequence ids, we will check it in serial replication, to make sure that we will replicate all edits in order, so if there are stale data, the worst case is that we will stop replicating as we think we still need to finish previous ranges first, although actually we have already replicated them out.

For hfile references, it is just used by hfile cleaner to not remove these hfiles before we replicate them out, so if there are stale data, the worst case is that we can not remove these hfiles, although actually they have already been replicated out.

So it is OK for us to just bring up the cluster first, and then use this method to delete the stale data, i.e, the data which are written before a specific timestamp.

Throws:

ReplicationException