org.apache.hadoop.hbase.tool

类 LoadIncrementalHFiles

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.hbase.tool.LoadIncrementalHFiles

所有已实现的接口:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

直接已知子类:

BulkLoadHFilesTool, LoadIncrementalHFiles

已过时。

since 2.2.0, will be removed in 3.0.0. Use BulkLoadHFiles instead. Please rewrite your code if you rely on methods other than the run(Map, TableName) and run(String, TableName), as all the methods other than them will be removed with no replacement.

@Deprecated
 @InterfaceAudience.Public
public class LoadIncrementalHFiles
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

Tool to load the output of HFileOutputFormat into an existing table.

Notice that, by default, this class should be kept till 4.0.0, but as this is a bad practice that we expose an implementation class instead of an interface, we want to fix it ASAP. That's why we will remove this class completely in 3.0.0. Please change your code to use BulkLoadHFiles.

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	LoadIncrementalHFiles.LoadQueueItem 已过时。 since 2.2.0 and will be removed in 3.0.0. Use BulkLoadHFiles instead.

字段概要

字段

限定符和类型	字段和说明
static String	ALWAYS_COPY_FILES 已过时。
static String	CREATE_TABLE_CONF_KEY 已过时。
static String	IGNORE_UNMATCHED_CF_CONF_KEY 已过时。
static String	MAX_FILES_PER_REGION_PER_FAMILY 已过时。
static String	NAME 已过时。 since 2.2.0, will be removed in 3.0.0, with no replacement. End user should not depend on this value.

构造器概要

构造器

构造器和说明
LoadIncrementalHFiles(org.apache.hadoop.conf.Configuration conf) 已过时。

方法概要

限定符和类型	方法和说明
protected ClientServiceCallable<byte[]>	buildClientServiceCallable(Connection conn, TableName tableName, byte[] first, Collection<LoadIncrementalHFiles.LoadQueueItem> lqis, boolean copyFile) 已过时。
protected void	bulkLoadPhase(Table table, Connection conn, ExecutorService pool, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups, boolean copyFile, Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> item2RegionMap) 已过时。 This takes the LQI's grouped by likely regions and attempts to bulk load them.
void	disableReplication() 已过时。 Disables replication for these bulkloaded files.
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	doBulkLoad(Map<byte[],List<org.apache.hadoop.fs.Path>> map, Admin admin, Table table, RegionLocator regionLocator, boolean silence, boolean copyFile) 已过时。 Perform a bulk load of the given directory into the given pre-existing table.
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	doBulkLoad(org.apache.hadoop.fs.Path hfofDir, Admin admin, Table table, RegionLocator regionLocator) 已过时。 Perform a bulk load of the given directory into the given pre-existing table.
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	doBulkLoad(org.apache.hadoop.fs.Path hfofDir, Admin admin, Table table, RegionLocator regionLocator, boolean silence, boolean copyFile) 已过时。 Perform a bulk load of the given directory into the given pre-existing table.
protected Pair<List<LoadIncrementalHFiles.LoadQueueItem>,String>	groupOrSplit(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups, LoadIncrementalHFiles.LoadQueueItem item, Table table, Pair<byte[][],byte[][]> startEndKeys) 已过时。 Attempt to assign the given load queue item into its target region group.
static byte[][]	inferBoundaries(SortedMap<byte[],Integer> bdryMap) 已过时。 Infers region boundaries for a new table.
void	loadHFileQueue(Table table, Connection conn, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, Pair<byte[][],byte[][]> startEndKeys) 已过时。 Used by the replication sink to load the hfiles from the source cluster.
void	loadHFileQueue(Table table, Connection conn, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, Pair<byte[][],byte[][]> startEndKeys, boolean copyFile) 已过时。 Used by the replication sink to load the hfiles from the source cluster.
static void	main(String[] args) 已过时。
void	prepareHFileQueue(Map<byte[],List<org.apache.hadoop.fs.Path>> map, Table table, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, boolean silence) 已过时。 Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.
void	prepareHFileQueue(org.apache.hadoop.fs.Path hfilesDir, Table table, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, boolean validateHFile) 已过时。 Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.
void	prepareHFileQueue(org.apache.hadoop.fs.Path hfilesDir, Table table, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, boolean validateHFile, boolean silence) 已过时。 Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	run(Map<byte[],List<org.apache.hadoop.fs.Path>> family2Files, TableName tableName) 已过时。 Perform bulk load on the given table.
protected Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	run(org.apache.hadoop.fs.Path hfofDir, TableName tableName) 已过时。
int	run(String[] args) 已过时。
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	run(String hfofDir, TableName tableName) 已过时。 Perform bulk load on the given table.
void	setBulkToken(String stagingDir) 已过时。 Called from replication sink, where it manages bulkToken(staging directory) by itself.
void	setClusterIds(List<String> clusterIds) 已过时。
protected List<LoadIncrementalHFiles.LoadQueueItem>	tryAtomicRegionLoad(ClientServiceCallable<byte[]> serviceCallable, TableName tableName, byte[] first, Collection<LoadIncrementalHFiles.LoadQueueItem> lqis) 已过时。 Attempts to do an atomic load of many hfiles into a region.

从类继承的方法 org.apache.hadoop.conf.Configured

getConf, setConf

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.hadoop.conf.Configurable

getConf, setConf

字段详细资料

NAME

@Deprecated
public static final String NAME

已过时。 since 2.2.0, will be removed in 3.0.0, with no replacement. End user should not depend on this value.

另请参阅:

常量字段值

MAX_FILES_PER_REGION_PER_FAMILY

public static final String MAX_FILES_PER_REGION_PER_FAMILY

已过时。

另请参阅:

常量字段值

CREATE_TABLE_CONF_KEY

public static final String CREATE_TABLE_CONF_KEY

已过时。

另请参阅:

常量字段值

IGNORE_UNMATCHED_CF_CONF_KEY

public static final String IGNORE_UNMATCHED_CF_CONF_KEY

已过时。

另请参阅:

常量字段值

ALWAYS_COPY_FILES

public static final String ALWAYS_COPY_FILES

已过时。

另请参阅:

常量字段值

构造器详细资料

LoadIncrementalHFiles

public LoadIncrementalHFiles(org.apache.hadoop.conf.Configuration conf)

已过时。

方法详细资料

prepareHFileQueue

public void prepareHFileQueue(org.apache.hadoop.fs.Path hfilesDir,
                              Table table,
                              Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                              boolean validateHFile)
                       throws IOException

已过时。

Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

参数:

hfilesDir - directory containing list of hfiles to be loaded into the table

table - table to which hfiles should be loaded

queue - queue which needs to be loaded into the table

validateHFile - if true hfiles will be validated for its format

抛出:

IOException - If any I/O or network error occurred

prepareHFileQueue

public void prepareHFileQueue(org.apache.hadoop.fs.Path hfilesDir,
                              Table table,
                              Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                              boolean validateHFile,
                              boolean silence)
                       throws IOException

已过时。

Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

参数:

hfilesDir - directory containing list of hfiles to be loaded into the table

table - table to which hfiles should be loaded

queue - queue which needs to be loaded into the table

validateHFile - if true hfiles will be validated for its format

silence - true to ignore unmatched column families

抛出:

IOException - If any I/O or network error occurred

prepareHFileQueue

public void prepareHFileQueue(Map<byte[],List<org.apache.hadoop.fs.Path>> map,
                              Table table,
                              Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                              boolean silence)
                       throws IOException

已过时。

Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

参数:

map - map of family to List of hfiles

table - table to which hfiles should be loaded

queue - queue which needs to be loaded into the table

silence - true to ignore unmatched column families

抛出:

IOException - If any I/O or network error occurred

doBulkLoad

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> doBulkLoad(org.apache.hadoop.fs.Path hfofDir,
                                                                      Admin admin,
                                                                      Table table,
                                                                      RegionLocator regionLocator)
                                                               throws TableNotFoundException,
                                                                      IOException

已过时。

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

参数:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

admin - the Admin

table - the table to load into

regionLocator - region locator

抛出:

TableNotFoundException - if table does not yet exist

IOException

doBulkLoad

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> doBulkLoad(Map<byte[],List<org.apache.hadoop.fs.Path>> map,
                                                                      Admin admin,
                                                                      Table table,
                                                                      RegionLocator regionLocator,
                                                                      boolean silence,
                                                                      boolean copyFile)
                                                               throws TableNotFoundException,
                                                                      IOException

已过时。

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

参数:

map - map of family to List of hfiles

admin - the Admin

table - the table to load into

regionLocator - region locator

silence - true to ignore unmatched column families

copyFile - always copy hfiles if true

抛出:

TableNotFoundException - if table does not yet exist

IOException

doBulkLoad

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> doBulkLoad(org.apache.hadoop.fs.Path hfofDir,
                                                                      Admin admin,
                                                                      Table table,
                                                                      RegionLocator regionLocator,
                                                                      boolean silence,
                                                                      boolean copyFile)
                                                               throws TableNotFoundException,
                                                                      IOException

已过时。

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

参数:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

admin - the Admin

table - the table to load into

regionLocator - region locator

silence - true to ignore unmatched column families

copyFile - always copy hfiles if true

抛出:

TableNotFoundException - if table does not yet exist

IOException

loadHFileQueue

public void loadHFileQueue(Table table,
                           Connection conn,
                           Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                           Pair<byte[][],byte[][]> startEndKeys)
                    throws IOException

已过时。

Used by the replication sink to load the hfiles from the source cluster. It does the following,

1. LoadIncrementalHFiles#groupOrSplitPhase(Table, ExecutorService, Deque, Pair)}
2. LoadIncrementalHFiles#bulkLoadPhase(Table, Connection, ExecutorService, Deque, Multimap)

参数:

table - Table to which these hfiles should be loaded to

conn - Connection to use

queue - LoadIncrementalHFiles.LoadQueueItem has hfiles yet to be loaded

startEndKeys - starting and ending row keys of the region

抛出:

IOException

loadHFileQueue

public void loadHFileQueue(Table table,
                           Connection conn,
                           Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                           Pair<byte[][],byte[][]> startEndKeys,
                           boolean copyFile)
                    throws IOException

已过时。

Used by the replication sink to load the hfiles from the source cluster. It does the following,

1. LoadIncrementalHFiles#groupOrSplitPhase(Table, ExecutorService, Deque, Pair)}
2. LoadIncrementalHFiles#bulkLoadPhase(Table, Connection, ExecutorService, Deque, Multimap)

参数:

table - Table to which these hfiles should be loaded to

conn - Connection to use

queue - LoadIncrementalHFiles.LoadQueueItem has hfiles yet to be loaded

startEndKeys - starting and ending row keys of the region

抛出:

IOException

bulkLoadPhase

protected void bulkLoadPhase(Table table,
                             Connection conn,
                             ExecutorService pool,
                             Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                             org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups,
                             boolean copyFile,
                             Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> item2RegionMap)
                      throws IOException

已过时。

This takes the LQI's grouped by likely regions and attempts to bulk load them. Any failures are re-queued for another pass with the groupOrSplitPhase.

protected for testing.

抛出:

IOException

buildClientServiceCallable

protected ClientServiceCallable<byte[]> buildClientServiceCallable(Connection conn,
                                                                   TableName tableName,
                                                                   byte[] first,
                                                                   Collection<LoadIncrementalHFiles.LoadQueueItem> lqis,
                                                                   boolean copyFile)

已过时。

groupOrSplit

protected Pair<List<LoadIncrementalHFiles.LoadQueueItem>,String> groupOrSplit(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups,
                                                                              LoadIncrementalHFiles.LoadQueueItem item,
                                                                              Table table,
                                                                              Pair<byte[][],byte[][]> startEndKeys)
                                                                       throws IOException

已过时。

Attempt to assign the given load queue item into its target region group. If the hfile boundary no longer fits into a region, physically splits the hfile such that the new bottom half will fit and returns the list of LQI's corresponding to the resultant hfiles.

protected for testing

抛出:

IOException - if an IO failure is encountered

tryAtomicRegionLoad

protected List<LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(ClientServiceCallable<byte[]> serviceCallable,
                                                                        TableName tableName,
                                                                        byte[] first,
                                                                        Collection<LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                 throws IOException

已过时。

Attempts to do an atomic load of many hfiles into a region. If it fails, it returns a list of hfiles that need to be retried. If it is successful it will return an empty list.

NOTE: To maintain row atomicity guarantees, region server callable should succeed atomically and fails atomically.

Protected for testing.

返回:

empty list if success, list of items to retry on recoverable failure

抛出:

IOException

run

protected final Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> run(org.apache.hadoop.fs.Path hfofDir,
                                                                        TableName tableName)
                                                                 throws IOException

已过时。

抛出:

IOException

run

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> run(String hfofDir,
                                                               TableName tableName)
                                                        throws IOException

已过时。

Perform bulk load on the given table.

参数:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

tableName - the table to load into

抛出:

IOException

run

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> run(Map<byte[],List<org.apache.hadoop.fs.Path>> family2Files,
                                                               TableName tableName)
                                                        throws IOException

已过时。

Perform bulk load on the given table.

参数:

family2Files - map of family to List of hfiles

tableName - the table to load into

抛出:

IOException

run

public int run(String[] args)
        throws Exception

已过时。

指定者:

run 在接口中 org.apache.hadoop.util.Tool

抛出:

Exception

main

public static void main(String[] args)
                 throws Exception

已过时。

抛出:

Exception

setBulkToken

public void setBulkToken(String stagingDir)

已过时。

Called from replication sink, where it manages bulkToken(staging directory) by itself. This is used only when SecureBulkLoadEndpoint is configured in hbase.coprocessor.region.classes property. This directory is used as a temporary directory where all files are initially copied/moved from user given directory, set all the required file permissions and then from their it is finally loaded into a table. This should be set only when, one would like to manage the staging directory by itself. Otherwise this tool will handle this by itself.

参数:

stagingDir - staging directory path

setClusterIds

public void setClusterIds(List<String> clusterIds)

已过时。

disableReplication

public void disableReplication()

已过时。

Disables replication for these bulkloaded files.

inferBoundaries

public static byte[][] inferBoundaries(SortedMap<byte[],Integer> bdryMap)

已过时。

Infers region boundaries for a new table.

Parameter:
bdryMap is a map between keys to an integer belonging to {+1, -1}

• If a key is a start key of a file, then it maps to +1
• If a key is an end key of a file, then it maps to -1

Algo:

1. Poll on the keys in order:
   1. Keep adding the mapped values to these keys (runningSum)
   2. Each time runningSum reaches 0, add the start Key from when the runningSum had started to a boundary list.
2. Return the boundary list.