org.apache.hadoop.hive.ql.exec.repl

Class OptimisedBootstrapUtils

java.lang.Object

org.apache.hadoop.hive.ql.exec.repl.OptimisedBootstrapUtils

public class OptimisedBootstrapUtils
extends Object

Utility class for handling operations regarding optimised bootstrap in case of replication.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ABORT_TXNS_FILE abort Txns file which contains all the txns that needs to be aborted on new source cluster(initial target)
static String	BOOTSTRAP_TABLES_LIST
static String	EVENT_ACK_FILE event ack file which contains the event id till which the cluster was last loaded.
static String	FILE_ENTRY_SEPARATOR Separator used to separate entries in the listing.
static String	TABLE_DIFF_COMPLETE_DIRECTORY table diff directory when complete
static String	TABLE_DIFF_INPROGRESS_DIRECTORY table diff directory when in progress

Constructor Summary

Constructors

Constructor and Description
OptimisedBootstrapUtils()

Method Summary

Modifier and Type	Method and Description
static boolean	checkFileExists(org.apache.hadoop.fs.Path dumpPath, HiveConf conf, String fileName)
static Long	createAndGetEventAckFile(org.apache.hadoop.fs.Path currentDumpPath, DumpMetaData dmd, org.apache.hadoop.fs.Path cmRoot, String dbEventId, String targetDbEventId, HiveConf conf, ReplDumpWork work) Creates the event ack file and sets the dump metadata post that marking completion of dump flow for first round of optimised failover dump.
static void	createBootstrapTableList(org.apache.hadoop.fs.Path newDumpPath, Set<String> tablesForBootstrap, HiveConf conf) Creates the bootstrap table list for the second load of optimised bootstrap to consume.
static String[]	getBootstrapTableList(org.apache.hadoop.fs.Path dumpPath, HiveConf conf) Gets the list of tables for the second load of optimised bootstrap from the BOOTSTRAP_TABLE_LIST in the dump directory.
static String[]	getEventIdFromFile(org.apache.hadoop.fs.Path dumpPath, HiveConf conf) Gets the source & target event id from the event ack file
static List<org.apache.hadoop.hive.metastore.api.NotificationEvent>	getListOfNotificationEvents(Long eventId, Hive hiveDb, ReplLoadWork work) Returns list of notificationEvents starting from eventId that are related to the database.
static HashSet<String>	getPathsFromTableFile(String file, org.apache.hadoop.fs.Path dumpPath, HiveConf conf) Extracts the recursive listing from the table file.
static String	getReplEventIdFromDatabase(String dbName, Hive hiveDb) Gets the event id stored in database denoting the last loaded event id.
static HashSet<String>	getTablesFromTableDiffFile(org.apache.hadoop.fs.Path dumpPath, HiveConf conf) Gets the name of tables in the table diff file.
static String	getTargetEventId(String dbName, Hive hiveDb) Fetches the notification id from the database with respect to target database.
static List<Long>	getTxnIdFromAbortTxnsFile(org.apache.hadoop.fs.Path dumpPath, HiveConf conf)
static boolean	isDbTargetOfFailover(String dbName, Hive hive) Gets & checks whether the database is target of replication.
static void	isFirstIncrementalPending(String dbName, Hive hiveDb) Validates if the first incremental is done before starting optimised bootstrap
static void	prepareAbortTxnsFile(List<org.apache.hadoop.hive.metastore.api.NotificationEvent> notificationEvents, Set<Long> allOpenTxns, org.apache.hadoop.fs.Path dumpPath, HiveConf conf)
static void	prepareTableDiffFile(List<org.apache.hadoop.hive.metastore.api.NotificationEvent> notificationEvents, Hive hiveDb, ReplLoadWork work, HiveConf conf) Prepares the table diff file, with tables modified post the specified event id.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FILE_ENTRY_SEPARATOR

public static final String FILE_ENTRY_SEPARATOR

Separator used to separate entries in the listing.

See Also:

Constant Field Values

TABLE_DIFF_INPROGRESS_DIRECTORY

public static final String TABLE_DIFF_INPROGRESS_DIRECTORY

table diff directory when in progress

See Also:

Constant Field Values

TABLE_DIFF_COMPLETE_DIRECTORY

public static final String TABLE_DIFF_COMPLETE_DIRECTORY

table diff directory when complete

See Also:

Constant Field Values

ABORT_TXNS_FILE

public static final String ABORT_TXNS_FILE

abort Txns file which contains all the txns that needs to be aborted on new source cluster(initial target)

See Also:

Constant Field Values

EVENT_ACK_FILE

public static final String EVENT_ACK_FILE

event ack file which contains the event id till which the cluster was last loaded.

See Also:

Constant Field Values

BOOTSTRAP_TABLES_LIST

public static final String BOOTSTRAP_TABLES_LIST

See Also:

Constant Field Values

Constructor Detail

OptimisedBootstrapUtils

public OptimisedBootstrapUtils()

Method Detail

isDbTargetOfFailover

public static boolean isDbTargetOfFailover(String dbName,
                                           Hive hive)
                                    throws HiveException

Gets & checks whether the database is target of replication.

Parameters:

dbName - name of database

hive - hive object

Returns:

true, if the database has repl.target.for property set.

Throws:

HiveException

checkFileExists

public static boolean checkFileExists(org.apache.hadoop.fs.Path dumpPath,
                                      HiveConf conf,
                                      String fileName)
                               throws IOException

Throws:

IOException

prepareAbortTxnsFile

public static void prepareAbortTxnsFile(List<org.apache.hadoop.hive.metastore.api.NotificationEvent> notificationEvents,
                                        Set<Long> allOpenTxns,
                                        org.apache.hadoop.fs.Path dumpPath,
                                        HiveConf conf)
                                 throws SemanticException

Throws:

SemanticException

getTxnIdFromAbortTxnsFile

public static List<Long> getTxnIdFromAbortTxnsFile(org.apache.hadoop.fs.Path dumpPath,
                                                   HiveConf conf)
                                            throws IOException

Throws:

IOException

getEventIdFromFile

public static String[] getEventIdFromFile(org.apache.hadoop.fs.Path dumpPath,
                                          HiveConf conf)
                                   throws IOException

Gets the source & target event id from the event ack file

Parameters:

dumpPath - the dump path

conf - the hive configuration

Returns:

the event id from file.

Throws:

IOException

getTablesFromTableDiffFile

public static HashSet<String> getTablesFromTableDiffFile(org.apache.hadoop.fs.Path dumpPath,
                                                         HiveConf conf)
                                                  throws Exception

Gets the name of tables in the table diff file.

Parameters:

dumpPath - the dump path

conf - the hive configuration

Returns:

Set with list of tables

Throws:

Exception

getPathsFromTableFile

public static HashSet<String> getPathsFromTableFile(String file,
                                                    org.apache.hadoop.fs.Path dumpPath,
                                                    HiveConf conf)
                                             throws IOException

Extracts the recursive listing from the table file.

Parameters:

file - the name of table

dumpPath - the dump path

conf - the hive conf

Returns:

the list of paths in the table.

Throws:

IOException

getReplEventIdFromDatabase

public static String getReplEventIdFromDatabase(String dbName,
                                                Hive hiveDb)
                                         throws HiveException

Gets the event id stored in database denoting the last loaded event id.

Parameters:

dbName - the name of database

hiveDb - the hive object

Returns:

event id from the database

Throws:

HiveException

isFirstIncrementalPending

public static void isFirstIncrementalPending(String dbName,
                                             Hive hiveDb)
                                      throws HiveException

Validates if the first incremental is done before starting optimised bootstrap

Parameters:

dbName - name of database

hiveDb - the hive object

Throws:

HiveException

createAndGetEventAckFile

public static Long createAndGetEventAckFile(org.apache.hadoop.fs.Path currentDumpPath,
                                            DumpMetaData dmd,
                                            org.apache.hadoop.fs.Path cmRoot,
                                            String dbEventId,
                                            String targetDbEventId,
                                            HiveConf conf,
                                            ReplDumpWork work)
                                     throws Exception

Creates the event ack file and sets the dump metadata post that marking completion of dump flow for first round of optimised failover dump.

Parameters:

currentDumpPath - the dump path

dmd - the dump metadata

cmRoot - the cmRoot

dbEventId - the database event id to which we have to write in the file.

targetDbEventId - the database event id with respect to target cluster.

conf - the hive configuration

work - the repldump work

Returns:

the lastReplId denoting a fake dump(-1) always

Throws:

SemanticException

Exception

getListOfNotificationEvents

public static List<org.apache.hadoop.hive.metastore.api.NotificationEvent> getListOfNotificationEvents(Long eventId,
                                                                                                       Hive hiveDb,
                                                                                                       ReplLoadWork work)
                                                                                                throws Exception

Returns list of notificationEvents starting from eventId that are related to the database.

Parameters:

eventId - Starting eventId

hiveDb - the hive object

work - the load work

Throws:

Exception

prepareTableDiffFile

public static void prepareTableDiffFile(List<org.apache.hadoop.hive.metastore.api.NotificationEvent> notificationEvents,
                                        Hive hiveDb,
                                        ReplLoadWork work,
                                        HiveConf conf)
                                 throws Exception

Prepares the table diff file, with tables modified post the specified event id.

Parameters:

notificationEvents - Events that can possibly contain table DDL/DML metadata.

hiveDb - the hive object

work - the load work

conf - hive configuration

Throws:

Exception

getTargetEventId

public static String getTargetEventId(String dbName,
                                      Hive hiveDb)
                               throws Exception

Fetches the notification id from the database with respect to target database.

Parameters:

dbName - name of database

hiveDb - the hive object

Returns:

the corresponding notification event id from target database

Throws:

Exception

createBootstrapTableList

public static void createBootstrapTableList(org.apache.hadoop.fs.Path newDumpPath,
                                            Set<String> tablesForBootstrap,
                                            HiveConf conf)
                                     throws SemanticException

Creates the bootstrap table list for the second load of optimised bootstrap to consume.

Parameters:

newDumpPath - the dump path

tablesForBootstrap - the list of tables.

conf - the hive configuration

Throws:

SemanticException - in case of any exception

getBootstrapTableList

public static String[] getBootstrapTableList(org.apache.hadoop.fs.Path dumpPath,
                                             HiveConf conf)
                                      throws IOException

Gets the list of tables for the second load of optimised bootstrap from the BOOTSTRAP_TABLE_LIST in the dump directory.

Parameters:

dumpPath - the dump path

conf - the hive configuration

Returns:

the array with list of tables to bootstrap

Throws:

IOException - in case of any exception.