org.apache.hadoop.hive.metastore

Class ReplChangeManager

java.lang.Object

org.apache.hadoop.hive.metastore.ReplChangeManager

public class ReplChangeManager
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ReplChangeManager.CMClearer Thread to clear old files of cmroot recursively
static class	ReplChangeManager.FileInfo
static class	ReplChangeManager.RecycleType

Field Summary

Fields

Modifier and Type	Field and Description
static String	CM_THREAD_NAME_PREFIX
static org.apache.hadoop.fs.PathFilter	CMROOT_PATH_FILTER
static String	SOURCE_OF_REPLICATION

Method Summary

Modifier and Type	Method and Description
static String	checksumFor(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.FileSystem fs)
static String[]	decodeFileUri(String fileURIStr) Split uri with fragment into file uri, subdirs, checksum and source cmroot uri.
String	encodeFileUri(String fileUriStr, String fileChecksum, String encodedSubDir) Concatenate filename, checksum, source cmroot uri and subdirectory with "#"
static String	encodeFileUri(String fileUriStr, String fileChecksum, String cmRoot, String encodedSubDir)
static ReplChangeManager.FileInfo	getFileInfo(org.apache.hadoop.fs.Path src, String checksumString, String srcCMRootURI, String subDir, org.apache.hadoop.conf.Configuration conf) Get original file specified by src and chksumString.
static ReplChangeManager	getInstance()
static ReplChangeManager	getInstance(org.apache.hadoop.conf.Configuration conf)
static String[]	getListFromSeparatedString(String commaSeparatedString)
static String	getReplPolicyIdString(Database db)
static boolean	isCMFileUri(org.apache.hadoop.fs.Path fromPath)
static boolean	isSourceOfReplication(Database db)
static String	joinWithSeparator(Iterable<?> strings)
int	recycle(org.apache.hadoop.fs.Path path, ReplChangeManager.RecycleType type, boolean ifPurge) Move a path into cmroot.
static boolean	shouldEnableCm(Database db, Table table)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SOURCE_OF_REPLICATION

public static final String SOURCE_OF_REPLICATION

See Also:

Constant Field Values

CM_THREAD_NAME_PREFIX

public static final String CM_THREAD_NAME_PREFIX

See Also:

Constant Field Values

CMROOT_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter CMROOT_PATH_FILTER

Method Detail

getInstance

public static ReplChangeManager getInstance(org.apache.hadoop.conf.Configuration conf)
                                     throws MetaException

Throws:

MetaException

getInstance

public static ReplChangeManager getInstance()

recycle

public int recycle(org.apache.hadoop.fs.Path path,
                   ReplChangeManager.RecycleType type,
                   boolean ifPurge)
            throws IOException

Move a path into cmroot. If the path is a directory (of a partition, or table if nonpartitioned), recursively move files inside directory to cmroot. Note the table must be managed table

Parameters:

path - a single file or directory

type - if the files to be copied or moved to cmpath. Copy is costly but preserve the source file

ifPurge - if the file should skip Trash when move/delete source file. This is referred only if type is MOVE.

Returns:

int

Throws:

IOException

checksumFor

public static String checksumFor(org.apache.hadoop.fs.Path path,
                                 org.apache.hadoop.fs.FileSystem fs)
                          throws IOException

Throws:

IOException

getFileInfo

public static ReplChangeManager.FileInfo getFileInfo(org.apache.hadoop.fs.Path src,
                                                     String checksumString,
                                                     String srcCMRootURI,
                                                     String subDir,
                                                     org.apache.hadoop.conf.Configuration conf)
                                              throws IOException

Get original file specified by src and chksumString. If the file exists and checksum matches, return the file; otherwise, use chksumString to retrieve it from cmroot

Parameters:

src - Original file location

checksumString - Checksum of the original file

srcCMRootURI - CM root URI of the source cluster

subDir - Sub directory to which the source file belongs to

conf - Hive configuration

Returns:

Corresponding FileInfo object

Throws:

IOException

encodeFileUri

public String encodeFileUri(String fileUriStr,
                            String fileChecksum,
                            String encodedSubDir)
                     throws IOException

Concatenate filename, checksum, source cmroot uri and subdirectory with "#"

Parameters:

fileUriStr - Filename string

fileChecksum - Checksum string

encodedSubDir - sub directory path into which this file belongs to. Here encoded means, the multiple levels of subdirectories are concatenated with path separator "/"

Returns:

Concatenated Uri string

Throws:

IOException

encodeFileUri

public static String encodeFileUri(String fileUriStr,
                                   String fileChecksum,
                                   String cmRoot,
                                   String encodedSubDir)

decodeFileUri

public static String[] decodeFileUri(String fileURIStr)

Split uri with fragment into file uri, subdirs, checksum and source cmroot uri. Currently using fileuri#checksum#cmrooturi#subdirs as the format.

Parameters:

fileURIStr - uri with fragment

Returns:

array of file name, subdirs, checksum and source CM root URI

isCMFileUri

public static boolean isCMFileUri(org.apache.hadoop.fs.Path fromPath)

shouldEnableCm

public static boolean shouldEnableCm(Database db,
                                     Table table)

isSourceOfReplication

public static boolean isSourceOfReplication(Database db)

getReplPolicyIdString

public static String getReplPolicyIdString(Database db)

joinWithSeparator

public static String joinWithSeparator(Iterable<?> strings)

getListFromSeparatedString

public static String[] getListFromSeparatedString(String commaSeparatedString)