org.apache.hadoop.hive.ql.exec.repl.util

Class SnapshotUtils

java.lang.Object

org.apache.hadoop.hive.ql.exec.repl.util.SnapshotUtils

public class SnapshotUtils
extends Object

Utility class for snapshot related operations.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SnapshotUtils.ReplSnapshotCount Snapshot count holder.
static class	SnapshotUtils.SnapshotCopyMode Specifies the mode of copy when using Snapshots for replication.

Field Summary

Fields

Modifier and Type	Field and Description
static String	NEW_SNAPSHOT
static String	OLD_SNAPSHOT

Constructor Summary

Constructors

Constructor and Description
SnapshotUtils()

Method Summary

Modifier and Type	Method and Description
static void	allowSnapshot(org.apache.hadoop.hdfs.DistributedFileSystem dfs, org.apache.hadoop.fs.Path snapshotPath, HiveConf conf) Attempts to allow snapshots on the path, with retry.
static void	cleanupSnapshots(org.apache.hadoop.fs.Path dumpRoot, String snapshotPrefix, HiveConf conf, SnapshotUtils.ReplSnapshotCount snapshotCount, boolean isLoad) Cleans up any snapshots by computing diff between the list of snapshots between two replication dumps.
static void	createSnapshot(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path snapshotPath, String snapshotName, HiveConf conf) Attempts creating a snapshot, with retries.
static boolean	deleteSnapshotIfExists(org.apache.hadoop.hdfs.DistributedFileSystem fs, org.apache.hadoop.fs.Path snapshotPath, String snapshotName, HiveConf conf) Attempts deleting a snapshot if it exists, with retries.
static boolean	deleteSnapshotSafe(org.apache.hadoop.hdfs.DistributedFileSystem dfs, org.apache.hadoop.fs.Path snapshotPath, String snapshotName) Attempts deleting a snapshot, if exists.
static void	disallowSnapshot(org.apache.hadoop.hdfs.DistributedFileSystem dfs, org.apache.hadoop.fs.Path snapshotPath) Attempts to disallow snapshot on a path, ignoring exceptions.
static String	firstSnapshot(String prefix) Gets the name of the first snapshot, combined with the prefix.
static org.apache.hadoop.hdfs.DistributedFileSystem	getDFS(org.apache.hadoop.fs.Path path, HiveConf conf) Gets a DistributedFileSystem object if possible from a path.
static ArrayList<String>	getListFromFileList(FileList fl) Extracts the entries from FileList into an ArrayList
static org.apache.hadoop.fs.Path	getSnapshotFileListPath(org.apache.hadoop.fs.Path dumpRoot) Gets the path where the snapshotfile needs to be created.
static boolean	isSnapshotAvailable(org.apache.hadoop.hdfs.DistributedFileSystem dfs, org.apache.hadoop.fs.Path path, String snapshotPrefix, String snapshotName, HiveConf conf) Checks whether a given snapshot exists or not.
static void	renameSnapshot(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path snapshotPath, String sourceSnapshotName, String targetSnapshotName, HiveConf conf) Renames a snapshot with retries.
static String	secondSnapshot(String prefix) Gets the name of the second snapshot, combined with the prefix.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OLD_SNAPSHOT

public static final String OLD_SNAPSHOT

See Also:

Constant Field Values

NEW_SNAPSHOT

public static final String NEW_SNAPSHOT

See Also:

Constant Field Values

Constructor Detail

SnapshotUtils

public SnapshotUtils()

Method Detail

getDFS

public static org.apache.hadoop.hdfs.DistributedFileSystem getDFS(org.apache.hadoop.fs.Path path,
                                                                  HiveConf conf)
                                                           throws IOException

Gets a DistributedFileSystem object if possible from a path.

Parameters:

path - path from which DistributedFileSystem needs to be extracted.

conf - Hive Configuration.

Returns:

DFS or null.

Throws:

IOException

isSnapshotAvailable

public static boolean isSnapshotAvailable(org.apache.hadoop.hdfs.DistributedFileSystem dfs,
                                          org.apache.hadoop.fs.Path path,
                                          String snapshotPrefix,
                                          String snapshotName,
                                          HiveConf conf)
                                   throws IOException

Checks whether a given snapshot exists or not.

Parameters:

dfs - DistributedFileSystem.

path - path of snapshot.

snapshotPrefix - snapshot name prefix.

snapshotName - name of snapshot.

conf - Hive configuration.

Returns:

true if the snapshot exists.

Throws:

IOException - in case of any error.

deleteSnapshotSafe

public static boolean deleteSnapshotSafe(org.apache.hadoop.hdfs.DistributedFileSystem dfs,
                                         org.apache.hadoop.fs.Path snapshotPath,
                                         String snapshotName)
                                  throws IOException

Attempts deleting a snapshot, if exists.

Parameters:

dfs - DistributedFileSystem

snapshotPath - path of snapshot

snapshotName - name of the snapshot

Returns:

true if the snapshot gets deleted.

Throws:

IOException

deleteSnapshotIfExists

public static boolean deleteSnapshotIfExists(org.apache.hadoop.hdfs.DistributedFileSystem fs,
                                             org.apache.hadoop.fs.Path snapshotPath,
                                             String snapshotName,
                                             HiveConf conf)
                                      throws IOException

Attempts deleting a snapshot if it exists, with retries.

Parameters:

fs - FileSystem object.

snapshotPath - Path of the snapshot.

snapshotName - Name of the snapshot.

conf - Hive Configuration.

Returns:

true if snapshot is no longer available.

Throws:

IOException

disallowSnapshot

public static void disallowSnapshot(org.apache.hadoop.hdfs.DistributedFileSystem dfs,
                                    org.apache.hadoop.fs.Path snapshotPath)

Attempts to disallow snapshot on a path, ignoring exceptions.

Parameters:

dfs - DistributedFileSystem

snapshotPath - path of snapshot

allowSnapshot

public static void allowSnapshot(org.apache.hadoop.hdfs.DistributedFileSystem dfs,
                                 org.apache.hadoop.fs.Path snapshotPath,
                                 HiveConf conf)
                          throws IOException

Attempts to allow snapshots on the path, with retry.

Parameters:

dfs - DistributedFileSystem.

snapshotPath - path of snapshot.

conf - Hive Configuration.

Throws:

IOException

createSnapshot

public static void createSnapshot(org.apache.hadoop.fs.FileSystem fs,
                                  org.apache.hadoop.fs.Path snapshotPath,
                                  String snapshotName,
                                  HiveConf conf)
                           throws IOException

Attempts creating a snapshot, with retries.

Parameters:

fs - FileSystem object.

snapshotPath - Path of the snapshot.

snapshotName - Name of the snapshot.

conf - Hive Configuration.

Throws:

IOException

renameSnapshot

public static void renameSnapshot(org.apache.hadoop.fs.FileSystem fs,
                                  org.apache.hadoop.fs.Path snapshotPath,
                                  String sourceSnapshotName,
                                  String targetSnapshotName,
                                  HiveConf conf)
                           throws IOException

Renames a snapshot with retries.

Parameters:

fs - the filesystem.

snapshotPath - the path where the snapshot lies.

sourceSnapshotName - current snapshot name.

targetSnapshotName - new snapshot name.

conf - configuration.

Throws:

IOException - in case of failure.

getListFromFileList

public static ArrayList<String> getListFromFileList(FileList fl)

Extracts the entries from FileList into an ArrayList

Parameters:

fl - the FileList

Returns:

the ArrayList containing the entries.

cleanupSnapshots

public static void cleanupSnapshots(org.apache.hadoop.fs.Path dumpRoot,
                                    String snapshotPrefix,
                                    HiveConf conf,
                                    SnapshotUtils.ReplSnapshotCount snapshotCount,
                                    boolean isLoad)
                             throws IOException,
                                    SemanticException

Cleans up any snapshots by computing diff between the list of snapshots between two replication dumps.

Parameters:

dumpRoot - Root of the dump

snapshotPrefix - prefix used by the snapshot

conf - Hive Configuration.

snapshotCount - the counter to store number of deleted snapshots.

isLoad - If this is called for clearing up snapshots at target cluster, In case of load it renames the snapshot file as well.

Throws:

IOException

SemanticException

getSnapshotFileListPath

public static org.apache.hadoop.fs.Path getSnapshotFileListPath(org.apache.hadoop.fs.Path dumpRoot)

Gets the path where the snapshotfile needs to be created. The hrepl directory.

Parameters:

dumpRoot - the dumproot.

Returns:

the path where the snapshot tracking file is to be created.

firstSnapshot

public static String firstSnapshot(String prefix)

Gets the name of the first snapshot, combined with the prefix.

Parameters:

prefix - the prefix that needs to be attached to the default hive replication first snapshot name.

Returns:

name to be used as first snapshot.

secondSnapshot

public static String secondSnapshot(String prefix)

Gets the name of the second snapshot, combined with the prefix.

Parameters:

prefix - the prefix that needs to be attached to the default hive replication second snapshot name.

Returns:

name to be used as second snapshot.