org.apache.hadoop.hive.ql.io

Class AcidUtils

java.lang.Object

org.apache.hadoop.hive.ql.io.AcidUtils

public class AcidUtils
extends Object

Utilities that are shared by all of the ACID input and output formats. They are used by the compactor and cleaner and thus must be format agnostic.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AcidUtils.AcidBaseFileType
static class	AcidUtils.AcidOperationalProperties Current syntax for creating full acid transactional tables is any one of following 3 ways: create table T (a int, b int) stored as orc tblproperties('transactional'='true').
static class	AcidUtils.AnyIdDirFilter
static class	AcidUtils.BucketMetaData Represents bucketId and copy_N suffix
static interface	AcidUtils.Directory
static class	AcidUtils.FileInfo A simple wrapper class that stores the information about a base file and its type.
static interface	AcidUtils.HdfsDirSnapshot DFS dir listing.
static class	AcidUtils.HdfsDirSnapshotImpl
static class	AcidUtils.IdFullPathFiler Full recursive PathFilter version of IdPathFilter (filtering files for a given writeId and stmtId).
static class	AcidUtils.IdPathFilter
static class	AcidUtils.MetaDataFile General facility to place a metadata file into a dir created by acid/compactor write.
static class	AcidUtils.Operation
static class	AcidUtils.OrcAcidVersion Logic related to versioning acid data format.
static class	AcidUtils.ParsedBase In addition to AcidUtils.ParsedBaseLight this knows if the data is in raw format, i.e.
static class	AcidUtils.ParsedBaseLight Since version 3 but prior to version 4, format of a base is "base_X" where X is a writeId.
static class	AcidUtils.ParsedDelta In addition to AcidUtils.ParsedDeltaLight this knows if the data is in raw format, i.e.
static class	AcidUtils.ParsedDeltaLight This encapsulates info obtained form the file path.
static interface	AcidUtils.ParsedDirectory
static class	AcidUtils.TableSnapshot

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	acidHiddenFileFilter
static org.apache.hadoop.fs.PathFilter	acidTempDirFilter
static String	BASE_PREFIX
static org.apache.hadoop.fs.PathFilter	baseFileFilter
static String	BUCKET_DIGITS
static Pattern	BUCKET_PATTERN
static String	BUCKET_PREFIX
static org.apache.hadoop.fs.PathFilter	bucketFileFilter
static String	COMPACTOR_TABLE_PROPERTY
static String	CONF_ACID_KEY
static String	DELETE_DELTA_PREFIX
static org.apache.hadoop.fs.PathFilter	deleteEventDeltaDirFilter
static String	DELTA_DIGITS
static String	DELTA_PREFIX
static String	DELTA_SIDE_FILE_SUFFIX Acid Streaming Ingest writes multiple transactions to the same file.
static org.apache.hadoop.fs.PathFilter	deltaFileFilter
static Pattern	LEGACY_BUCKET_DIGIT_PATTERN
static String	LEGACY_FILE_BUCKET_DIGITS
static int	MAX_STATEMENTS_PER_TXN This must be in sync with STATEMENT_DIGITS
static Pattern	ORIGINAL_PATTERN
static Pattern	ORIGINAL_PATTERN_COPY
static org.apache.hadoop.fs.PathFilter	originalBucketFilter A write into a non-aicd table produces files like 0000_0 or 0000_0_copy_1 (Unless via Load Data statement)
static String	STATEMENT_DIGITS 10K statements per tx.
static Pattern	VISIBILITY_PATTERN
static String	VISIBILITY_PREFIX

Method Summary

Modifier and Type	Method and Description
static boolean	acidTableWithoutTransactions(Table table)
static String	addVisibilitySuffix(String baseOrDeltaDir, long visibilityTxnId) Since Hive 4.0, compactor produces directories with VISIBILITY_PATTERN suffix.
static String	baseDir(long writeId)
static String	baseOrDeltaSubdir(boolean baseDirRequired, long min, long max, int statementId) Return a base or delta directory string according to the given "baseDirRequired".
static org.apache.hadoop.fs.Path	baseOrDeltaSubdirPath(org.apache.hadoop.fs.Path directory, AcidOutputFormat.Options options) Return a base or delta directory path according to the given "options".
static boolean	canBeMadeAcid(String fullTableName, org.apache.hadoop.hive.metastore.api.StorageDescriptor sd)
static CompactionState	compactionStateStr2Enum(String inputValue)
static org.apache.hadoop.hive.metastore.api.CompactionType	compactionTypeStr2ThriftType(String inputValue)
static org.apache.hadoop.fs.Path	createBucketFile(org.apache.hadoop.fs.Path subdir, int bucket) Create the bucket filename in Acid format
static org.apache.hadoop.fs.Path	createBucketFile(org.apache.hadoop.fs.Path subdir, int bucket, Integer attemptId)
static org.apache.hadoop.fs.Path	createFilename(org.apache.hadoop.fs.Path directory, AcidOutputFormat.Options options) Create a filename for a bucket file.
static String	deleteDeltaSubdir(long min, long max) This is format of delete delta dir name prior to Hive 2.2.x
static String	deleteDeltaSubdir(long min, long max, int statementId) Each write statement in a transaction creates its own delete delta dir, when split-update acid operational property is turned on.
static String	deltaSubdir(long min, long max) This is format of delta dir name prior to Hive 1.3.x
static String	deltaSubdir(long min, long max, int statementId) Each write statement in a transaction creates its own delta dir.
static org.apache.hadoop.fs.Path[]	deserializeDeleteDeltas(org.apache.hadoop.fs.Path root, List<AcidInputFormat.DeltaMetaData> deleteDeltas, Map<String,AcidInputFormat.DeltaMetaData> pathToDeltaMetaData) Convert the list of begin/end write id pairs to a list of delete delta directories.
static Long	extractWriteId(org.apache.hadoop.fs.Path file)
static List<org.apache.hadoop.fs.FileStatus>	getAcidFilesForStats(Table table, org.apache.hadoop.fs.Path dir, org.apache.hadoop.conf.Configuration jc, org.apache.hadoop.fs.FileSystem fs)
static AcidUtils.AcidOperationalProperties	getAcidOperationalProperties(org.apache.hadoop.conf.Configuration conf) Returns the acidOperationalProperties for a given configuration.
static AcidUtils.AcidOperationalProperties	getAcidOperationalProperties(Map<String,String> parameters) Returns the acidOperationalProperties for a given map.
static AcidUtils.AcidOperationalProperties	getAcidOperationalProperties(Properties props) Returns the acidOperationalProperties for a given set of properties.
static AcidUtils.AcidOperationalProperties	getAcidOperationalProperties(Table table) Returns the acidOperationalProperties for a given table.
static AcidDirectory	getAcidState(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path candidateDirectory, org.apache.hadoop.conf.Configuration conf, ValidWriteIdList writeIdList, Ref<Boolean> useFileIds, boolean ignoreEmptyFiles) Get the ACID state of the given directory.
static AcidDirectory	getAcidState(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path candidateDirectory, org.apache.hadoop.conf.Configuration conf, ValidWriteIdList writeIdList, Ref<Boolean> useFileIds, boolean ignoreEmptyFiles, Map<org.apache.hadoop.fs.Path,AcidUtils.HdfsDirSnapshot> dirSnapshots) GetAcidState implementation which uses the provided dirSnapshot.
static AcidDirectory	getAcidStateFromCache(Supplier<org.apache.hadoop.fs.FileSystem> fileSystem, org.apache.hadoop.fs.Path candidateDirectory, org.apache.hadoop.conf.Configuration conf, ValidWriteIdList writeIdList, Ref<Boolean> useFileIds, boolean ignoreEmptyFiles) Tries to get directory details from cache.
static String	getAcidSubDir(org.apache.hadoop.fs.Path dataPath)
static List<VirtualColumn>	getAcidVirtualColumns(Table table) Returns the virtual columns needed for update queries.
static Map<String,Integer>	getDeltaToAttemptIdMap(Map<String,AcidInputFormat.DeltaMetaData> pathToDeltaMetaData, org.apache.hadoop.fs.Path[] deleteDeltaDirs, int bucket) If the direct insert is on for ACID tables, the files will contain an "_attemptID" postfix.
static String	getFirstLevelAcidDirPath(org.apache.hadoop.fs.Path dataPath, org.apache.hadoop.fs.FileSystem fileSystem)
static String	getFullTableName(String dbName, String tableName)
static Map<org.apache.hadoop.fs.Path,AcidUtils.HdfsDirSnapshot>	getHdfsDirSnapshots(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static Map<org.apache.hadoop.fs.Path,AcidUtils.HdfsDirSnapshot>	getHdfsDirSnapshotsForCleaner(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) In case of the cleaner, we don't need to go into file level, it is enough to collect base/delta/deletedelta directories.
static long	getLogicalLength(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus file) See comments at DELTA_SIDE_FILE_SUFFIX.
static String	getPartitionName(Map<String,String> partitionSpec)
static org.apache.hadoop.fs.Path[]	getPaths(List<AcidUtils.ParsedDelta> deltas) Convert a list of deltas to a list of delta directories.
static String	getPathSuffix(long txnId)
static AcidUtils.TableSnapshot	getTableSnapshot(org.apache.hadoop.conf.Configuration conf, Table tbl)
static AcidUtils.TableSnapshot	getTableSnapshot(org.apache.hadoop.conf.Configuration conf, Table tbl, boolean isStatsUpdater) Note: this is generally called in Hive.java; so, the callers of Hive.java make sure to set up the acid state during compile, and Hive.java retrieves it if needed.
static AcidUtils.TableSnapshot	getTableSnapshot(org.apache.hadoop.conf.Configuration conf, Table tbl, String dbName, String tblName, boolean isStatsUpdater) Note: this is generally called in Hive.java; so, the callers of Hive.java make sure to set up the acid state during compile, and Hive.java retrieves it if needed.
static ValidWriteIdList	getTableValidWriteIdList(org.apache.hadoop.conf.Configuration conf, String fullTableName) Extract the ValidWriteIdList for the given table from the list of tables' ValidWriteIdList.
static ValidWriteIdList	getTableValidWriteIdListWithTxnList(org.apache.hadoop.conf.Configuration conf, String dbName, String tableName) Returns ValidWriteIdList for the table with the given "dbName" and "tableName".
static org.apache.hadoop.hive.metastore.api.TxnType	getTxnType(org.apache.hadoop.conf.Configuration conf, ASTNode tree) Determines transaction type based on query AST.
static List<org.apache.hadoop.fs.Path>	getValidDataPaths(org.apache.hadoop.fs.Path dataPath, org.apache.hadoop.conf.Configuration conf, String validWriteIdStr)
static ValidTxnWriteIdList	getValidTxnWriteIdList(org.apache.hadoop.conf.Configuration conf) Get the ValidTxnWriteIdList saved in the configuration.
static void	initDirCache(int durationInMts)
static boolean	isAcid(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path directory, org.apache.hadoop.conf.Configuration conf)
static boolean	isAcid(org.apache.hadoop.fs.Path directory, org.apache.hadoop.conf.Configuration conf) Is the given directory in ACID format?
static boolean	isAcidEnabled(HiveConf hiveConf)
static boolean	isChildOfDelta(org.apache.hadoop.fs.Path childDir, org.apache.hadoop.fs.Path rootPath)
static boolean	isCompactionTable(Map<String,String> parameters) Determine if a table is used during query based compaction.
static boolean	isCompactionTable(Properties tblProperties) Determine if a table is used during query based compaction.
static boolean	isDeleteDelta(org.apache.hadoop.fs.Path p)
static boolean	isExclusiveCTAS(Set<WriteEntity> outputs, HiveConf conf)
static boolean	isExclusiveCTASEnabled(org.apache.hadoop.conf.Configuration conf)
static boolean	isFullAcidScan(org.apache.hadoop.conf.Configuration conf)
static boolean	isFullAcidTable(CreateTableDesc td)
static boolean	isFullAcidTable(Map<String,String> params)
static boolean	isFullAcidTable(Table table) Should produce the same result as TxnUtils.isAcidTable(org.apache.hadoop.hive.metastore.api.Table)
static boolean	isFullAcidTable(org.apache.hadoop.hive.metastore.api.Table table) Should produce the same result as TxnUtils.isAcidTable(org.apache.hadoop.hive.metastore.api.Table)
static boolean	isInsertDelta(org.apache.hadoop.fs.Path p)
static boolean	isInsertOnlyFetchBucketId(org.apache.hadoop.conf.Configuration conf)
static boolean	isInsertOnlyTable(Map<String,String> params) Checks if a table is a transactional table that only supports INSERT, but not UPDATE/DELETE
static boolean	isInsertOnlyTable(Properties params)
static boolean	isInsertOnlyTable(Table table)
static boolean	isLocklessReadsEnabled(Table table, HiveConf conf)
static boolean	isNonNativeAcidTable(Table table)
static boolean	isRemovedInsertOnlyTable(Set<String> removedSet)
static boolean	isTablePropertyTransactional(Map<String,String> parameters)
static boolean	isTablePropertyTransactional(Properties props)
static boolean	isTableSoftDeleteEnabled(Table table, HiveConf conf)
static Boolean	isToFullAcid(Table table, Map<String,String> props)
static Boolean	isToInsertOnlyTable(Table tbl, Map<String,String> props) The method for altering table props; may set the table to MM, non-MM, or not affect MM.
static boolean	isTransactionalTable(CreateTableDesc table)
static boolean	isTransactionalTable(Map<String,String> props)
static boolean	isTransactionalTable(Table table)
static boolean	isTransactionalTable(org.apache.hadoop.hive.metastore.api.Table table)
static boolean	isTransactionalView(CreateMaterializedViewDesc view)
static List<org.apache.hadoop.hive.metastore.api.LockComponent>	makeLockComponents(Set<WriteEntity> outputs, Set<ReadEntity> inputs, Context.Operation operation, HiveConf conf) Create lock components from write/read entities.
static Integer	parseAttemptId(org.apache.hadoop.fs.Path bucketFile)
static AcidOutputFormat.Options	parseBaseOrDeltaBucketFilename(org.apache.hadoop.fs.Path bucketFile, org.apache.hadoop.conf.Configuration conf) Parse a bucket filename back into the options that would have created the file.
static int	parseBucketId(org.apache.hadoop.fs.Path bucketFile) Get the bucket id from the file path
static AcidUtils.ParsedDelta	parsedDelta(org.apache.hadoop.fs.Path deltaDir, org.apache.hadoop.fs.FileSystem fs) This will look at a footer of one of the files in the delta to see if the file is in Acid format, i.e.
static List<AcidInputFormat.DeltaMetaData>	serializeDeleteDeltas(List<AcidUtils.ParsedDelta> deltas, org.apache.hadoop.fs.FileSystem fs) Convert the list of deltas into an equivalent list of begin/end write id pairs.
static void	setAcidOperationalProperties(org.apache.hadoop.conf.Configuration conf, boolean isTxnTable, AcidUtils.AcidOperationalProperties properties) Sets the acidOperationalProperties in the configuration object argument.
static void	setAcidOperationalProperties(Map<String,String> parameters, boolean isTxnTable, AcidUtils.AcidOperationalProperties properties) Sets the acidOperationalProperties in the map object argument.
static void	setNonTransactional(Map<String,String> tblProps)
static void	setValidWriteIdList(org.apache.hadoop.conf.Configuration conf, TableScanDesc tsDesc) Set the valid write id list for the current table scan.
static void	setValidWriteIdList(org.apache.hadoop.conf.Configuration conf, ValidWriteIdList validWriteIds) Set the valid write id list for the current table scan.
static org.apache.hadoop.hive.metastore.api.DataOperationType	toDataOperationType(AcidUtils.Operation op) Logically this should have been defined in Operation but that causes a dependency on metastore package from exec jar (from the cluster) which is not allowed.
static void	tryInvalidateDirCache(org.apache.hadoop.hive.metastore.api.Table table)
static void	validateAcidFiles(Table table, org.apache.hadoop.fs.FileStatus[] srcs, org.apache.hadoop.fs.FileSystem fs) Safety check to make sure a file take from one acid table is not added into another acid table since the ROW__IDs embedded as part a write to one table won't make sense in different table/cluster.
static void	validateAcidPartitionLocation(String location, org.apache.hadoop.conf.Configuration conf) Safety check to make sure the given location is not the location of acid table and all it's files will be not added into another acid table

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONF_ACID_KEY

public static final String CONF_ACID_KEY

See Also:

Constant Field Values

BASE_PREFIX

public static final String BASE_PREFIX

See Also:

Constant Field Values

COMPACTOR_TABLE_PROPERTY

public static final String COMPACTOR_TABLE_PROPERTY

See Also:

Constant Field Values

baseFileFilter

public static final org.apache.hadoop.fs.PathFilter baseFileFilter

DELTA_PREFIX

public static final String DELTA_PREFIX

See Also:

Constant Field Values

DELETE_DELTA_PREFIX

public static final String DELETE_DELTA_PREFIX

See Also:

Constant Field Values

DELTA_SIDE_FILE_SUFFIX

public static final String DELTA_SIDE_FILE_SUFFIX

Acid Streaming Ingest writes multiple transactions to the same file. It also maintains a OrcAcidUtils.getSideFile(Path) side file which stores the length of the primary file as of the last commit (OrcRecordUpdater.flush()). That is the 'logical length'. Once the primary is closed, the side file is deleted (logical length = actual length) but if the writer dies or the primary file is being read while its still being written to, anything past the logical length should be ignored.

See Also:

OrcAcidUtils.DELTA_SIDE_FILE_SUFFIX, OrcAcidUtils.getLastFlushLength(FileSystem, Path), getLogicalLength(FileSystem, FileStatus), Constant Field Values

deltaFileFilter

public static final org.apache.hadoop.fs.PathFilter deltaFileFilter

deleteEventDeltaDirFilter

public static final org.apache.hadoop.fs.PathFilter deleteEventDeltaDirFilter

BUCKET_PREFIX

public static final String BUCKET_PREFIX

See Also:

Constant Field Values

bucketFileFilter

public static final org.apache.hadoop.fs.PathFilter bucketFileFilter

BUCKET_DIGITS

public static final String BUCKET_DIGITS

See Also:

Constant Field Values

LEGACY_FILE_BUCKET_DIGITS

public static final String LEGACY_FILE_BUCKET_DIGITS

See Also:

Constant Field Values

DELTA_DIGITS

public static final String DELTA_DIGITS

See Also:

Constant Field Values

STATEMENT_DIGITS

public static final String STATEMENT_DIGITS

10K statements per tx. Probably overkill ... since that many delta files would not be good for performance

See Also:

Constant Field Values

MAX_STATEMENTS_PER_TXN

public static final int MAX_STATEMENTS_PER_TXN

This must be in sync with STATEMENT_DIGITS

See Also:

Constant Field Values

LEGACY_BUCKET_DIGIT_PATTERN

public static final Pattern LEGACY_BUCKET_DIGIT_PATTERN

BUCKET_PATTERN

public static final Pattern BUCKET_PATTERN

originalBucketFilter

public static final org.apache.hadoop.fs.PathFilter originalBucketFilter

A write into a non-aicd table produces files like 0000_0 or 0000_0_copy_1 (Unless via Load Data statement)

ORIGINAL_PATTERN

public static final Pattern ORIGINAL_PATTERN

ORIGINAL_PATTERN_COPY

public static final Pattern ORIGINAL_PATTERN_COPY

See Also:

Utilities.COPY_KEYWORD

acidHiddenFileFilter

public static final org.apache.hadoop.fs.PathFilter acidHiddenFileFilter

acidTempDirFilter

public static final org.apache.hadoop.fs.PathFilter acidTempDirFilter

VISIBILITY_PREFIX

public static final String VISIBILITY_PREFIX

See Also:

Constant Field Values

VISIBILITY_PATTERN

public static final Pattern VISIBILITY_PATTERN

Method Detail

createBucketFile

public static org.apache.hadoop.fs.Path createBucketFile(org.apache.hadoop.fs.Path subdir,
                                                         int bucket)

Create the bucket filename in Acid format

Parameters:

subdir - the subdirectory for the bucket.

bucket - the bucket number

Returns:

the filename

createBucketFile

public static org.apache.hadoop.fs.Path createBucketFile(org.apache.hadoop.fs.Path subdir,
                                                         int bucket,
                                                         Integer attemptId)

deltaSubdir

public static String deltaSubdir(long min,
                                 long max)

This is format of delta dir name prior to Hive 1.3.x

deltaSubdir

public static String deltaSubdir(long min,
                                 long max,
                                 int statementId)

Each write statement in a transaction creates its own delta dir.

Since:

1.3.x

deleteDeltaSubdir

public static String deleteDeltaSubdir(long min,
                                       long max)

This is format of delete delta dir name prior to Hive 2.2.x

deleteDeltaSubdir

public static String deleteDeltaSubdir(long min,
                                       long max,
                                       int statementId)

Each write statement in a transaction creates its own delete delta dir, when split-update acid operational property is turned on.

Since:

2.2.x

baseDir

public static String baseDir(long writeId)

baseOrDeltaSubdir

public static String baseOrDeltaSubdir(boolean baseDirRequired,
                                       long min,
                                       long max,
                                       int statementId)

Return a base or delta directory string according to the given "baseDirRequired".

baseOrDeltaSubdirPath

public static org.apache.hadoop.fs.Path baseOrDeltaSubdirPath(org.apache.hadoop.fs.Path directory,
                                                              AcidOutputFormat.Options options)

Return a base or delta directory path according to the given "options".

createFilename

public static org.apache.hadoop.fs.Path createFilename(org.apache.hadoop.fs.Path directory,
                                                       AcidOutputFormat.Options options)

Create a filename for a bucket file.

Parameters:

directory - the partition directory

options - the options for writing the bucket

Returns:

the filename that should store the bucket

addVisibilitySuffix

public static String addVisibilitySuffix(String baseOrDeltaDir,
                                         long visibilityTxnId)

Since Hive 4.0, compactor produces directories with VISIBILITY_PATTERN suffix. _v0 is equivalent to no suffix, for backwards compatibility.

isLocklessReadsEnabled

public static boolean isLocklessReadsEnabled(Table table,
                                             HiveConf conf)

isTableSoftDeleteEnabled

public static boolean isTableSoftDeleteEnabled(Table table,
                                               HiveConf conf)

isCompactionTable

public static boolean isCompactionTable(Properties tblProperties)

Determine if a table is used during query based compaction.

Parameters:

tblProperties - table properties

Returns:

true, if the tblProperties contains COMPACTOR_TABLE_PROPERTY

isCompactionTable

public static boolean isCompactionTable(Map<String,String> parameters)

Determine if a table is used during query based compaction.

Parameters:

parameters - table properties map

Returns:

true, if the parameters contains COMPACTOR_TABLE_PROPERTY

parseBucketId

public static int parseBucketId(org.apache.hadoop.fs.Path bucketFile)

Get the bucket id from the file path

Parameters:

bucketFile - - bucket file path

Returns:

- bucket id

parseAttemptId

public static Integer parseAttemptId(org.apache.hadoop.fs.Path bucketFile)

parseBaseOrDeltaBucketFilename

public static AcidOutputFormat.Options parseBaseOrDeltaBucketFilename(org.apache.hadoop.fs.Path bucketFile,
                                                                      org.apache.hadoop.conf.Configuration conf)

Parse a bucket filename back into the options that would have created the file.

Parameters:

bucketFile - the path to a bucket file

conf - the configuration

Returns:

the options used to create that filename

getDeltaToAttemptIdMap

public static Map<String,Integer> getDeltaToAttemptIdMap(Map<String,AcidInputFormat.DeltaMetaData> pathToDeltaMetaData,
                                                         org.apache.hadoop.fs.Path[] deleteDeltaDirs,
                                                         int bucket)

If the direct insert is on for ACID tables, the files will contain an "_attemptID" postfix. In order to be able to read the files from the delete deltas, we need to know which attemptId belongs to which delta. To make this lookup easy, this method created a map to link the deltas to the attemptId.

Parameters:

pathToDeltaMetaData -

deleteDeltaDirs -

bucket -

Returns:

toDataOperationType

public static org.apache.hadoop.hive.metastore.api.DataOperationType toDataOperationType(AcidUtils.Operation op)

Logically this should have been defined in Operation but that causes a dependency on metastore package from exec jar (from the cluster) which is not allowed. This method should only be called from client side where metastore.* classes are present. Not following this will not be caught by unit tests since they have all the jar loaded.

getPaths

public static org.apache.hadoop.fs.Path[] getPaths(List<AcidUtils.ParsedDelta> deltas)

Convert a list of deltas to a list of delta directories.

Parameters:

deltas - the list of deltas out of a Directory object.

Returns:

a list of delta directory paths that need to be read

serializeDeleteDeltas

public static List<AcidInputFormat.DeltaMetaData> serializeDeleteDeltas(List<AcidUtils.ParsedDelta> deltas,
                                                                        org.apache.hadoop.fs.FileSystem fs)
                                                                 throws IOException

Convert the list of deltas into an equivalent list of begin/end write id pairs. Assumes deltas is sorted.

Parameters:

deltas - sorted delete delta list

fs - FileSystem

Returns:

the list of write ids to serialize

Throws:

IOException - ex

deserializeDeleteDeltas

public static org.apache.hadoop.fs.Path[] deserializeDeleteDeltas(org.apache.hadoop.fs.Path root,
                                                                  List<AcidInputFormat.DeltaMetaData> deleteDeltas,
                                                                  Map<String,AcidInputFormat.DeltaMetaData> pathToDeltaMetaData)

Convert the list of begin/end write id pairs to a list of delete delta directories. Note that there may be multiple delete_delta files for the exact same txn range starting with 2.2.x; see deltaSubdir(long, long, int)

Parameters:

root - the root directory

deleteDeltas - list of begin/end write id pairs

Returns:

the list of delta paths

parsedDelta

public static AcidUtils.ParsedDelta parsedDelta(org.apache.hadoop.fs.Path deltaDir,
                                                org.apache.hadoop.fs.FileSystem fs)
                                         throws IOException

This will look at a footer of one of the files in the delta to see if the file is in Acid format, i.e. has acid metadata columns. The assumption is that for any dir, either all files are acid or all are not.

Throws:

IOException

isAcid

public static boolean isAcid(org.apache.hadoop.fs.Path directory,
                             org.apache.hadoop.conf.Configuration conf)
                      throws IOException

Is the given directory in ACID format?

Parameters:

directory - the partition directory to check

conf - the query configuration

Returns:

true, if it is an ACID directory

Throws:

IOException

isAcid

public static boolean isAcid(org.apache.hadoop.fs.FileSystem fileSystem,
                             org.apache.hadoop.fs.Path directory,
                             org.apache.hadoop.conf.Configuration conf)
                      throws IOException

Throws:

IOException

getAcidState

public static AcidDirectory getAcidState(org.apache.hadoop.fs.FileSystem fileSystem,
                                         org.apache.hadoop.fs.Path candidateDirectory,
                                         org.apache.hadoop.conf.Configuration conf,
                                         ValidWriteIdList writeIdList,
                                         Ref<Boolean> useFileIds,
                                         boolean ignoreEmptyFiles)
                                  throws IOException

Get the ACID state of the given directory. It finds the minimal set of base and diff directories. Note that because major compactions don't preserve the history, we can't use a base directory that includes a write id that we must exclude.

Parameters:

fileSystem - optional, it it is not provided, it will be derived from the candidateDirectory

candidateDirectory - the partition directory to analyze

conf - the configuration

writeIdList - the list of write ids that we are reading

useFileIds - It will be set to true, if the FileSystem supports listing with fileIds

ignoreEmptyFiles - Ignore files with 0 length

Returns:

the state of the directory

Throws:

IOException - on filesystem errors

getAcidState

public static AcidDirectory getAcidState(org.apache.hadoop.fs.FileSystem fileSystem,
                                         org.apache.hadoop.fs.Path candidateDirectory,
                                         org.apache.hadoop.conf.Configuration conf,
                                         ValidWriteIdList writeIdList,
                                         Ref<Boolean> useFileIds,
                                         boolean ignoreEmptyFiles,
                                         Map<org.apache.hadoop.fs.Path,AcidUtils.HdfsDirSnapshot> dirSnapshots)
                                  throws IOException

GetAcidState implementation which uses the provided dirSnapshot. Generates a new one if needed and the provided one is null.

Parameters:

fileSystem - optional, it it is not provided, it will be derived from the candidateDirectory

candidateDirectory - the partition directory to analyze

conf - the configuration

writeIdList - the list of write ids that we are reading

useFileIds - It will be set to true, if the FileSystem supports listing with fileIds

ignoreEmptyFiles - Ignore files with 0 length

dirSnapshots - The listed directory snapshot, if null new will be generated

Returns:

the state of the directory

Throws:

IOException - on filesystem errors

getHdfsDirSnapshotsForCleaner

public static Map<org.apache.hadoop.fs.Path,AcidUtils.HdfsDirSnapshot> getHdfsDirSnapshotsForCleaner(org.apache.hadoop.fs.FileSystem fs,
                                                                                                     org.apache.hadoop.fs.Path path)
                                                                                              throws IOException

In case of the cleaner, we don't need to go into file level, it is enough to collect base/delta/deletedelta directories.

Parameters:

fs - the filesystem used for the directory lookup

path - the path of the table or partition needs to be cleaned

Returns:

The listed directory snapshot needs to be checked for cleaning

Throws:

IOException - on filesystem errors

getHdfsDirSnapshots

public static Map<org.apache.hadoop.fs.Path,AcidUtils.HdfsDirSnapshot> getHdfsDirSnapshots(org.apache.hadoop.fs.FileSystem fs,
                                                                                           org.apache.hadoop.fs.Path path)
                                                                                    throws IOException

Throws:

IOException

isChildOfDelta

public static boolean isChildOfDelta(org.apache.hadoop.fs.Path childDir,
                                     org.apache.hadoop.fs.Path rootPath)

isTablePropertyTransactional

public static boolean isTablePropertyTransactional(Properties props)

isTablePropertyTransactional

public static boolean isTablePropertyTransactional(Map<String,String> parameters)

isDeleteDelta

public static boolean isDeleteDelta(org.apache.hadoop.fs.Path p)

Parameters:

p - - not null

isInsertDelta

public static boolean isInsertDelta(org.apache.hadoop.fs.Path p)

isTransactionalTable

public static boolean isTransactionalTable(CreateTableDesc table)

isTransactionalTable

public static boolean isTransactionalTable(Table table)

isTransactionalTable

public static boolean isTransactionalTable(org.apache.hadoop.hive.metastore.api.Table table)

isTransactionalTable

public static boolean isTransactionalTable(Map<String,String> props)

isTransactionalView

public static boolean isTransactionalView(CreateMaterializedViewDesc view)

isFullAcidTable

public static boolean isFullAcidTable(CreateTableDesc td)

isFullAcidTable

public static boolean isFullAcidTable(Table table)

Should produce the same result as TxnUtils.isAcidTable(org.apache.hadoop.hive.metastore.api.Table)

isFullAcidTable

public static boolean isFullAcidTable(org.apache.hadoop.hive.metastore.api.Table table)

Should produce the same result as TxnUtils.isAcidTable(org.apache.hadoop.hive.metastore.api.Table)

isFullAcidTable

public static boolean isFullAcidTable(Map<String,String> params)

isFullAcidScan

public static boolean isFullAcidScan(org.apache.hadoop.conf.Configuration conf)

isInsertOnlyFetchBucketId

public static boolean isInsertOnlyFetchBucketId(org.apache.hadoop.conf.Configuration conf)

setAcidOperationalProperties

public static void setAcidOperationalProperties(org.apache.hadoop.conf.Configuration conf,
                                                boolean isTxnTable,
                                                AcidUtils.AcidOperationalProperties properties)

Sets the acidOperationalProperties in the configuration object argument.

Parameters:

conf - Mutable configuration object

properties - An acidOperationalProperties object to initialize from. If this is null, we assume this is a full transactional table.

setAcidOperationalProperties

public static void setAcidOperationalProperties(Map<String,String> parameters,
                                                boolean isTxnTable,
                                                AcidUtils.AcidOperationalProperties properties)

Sets the acidOperationalProperties in the map object argument.

Parameters:

parameters - Mutable map object

properties - An acidOperationalProperties object to initialize from.

getAcidOperationalProperties

public static AcidUtils.AcidOperationalProperties getAcidOperationalProperties(Table table)

Returns the acidOperationalProperties for a given table.

Parameters:

table - A table object

Returns:

the acidOperationalProperties object for the corresponding table.

getAcidOperationalProperties

public static AcidUtils.AcidOperationalProperties getAcidOperationalProperties(org.apache.hadoop.conf.Configuration conf)

Returns the acidOperationalProperties for a given configuration.

Parameters:

conf - A configuration object

Returns:

the acidOperationalProperties object for the corresponding configuration.

getAcidOperationalProperties

public static AcidUtils.AcidOperationalProperties getAcidOperationalProperties(Properties props)

Returns the acidOperationalProperties for a given set of properties.

Parameters:

props - A properties object

Returns:

the acidOperationalProperties object for the corresponding properties.

getAcidOperationalProperties

public static AcidUtils.AcidOperationalProperties getAcidOperationalProperties(Map<String,String> parameters)

Returns the acidOperationalProperties for a given map.

Parameters:

parameters - A parameters object

Returns:

the acidOperationalProperties object for the corresponding map.

getLogicalLength

public static long getLogicalLength(org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.FileStatus file)
                             throws IOException

See comments at DELTA_SIDE_FILE_SUFFIX. Returns the logical end of file for an acid data file. This relies on the fact that if delta_x_y has no committed transactions it wil be filtered out by getAcidState(FileSystem, Path, Configuration, ValidWriteIdList, Ref, boolean) and so won't be read at all.

Parameters:

file - - data file to read/compute splits on

Throws:

IOException

isInsertOnlyTable

public static boolean isInsertOnlyTable(Map<String,String> params)

Checks if a table is a transactional table that only supports INSERT, but not UPDATE/DELETE

Parameters:

params - table properties

Returns:

true if table is an INSERT_ONLY table, false otherwise

isInsertOnlyTable

public static boolean isInsertOnlyTable(Table table)

isInsertOnlyTable

public static boolean isInsertOnlyTable(Properties params)

isToInsertOnlyTable

public static Boolean isToInsertOnlyTable(Table tbl,
                                          Map<String,String> props)

The method for altering table props; may set the table to MM, non-MM, or not affect MM. todo: All such validation logic should be TransactionValidationListener

Parameters:

tbl - object image before alter table command (or null if not retrieved yet).

props - prop values set in this alter table command

isToFullAcid

public static Boolean isToFullAcid(Table table,
                                   Map<String,String> props)

canBeMadeAcid

public static boolean canBeMadeAcid(String fullTableName,
                                    org.apache.hadoop.hive.metastore.api.StorageDescriptor sd)

isRemovedInsertOnlyTable

public static boolean isRemovedInsertOnlyTable(Set<String> removedSet)

getValidTxnWriteIdList

public static ValidTxnWriteIdList getValidTxnWriteIdList(org.apache.hadoop.conf.Configuration conf)

Get the ValidTxnWriteIdList saved in the configuration.

getTableValidWriteIdList

public static ValidWriteIdList getTableValidWriteIdList(org.apache.hadoop.conf.Configuration conf,
                                                        String fullTableName)

Extract the ValidWriteIdList for the given table from the list of tables' ValidWriteIdList.

setValidWriteIdList

public static void setValidWriteIdList(org.apache.hadoop.conf.Configuration conf,
                                       ValidWriteIdList validWriteIds)

Set the valid write id list for the current table scan.

setValidWriteIdList

public static void setValidWriteIdList(org.apache.hadoop.conf.Configuration conf,
                                       TableScanDesc tsDesc)

Set the valid write id list for the current table scan.

getTableSnapshot

public static AcidUtils.TableSnapshot getTableSnapshot(org.apache.hadoop.conf.Configuration conf,
                                                       Table tbl)
                                                throws LockException

Throws:

LockException

getTableSnapshot

public static AcidUtils.TableSnapshot getTableSnapshot(org.apache.hadoop.conf.Configuration conf,
                                                       Table tbl,
                                                       boolean isStatsUpdater)
                                                throws LockException

Note: this is generally called in Hive.java; so, the callers of Hive.java make sure to set up the acid state during compile, and Hive.java retrieves it if needed.

Throws:

LockException

getTableSnapshot

public static AcidUtils.TableSnapshot getTableSnapshot(org.apache.hadoop.conf.Configuration conf,
                                                       Table tbl,
                                                       String dbName,
                                                       String tblName,
                                                       boolean isStatsUpdater)
                                                throws LockException,
                                                       AssertionError

Note: this is generally called in Hive.java; so, the callers of Hive.java make sure to set up the acid state during compile, and Hive.java retrieves it if needed.

Throws:

LockException

AssertionError

getTableValidWriteIdListWithTxnList

public static ValidWriteIdList getTableValidWriteIdListWithTxnList(org.apache.hadoop.conf.Configuration conf,
                                                                   String dbName,
                                                                   String tableName)
                                                            throws LockException

Returns ValidWriteIdList for the table with the given "dbName" and "tableName". This is called when HiveConf has no list for the table. Otherwise use getTableSnapshot().

Parameters:

conf - Configuration

dbName -

tableName -

Returns:

ValidWriteIdList on success, null on failure to get a list.

Throws:

LockException

getFullTableName

public static String getFullTableName(String dbName,
                                      String tableName)

getAcidFilesForStats

public static List<org.apache.hadoop.fs.FileStatus> getAcidFilesForStats(Table table,
                                                                         org.apache.hadoop.fs.Path dir,
                                                                         org.apache.hadoop.conf.Configuration jc,
                                                                         org.apache.hadoop.fs.FileSystem fs)
                                                                  throws IOException

Throws:

IOException

getValidDataPaths

public static List<org.apache.hadoop.fs.Path> getValidDataPaths(org.apache.hadoop.fs.Path dataPath,
                                                                org.apache.hadoop.conf.Configuration conf,
                                                                String validWriteIdStr)
                                                         throws IOException

Throws:

IOException

getAcidSubDir

public static String getAcidSubDir(org.apache.hadoop.fs.Path dataPath)

getFirstLevelAcidDirPath

public static String getFirstLevelAcidDirPath(org.apache.hadoop.fs.Path dataPath,
                                              org.apache.hadoop.fs.FileSystem fileSystem)
                                       throws IOException

Throws:

IOException

isAcidEnabled

public static boolean isAcidEnabled(HiveConf hiveConf)

extractWriteId

public static Long extractWriteId(org.apache.hadoop.fs.Path file)

setNonTransactional

public static void setNonTransactional(Map<String,String> tblProps)

makeLockComponents

public static List<org.apache.hadoop.hive.metastore.api.LockComponent> makeLockComponents(Set<WriteEntity> outputs,
                                                                                          Set<ReadEntity> inputs,
                                                                                          Context.Operation operation,
                                                                                          HiveConf conf)

Create lock components from write/read entities.

Parameters:

outputs - write entities

inputs - read entities

conf -

Returns:

list with lock components

isExclusiveCTASEnabled

public static boolean isExclusiveCTASEnabled(org.apache.hadoop.conf.Configuration conf)

isExclusiveCTAS

public static boolean isExclusiveCTAS(Set<WriteEntity> outputs,
                                      HiveConf conf)

validateAcidFiles

public static void validateAcidFiles(Table table,
                                     org.apache.hadoop.fs.FileStatus[] srcs,
                                     org.apache.hadoop.fs.FileSystem fs)
                              throws SemanticException

Safety check to make sure a file take from one acid table is not added into another acid table since the ROW__IDs embedded as part a write to one table won't make sense in different table/cluster.

Throws:

SemanticException

validateAcidPartitionLocation

public static void validateAcidPartitionLocation(String location,
                                                 org.apache.hadoop.conf.Configuration conf)
                                          throws SemanticException

Safety check to make sure the given location is not the location of acid table and all it's files will be not added into another acid table

Throws:

SemanticException

getTxnType

public static org.apache.hadoop.hive.metastore.api.TxnType getTxnType(org.apache.hadoop.conf.Configuration conf,
                                                                      ASTNode tree)

Determines transaction type based on query AST.

Parameters:

tree - AST

getPathSuffix

public static String getPathSuffix(long txnId)

initDirCache

public static void initDirCache(int durationInMts)

getAcidStateFromCache

public static AcidDirectory getAcidStateFromCache(Supplier<org.apache.hadoop.fs.FileSystem> fileSystem,
                                                  org.apache.hadoop.fs.Path candidateDirectory,
                                                  org.apache.hadoop.conf.Configuration conf,
                                                  ValidWriteIdList writeIdList,
                                                  Ref<Boolean> useFileIds,
                                                  boolean ignoreEmptyFiles)
                                           throws IOException

Tries to get directory details from cache. For now, cache is valid only when base directory is available and no deltas are present. This should be used only in BI strategy and for ACID tables.

Parameters:

fileSystem - file system supplier

candidateDirectory - the partition directory to analyze

conf - the configuration

writeIdList - the list of write ids that we are reading

useFileIds - It will be set to true, if the FileSystem supports listing with fileIds

ignoreEmptyFiles - Ignore files with 0 length

Returns:

directory state

Throws:

IOException - on errors

tryInvalidateDirCache

public static void tryInvalidateDirCache(org.apache.hadoop.hive.metastore.api.Table table)

isNonNativeAcidTable

public static boolean isNonNativeAcidTable(Table table)

getAcidVirtualColumns

public static List<VirtualColumn> getAcidVirtualColumns(Table table)

Returns the virtual columns needed for update queries. For ACID queries it is a single ROW__ID, for non-native tables the list is provided by the HiveStorageHandler.acidVirtualColumns().

Parameters:

table - The table for which we run the query

Returns:

The list of virtual columns used

acidTableWithoutTransactions

public static boolean acidTableWithoutTransactions(Table table)

getPartitionName

public static String getPartitionName(Map<String,String> partitionSpec)
                               throws SemanticException

Throws:

SemanticException

compactionTypeStr2ThriftType

public static org.apache.hadoop.hive.metastore.api.CompactionType compactionTypeStr2ThriftType(String inputValue)
                                                                                        throws SemanticException

Throws:

SemanticException

compactionStateStr2Enum

public static CompactionState compactionStateStr2Enum(String inputValue)
                                               throws SemanticException

Throws:

SemanticException