org.apache.hadoop.hive.ql.exec

Class FileSinkOperator.FSPaths

java.lang.Object

org.apache.hadoop.hive.ql.exec.FileSinkOperator.FSPaths

All Implemented Interfaces:

Cloneable

Enclosing class:

FileSinkOperator

public class FileSinkOperator.FSPaths
extends Object
implements Cloneable

Constructor Summary

Constructors

Constructor and Description
FSPaths(org.apache.hadoop.fs.Path specPath, boolean isMmTable, boolean isDirectInsert, boolean isInsertOverwrite, AcidUtils.Operation acidOperation)

Method Summary

Modifier and Type	Method and Description
void	abortWritersAndUpdaters(org.apache.hadoop.fs.FileSystem fs, boolean abort, boolean delete)
void	addToStat(String statType, long amount)
org.apache.hadoop.fs.Path	buildTaskOutputTempPath()
org.apache.hadoop.fs.Path	buildTmpPath()
void	closeWriters(boolean abort, List<org.apache.hadoop.fs.Path> deleteDeltas)
int	createDynamicBucket(int bucketNum) This method is intended for use with ACID unbucketed tables, where the DELETE ops behave as though they are bucketed, but without an explicit pre-specified bucket count.
Collection<String>	getStoredStats()
void	initializeBucketPaths(int filesIdx, String taskId, boolean isNativeTable, boolean isSkewedStoredAsSubDirectories)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FSPaths

public FSPaths(org.apache.hadoop.fs.Path specPath,
               boolean isMmTable,
               boolean isDirectInsert,
               boolean isInsertOverwrite,
               AcidUtils.Operation acidOperation)

Method Detail

closeWriters

public void closeWriters(boolean abort,
                         List<org.apache.hadoop.fs.Path> deleteDeltas)
                  throws HiveException

Throws:

HiveException

abortWritersAndUpdaters

public void abortWritersAndUpdaters(org.apache.hadoop.fs.FileSystem fs,
                                    boolean abort,
                                    boolean delete)
                             throws HiveException

Throws:

HiveException

initializeBucketPaths

public void initializeBucketPaths(int filesIdx,
                                  String taskId,
                                  boolean isNativeTable,
                                  boolean isSkewedStoredAsSubDirectories)

buildTmpPath

public org.apache.hadoop.fs.Path buildTmpPath()

buildTaskOutputTempPath

public org.apache.hadoop.fs.Path buildTaskOutputTempPath()

addToStat

public void addToStat(String statType,
                      long amount)

getStoredStats

public Collection<String> getStoredStats()

createDynamicBucket

public int createDynamicBucket(int bucketNum)

This method is intended for use with ACID unbucketed tables, where the DELETE ops behave as though they are bucketed, but without an explicit pre-specified bucket count. The bucketNum is read out of the middle value of the ROW__ID variable and this is written out from a single FileSink, in ways similar to the multi file spray, but without knowing the total number of buckets ahead of time. ROW__ID (1,2[0],3) => bucket_00002 ROW__ID (1,3[0],4) => bucket_00003 etc A new FSP is created for each partition, so this only requires the bucket numbering and that is mapped in directly as an index. This relies on ReduceSinkOperator to shuffle update/delete rows by UDFToInteger(RecordIdentifier), i.e. by writerId in ROW__ID. SemanticAnalyzer.getPartitionColsFromBucketColsForUpdateDelete(Operator, boolean)