org.apache.orc.impl

类 WriterImpl

java.lang.Object

org.apache.orc.impl.WriterImpl

所有已实现的接口:

Closeable, AutoCloseable, org.apache.orc.impl.WriterInternal, org.apache.orc.MemoryManager.Callback, org.apache.orc.Writer

public class WriterImpl
extends Object
implements org.apache.orc.impl.WriterInternal, org.apache.orc.MemoryManager.Callback

An ORC file writer. The file is divided into stripes, which is the natural unit of work when reading. Each stripe is buffered in memory until the memory reaches the stripe size and then it is written out broken down by columns. Each column is written by a TreeWriter that is specific to that type of column. TreeWriters may have children TreeWriters that handle the sub-types. Each of the TreeWriters writes the column's data as a set of streams.

This class is unsynchronized like most Stream objects, so from the creation of an OrcFile and all access to a single instance has to be from a single thread.

There are no known cases where these happen between different threads today.

Caveat: the MemoryManager is created during WriterOptions create, that has to be confined to a single thread as well.

NOTE: The file was copied and modified to support zstd-jni. This feature is only supported in ORC 2.0, but 2.0 only supports JDK17. We need to support JDK8.

构造器概要

构造器

构造器和说明
WriterImpl(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, OrcFile.WriterOptions opts)

方法概要

限定符和类型	方法和说明
void	addRowBatch(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch batch)
void	addUserMetadata(String name, ByteBuffer value)
void	appendStripe(byte[] stripe, int offset, int length, org.apache.orc.StripeInformation stripeInfo, org.apache.orc.OrcProto.StripeStatistics stripeStatistics)
void	appendStripe(byte[] stripe, int offset, int length, org.apache.orc.StripeInformation stripeInfo, org.apache.orc.StripeStatistics[] stripeStatistics)
void	appendUserMetadata(List<org.apache.orc.OrcProto.UserMetadataItem> userMetadata)
boolean	checkMemory(double newScale)
void	close()
static org.apache.orc.CompressionCodec	createCodec(CompressionKind kind)
long	estimateMemory()
org.apache.orc.CompressionCodec	getCompressionCodec()
static int	getEstimatedBufferSize(long stripeSize, int numColumns, int bs)
long	getNumberOfRows() Row count gets updated when flushing the stripes.
long	getRawDataSize() Raw data size will be compute when writing the file footer.
org.apache.orc.TypeDescription	getSchema()
org.apache.orc.ColumnStatistics[]	getStatistics()
List<org.apache.orc.StripeInformation>	getStripes()
void	increaseCompressionSize(int newSize)
long	writeIntermediateFooter()

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

WriterImpl

public WriterImpl(org.apache.hadoop.fs.FileSystem fs,
                  org.apache.hadoop.fs.Path path,
                  OrcFile.WriterOptions opts)
           throws IOException

抛出:

IOException

方法详细资料

getEstimatedBufferSize

public static int getEstimatedBufferSize(long stripeSize,
                                         int numColumns,
                                         int bs)

increaseCompressionSize

public void increaseCompressionSize(int newSize)

指定者:

increaseCompressionSize 在接口中 org.apache.orc.impl.WriterInternal

createCodec

public static org.apache.orc.CompressionCodec createCodec(CompressionKind kind)

checkMemory

public boolean checkMemory(double newScale)
                    throws IOException

指定者:

checkMemory 在接口中 org.apache.orc.MemoryManager.Callback

抛出:

IOException

getSchema

public org.apache.orc.TypeDescription getSchema()

指定者:

getSchema 在接口中 org.apache.orc.Writer

addUserMetadata

public void addUserMetadata(String name,
                            ByteBuffer value)

指定者:

addUserMetadata 在接口中 org.apache.orc.Writer

addRowBatch

public void addRowBatch(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch batch)
                 throws IOException

指定者:

addRowBatch 在接口中 org.apache.orc.Writer

抛出:

IOException

close

public void close()
           throws IOException

指定者:

close 在接口中 Closeable

指定者:

close 在接口中 AutoCloseable

指定者:

close 在接口中 org.apache.orc.Writer

抛出:

IOException

getRawDataSize

public long getRawDataSize()

Raw data size will be compute when writing the file footer. Hence raw data size value will be available only after closing the writer.

指定者:

getRawDataSize 在接口中 org.apache.orc.Writer

getNumberOfRows

public long getNumberOfRows()

Row count gets updated when flushing the stripes. To get accurate row count call this method after writer is closed.

指定者:

getNumberOfRows 在接口中 org.apache.orc.Writer

writeIntermediateFooter

public long writeIntermediateFooter()
                             throws IOException

指定者:

writeIntermediateFooter 在接口中 org.apache.orc.Writer

抛出:

IOException

appendStripe

public void appendStripe(byte[] stripe,
                         int offset,
                         int length,
                         org.apache.orc.StripeInformation stripeInfo,
                         org.apache.orc.OrcProto.StripeStatistics stripeStatistics)
                  throws IOException

指定者:

appendStripe 在接口中 org.apache.orc.Writer

抛出:

IOException

appendStripe

public void appendStripe(byte[] stripe,
                         int offset,
                         int length,
                         org.apache.orc.StripeInformation stripeInfo,
                         org.apache.orc.StripeStatistics[] stripeStatistics)
                  throws IOException

指定者:

appendStripe 在接口中 org.apache.orc.Writer

抛出:

IOException

appendUserMetadata

public void appendUserMetadata(List<org.apache.orc.OrcProto.UserMetadataItem> userMetadata)

指定者:

appendUserMetadata 在接口中 org.apache.orc.Writer

getStatistics

public org.apache.orc.ColumnStatistics[] getStatistics()

指定者:

getStatistics 在接口中 org.apache.orc.Writer

getStripes

public List<org.apache.orc.StripeInformation> getStripes()
                                                  throws IOException

指定者:

getStripes 在接口中 org.apache.orc.Writer

抛出:

IOException

getCompressionCodec

public org.apache.orc.CompressionCodec getCompressionCodec()

estimateMemory

public long estimateMemory()

指定者:

estimateMemory 在接口中 org.apache.orc.Writer