class SingleDirectoryDataWriter extends FileFormatDataWriter
Writes data to a single directory (used for non-dynamic-partition writes).
- Alphabetic
- By Inheritance
- SingleDirectoryDataWriter
- FileFormatDataWriter
- DataWriter
- Closeable
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new SingleDirectoryDataWriter(description: WriteJobDescription, taskAttemptContext: TaskAttemptContext, committer: FileCommitProtocol, customMetrics: Map[String, SQLMetric] = Map.empty)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val MAX_FILE_COUNTER: Int
Max number of files a single task writes out due to file size.
Max number of files a single task writes out due to file size. In most cases the number of files written should be very small. This is just a safe guard to protect some really bad settings, e.g. maxRecordsPerFile = 1.
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
- def abort(): Unit
- Definition Classes
- FileFormatDataWriter → DataWriter
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def close(): Unit
- Definition Classes
- FileFormatDataWriter → Closeable → AutoCloseable
- def commit(): WriteTaskResult
Returns the summary of relative information which includes the list of partition strings written out.
Returns the summary of relative information which includes the list of partition strings written out. The list of partitions is sent back to the driver and used to update the catalog. Other information will be sent back to the driver too and used to e.g. update the metrics in UI.
- Definition Classes
- FileFormatDataWriter → DataWriter
- def currentMetricsValues(): Array[CustomTaskMetric]
- Definition Classes
- DataWriter
- var currentWriter: OutputWriter
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def releaseCurrentWriter(): Unit
Release resources of
currentWriter.Release resources of
currentWriter.- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
- def releaseResources(): Unit
Release all resources.
Release all resources.
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
- val statsTrackers: Seq[WriteTaskStatsTracker]
Trackers for computing various statistics on the data as it's being written out.
Trackers for computing various statistics on the data as it's being written out.
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- val updatedPartitions: Set[String]
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- def write(record: InternalRow): Unit
Writes a record.
Writes a record.
- Definition Classes
- SingleDirectoryDataWriter → FileFormatDataWriter → DataWriter
- def writeWithIterator(iterator: Iterator[InternalRow]): Unit
Write an iterator of records.
Write an iterator of records.
- Definition Classes
- FileFormatDataWriter
- def writeWithMetrics(record: InternalRow, count: Long): Unit
- Definition Classes
- FileFormatDataWriter