class SingleDirectoryDataWriter extends FileFormatDataWriter
Writes data to a single directory (used for non-dynamic-partition writes).
- Alphabetic
- By Inheritance
- SingleDirectoryDataWriter
- FileFormatDataWriter
- DataWriter
- Closeable
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new SingleDirectoryDataWriter(description: WriteJobDescription, taskAttemptContext: TaskAttemptContext, committer: FileCommitProtocol, customMetrics: Map[String, SQLMetric] = Map.empty)
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
val
MAX_FILE_COUNTER: Int
Max number of files a single task writes out due to file size.
Max number of files a single task writes out due to file size. In most cases the number of files written should be very small. This is just a safe guard to protect some really bad settings, e.g. maxRecordsPerFile = 1.
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
-
def
abort(): Unit
- Definition Classes
- FileFormatDataWriter → DataWriter
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
close(): Unit
- Definition Classes
- FileFormatDataWriter → Closeable → AutoCloseable
-
def
commit(): WriteTaskResult
Returns the summary of relative information which includes the list of partition strings written out.
Returns the summary of relative information which includes the list of partition strings written out. The list of partitions is sent back to the driver and used to update the catalog. Other information will be sent back to the driver too and used to e.g. update the metrics in UI.
- Definition Classes
- FileFormatDataWriter → DataWriter
-
def
currentMetricsValues(): Array[CustomTaskMetric]
- Definition Classes
- DataWriter
-
var
currentWriter: OutputWriter
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
releaseCurrentWriter(): Unit
Release resources of
currentWriter.Release resources of
currentWriter.- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
-
def
releaseResources(): Unit
Release all resources.
Release all resources.
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
-
val
statsTrackers: Seq[WriteTaskStatsTracker]
Trackers for computing various statistics on the data as it's being written out.
Trackers for computing various statistics on the data as it's being written out.
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
val
updatedPartitions: Set[String]
- Attributes
- protected
- Definition Classes
- FileFormatDataWriter
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
write(record: InternalRow): Unit
Writes a record.
Writes a record.
- Definition Classes
- SingleDirectoryDataWriter → FileFormatDataWriter → DataWriter
-
def
writeWithIterator(iterator: Iterator[InternalRow]): Unit
Write an iterator of records.
Write an iterator of records.
- Definition Classes
- FileFormatDataWriter
-
def
writeWithMetrics(record: InternalRow, count: Long): Unit
- Definition Classes
- FileFormatDataWriter