class GpuMultiFileCloudAvroPartitionReader extends MultiFileCloudPartitionReaderBase with MultiFileReaderFunctions with GpuAvroReaderBase
A PartitionReader that can read multiple AVRO files in parallel. This is most efficient running in a cloud environment where the I/O of reading is slow.
When reading a file, it
- seeks to the start position of the first block located in this partition.
- next, parses the meta and sync, rewrites the meta and sync, and copies the data to a batch buffer per block, until reaching the last one of the current partition.
- sends batches to GPU at last.
- Alphabetic
- By Inheritance
- GpuMultiFileCloudAvroPartitionReader
- GpuAvroReaderBase
- MultiFileReaderFunctions
- MultiFileCloudPartitionReaderBase
- FilePartitionReaderBase
- ScanWithMetrics
- Logging
- PartitionReader
- Closeable
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
GpuMultiFileCloudAvroPartitionReader(conf: Configuration, files: Array[PartitionedFile], numThreads: Int, maxNumFileProcessed: Int, filters: Array[Filter], execMetrics: Map[String, GpuMetric], ignoreCorruptFiles: Boolean, ignoreMissingFiles: Boolean, debugDumpPrefix: Option[String], debugDumpAlways: Boolean, readDataSchema: StructType, partitionSchema: StructType, maxReadBatchSizeRows: Integer, maxReadBatchSizeBytes: Long, maxGpuColumnSizeBytes: Long)
- conf
the Hadoop configuration
- files
the partitioned files to read
- numThreads
the size of the threadpool
- maxNumFileProcessed
threshold to control the maximum file number to be submitted to threadpool
- filters
filters passed into the filterHandler
- execMetrics
the metrics
- ignoreCorruptFiles
Whether to ignore corrupt files
- ignoreMissingFiles
Whether to ignore missing files
- debugDumpPrefix
a path prefix to use for dumping the fabricated AVRO data or null
- debugDumpAlways
whether to debug dump always or only on errors
- readDataSchema
the Spark schema describing what will be read
- partitionSchema
Schema of partitions.
- maxReadBatchSizeRows
soft limit on the maximum number of rows to be read per batch
- maxReadBatchSizeBytes
soft limit on the maximum number of bytes to be read per batch
- maxGpuColumnSizeBytes
maximum number of bytes for a GPU column
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
var
batchIter: Iterator[ColumnarBatch]
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
val
cacheBufferSize: Int
- Definition Classes
- GpuAvroReaderBase
-
def
canUseCombine: Boolean
- Definition Classes
- MultiFileCloudPartitionReaderBase
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
close(): Unit
- Definition Classes
- MultiFileCloudPartitionReaderBase → FilePartitionReaderBase → Closeable → AutoCloseable
-
def
combineHMBs(results: Array[HostMemoryBuffersWithMetaDataBase]): HostMemoryBuffersWithMetaDataBase
- Definition Classes
- MultiFileCloudPartitionReaderBase
-
var
combineLeftOverFiles: Option[Array[HostMemoryBuffersWithMetaDataBase]]
- Attributes
- protected
- Definition Classes
- MultiFileCloudPartitionReaderBase
-
val
conf: Configuration
- Definition Classes
- GpuMultiFileCloudAvroPartitionReader → GpuAvroReaderBase
-
final
def
copyBlocksData(blocks: Seq[BlockInfo], in: FSDataInputStream, out: OutputStream, sync: Option[Array[Byte]] = None): Seq[BlockInfo]
Copy the data specified by the blocks from
intooutCopy the data specified by the blocks from
intoout- Attributes
- protected
- Definition Classes
- GpuAvroReaderBase
-
var
currentFileHostBuffers: Option[HostMemoryBuffersWithMetaDataBase]
- Attributes
- protected
- Definition Classes
- MultiFileCloudPartitionReaderBase
-
def
currentMetricsValues(): Array[CustomTaskMetric]
- Definition Classes
- PartitionReader
-
val
debugDumpAlways: Boolean
- Definition Classes
- GpuMultiFileCloudAvroPartitionReader → GpuAvroReaderBase
-
val
debugDumpPrefix: Option[String]
- Definition Classes
- GpuMultiFileCloudAvroPartitionReader → GpuAvroReaderBase
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
estimateOutputSize(blocks: Seq[BlockInfo], headerSize: Long): Long
Estimate the total size from the given blocks and header
Estimate the total size from the given blocks and header
- Attributes
- protected
- Definition Classes
- GpuAvroReaderBase
-
def
fileSystemBytesRead(): Long
- Attributes
- protected
- Definition Classes
- MultiFileReaderFunctions
- Annotations
- @nowarn()
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
get(): ColumnarBatch
- Definition Classes
- FilePartitionReaderBase → PartitionReader
-
def
getBatchRunner(tc: TaskContext, file: PartitionedFile, origFile: Option[PartitionedFile], config: Configuration, filters: Array[Filter]): Callable[HostMemoryBuffersWithMetaDataBase]
The sub-class must implement the real file reading logic in a Callable which will be running in a thread pool
The sub-class must implement the real file reading logic in a Callable which will be running in a thread pool
- tc
task context to use
- file
file to be read
- origFile
optional original unmodified file if replaced with Alluxio
- filters
push down filters
- returns
Callable[HostMemoryBuffersWithMetaDataBase]
- Definition Classes
- GpuMultiFileCloudAvroPartitionReader → MultiFileCloudPartitionReaderBase
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
getFileFormatShortName: String
File format short name used for logging and other things to uniquely identity which file format is being used.
File format short name used for logging and other things to uniquely identity which file format is being used.
- returns
the file format short name
- Definition Classes
- GpuMultiFileCloudAvroPartitionReader → MultiFileCloudPartitionReaderBase
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
var
isDone: Boolean
- Attributes
- protected
- Definition Classes
- FilePartitionReaderBase
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
metrics: Map[String, GpuMetric]
- Definition Classes
- ScanWithMetrics
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
next(): Boolean
- Definition Classes
- MultiFileCloudPartitionReaderBase → PartitionReader
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
populateCurrentBlockChunk(blockIter: BufferedIterator[BlockInfo], maxReadBatchSizeRows: Int, maxReadBatchSizeBytes: Long): Seq[BlockInfo]
Get the block chunk according to the max batch size and max rows.
Get the block chunk according to the max batch size and max rows.
- blockIter
blocks to be evaluated
- maxReadBatchSizeRows
soft limit on the maximum number of rows the reader reads per batch
- maxReadBatchSizeBytes
soft limit on the maximum number of bytes the reader reads per batch
- Attributes
- protected
- Definition Classes
- GpuAvroReaderBase
-
def
readBatches(fileBufsAndMeta: HostMemoryBuffersWithMetaDataBase): Iterator[ColumnarBatch]
Decode HostMemoryBuffers in GPU
Decode HostMemoryBuffers in GPU
- fileBufsAndMeta
the file HostMemoryBuffer read from a PartitionedFile
- returns
an iterator of batches that were decoded
- Definition Classes
- GpuMultiFileCloudAvroPartitionReader → MultiFileCloudPartitionReaderBase
-
val
readDataSchema: StructType
- Definition Classes
- GpuMultiFileCloudAvroPartitionReader → GpuAvroReaderBase
-
final
def
readPartFile(partFilePath: Path, blocks: Seq[BlockInfo], headerSize: Long, conf: Configuration): (HostMemoryBuffer, Long)
Read a split into a host buffer, preparing for sending to GPU
Read a split into a host buffer, preparing for sending to GPU
- Attributes
- protected
- Definition Classes
- GpuAvroReaderBase
-
final
def
sendToGpu(hostBuf: HostMemoryBuffer, bufSize: Long, splits: Array[PartitionedFile]): Option[ColumnarBatch]
Send a host buffer to GPU for decoding, and return it as a ColumnarBatch.
Send a host buffer to GPU for decoding, and return it as a ColumnarBatch. The input hostBuf will be closed after returning, please do not use it anymore. 'splits' is used only for debugging.
- Attributes
- protected
- Definition Classes
- GpuAvroReaderBase
-
final
def
sendToGpuUnchecked(hostBuf: HostMemoryBuffer, bufSize: Long, splits: Array[PartitionedFile]): Table
Read the host data to GPU for decoding, and return it as a cuDF Table.
Read the host data to GPU for decoding, and return it as a cuDF Table. The input host buffer should contain valid data, otherwise the behavior is undefined. 'splits' is used only for debugging.
- Attributes
- protected
- Definition Classes
- GpuAvroReaderBase
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()