com.google.cloud.spark.bigquery.write.context

Class BigQueryIndirectDataSourceWriterContext

java.lang.Object

com.google.cloud.spark.bigquery.write.context.BigQueryIndirectDataSourceWriterContext

All Implemented Interfaces:

DataSourceWriterContext

public class BigQueryIndirectDataSourceWriterContext
extends Object
implements DataSourceWriterContext

A DataSourceWriter implemented by first writing the DataFrame's data into GCS in an intermediate format, and then triggering a BigQuery load job on this data. Hence the "indirect" - the data goes through an intermediate storage.

Constructor Summary

Constructors

Constructor and Description

BigQueryIndirectDataSourceWriterContext(BigQueryClient bigQueryClient, SparkBigQueryConfig config, org.apache.hadoop.conf.Configuration hadoopConfiguration, org.apache.spark.sql.types.StructType sparkSchema, String writeUUID, org.apache.spark.sql.SaveMode saveMode, org.apache.hadoop.fs.Path gcsPath, Optional<IntermediateDataCleaner> intermediateDataCleaner, org.apache.spark.SparkContext sparkContext)

Method Summary

Modifier and Type	Method and Description
void	abort(WriterCommitMessageContext[] messages)
void	commit(WriterCommitMessageContext[] messages)
DataWriterContextFactory<org.apache.spark.sql.catalyst.InternalRow>	createWriterContextFactory()
void	setTableInfo(com.google.cloud.bigquery.TableInfo tableInfo)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.google.cloud.spark.bigquery.write.context.DataSourceWriterContext

create, onDataWriterCommit, useCommitCoordinator

Constructor Detail

BigQueryIndirectDataSourceWriterContext

public BigQueryIndirectDataSourceWriterContext(BigQueryClient bigQueryClient,
                                               SparkBigQueryConfig config,
                                               org.apache.hadoop.conf.Configuration hadoopConfiguration,
                                               org.apache.spark.sql.types.StructType sparkSchema,
                                               String writeUUID,
                                               org.apache.spark.sql.SaveMode saveMode,
                                               org.apache.hadoop.fs.Path gcsPath,
                                               Optional<IntermediateDataCleaner> intermediateDataCleaner,
                                               org.apache.spark.SparkContext sparkContext)

Method Detail

createWriterContextFactory

public DataWriterContextFactory<org.apache.spark.sql.catalyst.InternalRow> createWriterContextFactory()

Specified by:

createWriterContextFactory in interface DataSourceWriterContext

commit

public void commit(WriterCommitMessageContext[] messages)

Specified by:

commit in interface DataSourceWriterContext

abort

public void abort(WriterCommitMessageContext[] messages)

Specified by:

abort in interface DataSourceWriterContext

setTableInfo

public void setTableInfo(com.google.cloud.bigquery.TableInfo tableInfo)

Specified by:

setTableInfo in interface DataSourceWriterContext