com.google.cloud.spark.bigquery.write.context

Class BigQueryDirectDataSourceWriterContext

java.lang.Object

com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataSourceWriterContext

All Implemented Interfaces:

DataSourceWriterContext

public class BigQueryDirectDataSourceWriterContext
extends Object
implements DataSourceWriterContext

Constructor Summary

Constructors

Constructor and Description

BigQueryDirectDataSourceWriterContext(BigQueryClient bigQueryClient, BigQueryClientFactory bigQueryWriteClientFactory, com.google.cloud.bigquery.TableId destinationTableId, String writeUUID, org.apache.spark.sql.SaveMode saveMode, org.apache.spark.sql.types.StructType sparkSchema, com.google.api.gax.retrying.RetrySettings bigqueryDataWriterHelperRetrySettings, com.google.common.base.Optional<String> traceId, boolean enableModeCheckForSchemaFields, com.google.common.collect.ImmutableMap<String,String> tableLabels, SchemaConvertersConfiguration schemaConvertersConfiguration, Optional<String> destinationTableKmsKeyName, boolean writeAtLeastOnce, PartitionOverwriteMode overwriteMode, org.apache.spark.SparkContext sparkContext)

Method Summary

Modifier and Type	Method and Description
void	abort(WriterCommitMessageContext[] messages) If not in WritingMode IGNORE_INPUTS, the BigQuery Storage Write API WriteClient is shut down.
void	commit(WriterCommitMessageContext[] messages) This function will determine, based on the WritingMode: if in IGNORE_INPUTS mode, no work is to be done; otherwise all streams will be batch committed using the BigQuery Storage Write API, and then: if in OVERWRITE mode, the overwriteDestinationWithTemporary function from BigQueryClient will be called to replace the destination table with all the data from the temporary table; if in OVERWRITE mode with dynamic partitions enabled, overwriteDestinationWithTemporaryDynamicPartitons from BigQueryClient will be called to replace the required partitions;if in ALL_ELSE mode no more work needs to be done.
DataWriterContextFactory<org.apache.spark.sql.catalyst.InternalRow>	createWriterContextFactory()
void	onDataWriterCommit(WriterCommitMessageContext message)
void	setTableInfo(com.google.cloud.bigquery.TableInfo tableInfo)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.google.cloud.spark.bigquery.write.context.DataSourceWriterContext

create, useCommitCoordinator

Constructor Detail

BigQueryDirectDataSourceWriterContext

public BigQueryDirectDataSourceWriterContext(BigQueryClient bigQueryClient,
                                             BigQueryClientFactory bigQueryWriteClientFactory,
                                             com.google.cloud.bigquery.TableId destinationTableId,
                                             String writeUUID,
                                             org.apache.spark.sql.SaveMode saveMode,
                                             org.apache.spark.sql.types.StructType sparkSchema,
                                             com.google.api.gax.retrying.RetrySettings bigqueryDataWriterHelperRetrySettings,
                                             com.google.common.base.Optional<String> traceId,
                                             boolean enableModeCheckForSchemaFields,
                                             com.google.common.collect.ImmutableMap<String,String> tableLabels,
                                             SchemaConvertersConfiguration schemaConvertersConfiguration,
                                             Optional<String> destinationTableKmsKeyName,
                                             boolean writeAtLeastOnce,
                                             PartitionOverwriteMode overwriteMode,
                                             org.apache.spark.SparkContext sparkContext)
                                      throws IllegalArgumentException

Throws:

IllegalArgumentException

Method Detail

createWriterContextFactory

public DataWriterContextFactory<org.apache.spark.sql.catalyst.InternalRow> createWriterContextFactory()

Specified by:

createWriterContextFactory in interface DataSourceWriterContext

onDataWriterCommit

public void onDataWriterCommit(WriterCommitMessageContext message)

Specified by:

onDataWriterCommit in interface DataSourceWriterContext

commit

public void commit(WriterCommitMessageContext[] messages)

This function will determine, based on the WritingMode: if in IGNORE_INPUTS mode, no work is to be done; otherwise all streams will be batch committed using the BigQuery Storage Write API, and then: if in OVERWRITE mode, the overwriteDestinationWithTemporary function from BigQueryClient will be called to replace the destination table with all the data from the temporary table; if in OVERWRITE mode with dynamic partitions enabled, overwriteDestinationWithTemporaryDynamicPartitons from BigQueryClient will be called to replace the required partitions;if in ALL_ELSE mode no more work needs to be done.

Specified by:

commit in interface DataSourceWriterContext

Parameters:

messages - the BigQueryWriterCommitMessage array returned by the BigQueryDataWriter's.

See Also:

WritingMode, BigQueryClient.overwriteDestinationWithTemporary(TableId temporaryTableId, TableId destinationTableId)

abort

public void abort(WriterCommitMessageContext[] messages)

If not in WritingMode IGNORE_INPUTS, the BigQuery Storage Write API WriteClient is shut down.

Specified by:

abort in interface DataSourceWriterContext

Parameters:

messages - the BigQueryWriterCommitMessage array returned by the BigQueryDataWriter's.

See Also:

BigQueryWriteClient

setTableInfo

public void setTableInfo(com.google.cloud.bigquery.TableInfo tableInfo)

Specified by:

setTableInfo in interface DataSourceWriterContext