com.google.cloud.bigquery.connector.common

Class BigQueryDirectDataWriterHelper

java.lang.Object

com.google.cloud.bigquery.connector.common.BigQueryDirectDataWriterHelper

public class BigQueryDirectDataWriterHelper
extends Object

The class which sketches out the necessary functions in order for a Spark DataWriter to interact with the BigQuery Storage Write API.

Constructor Summary

Constructors

Constructor and Description
BigQueryDirectDataWriterHelper(BigQueryClientFactory writeClientFactory, String tablePath, com.google.cloud.bigquery.storage.v1.ProtoSchema protoSchema, com.google.api.gax.retrying.RetrySettings bigqueryDataWriterHelperRetrySettings, com.google.common.base.Optional<String> traceId, int partitionId, boolean writeAtLeastOnce)

Method Summary

Modifier and Type	Method and Description
void	abort() Deletes the data left over in the protoRows, using method clearProtoRows, closes the StreamWriter, shuts down the WriteClient, and nulls out the protoRows and write-stream-name.
void	addRow(com.google.protobuf.ByteString message) Adds a row to the protoRows, which acts as a buffer; but before, checks if the current message size in bytes will cause the protoRows buffer to exceed the maximum APPEND_REQUEST_SIZE, and if it will, sends an append rows request first.
WriteStreamStatistics	finalizeStream() Appends any data that remains in the protoRows, waits for 500 milliseconds, and finalizes the write-stream.
String	getWriteStreamName()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigQueryDirectDataWriterHelper

public BigQueryDirectDataWriterHelper(BigQueryClientFactory writeClientFactory,
                                      String tablePath,
                                      com.google.cloud.bigquery.storage.v1.ProtoSchema protoSchema,
                                      com.google.api.gax.retrying.RetrySettings bigqueryDataWriterHelperRetrySettings,
                                      com.google.common.base.Optional<String> traceId,
                                      int partitionId,
                                      boolean writeAtLeastOnce)

Method Detail

addRow

public void addRow(com.google.protobuf.ByteString message)
            throws IOException

Adds a row to the protoRows, which acts as a buffer; but before, checks if the current message size in bytes will cause the protoRows buffer to exceed the maximum APPEND_REQUEST_SIZE, and if it will, sends an append rows request first.

Parameters:

message - The row, in a ByteString message, to be added to protoRows.

Throws:

IOException - If sendAppendRowsRequest fails.

See Also:

this#sendAppendRowsRequest()

finalizeStream

public WriteStreamStatistics finalizeStream()
                                     throws IOException

Appends any data that remains in the protoRows, waits for 500 milliseconds, and finalizes the write-stream. This also closes the internal StreamWriter, so that the helper instance is not usable after calling finalizeStream().

Returns:

The finalized row-count of the write-stream.

Throws:

IOException - If the row-count returned by the FinalizeWriteStreamResponse does not match the expected offset (which is equal to the number of rows appended thus far).

See Also:

this#writeStreamRowCount

abort

public void abort()

Deletes the data left over in the protoRows, using method clearProtoRows, closes the StreamWriter, shuts down the WriteClient, and nulls out the protoRows and write-stream-name. This also closes the internal StreamWriter, so that the helper instance is not * usable after calling commit().

getWriteStreamName

public String getWriteStreamName()