com.google.cloud.bigquery.storage.v1

Class StreamWriter

java.lang.Object

com.google.cloud.bigquery.storage.v1.StreamWriter

All Implemented Interfaces:

AutoCloseable

public class StreamWriter
extends Object
implements AutoCloseable

A BigQuery Stream Writer that can be used to write data into BigQuery Table.

TODO: Support batching.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StreamWriter.Builder A builder of StreamWriters.

Method Summary

Modifier and Type	Method and Description
com.google.api.core.ApiFuture<AppendRowsResponse>	append(ProtoRows rows) Schedules the writing of rows at the end of current stream.
com.google.api.core.ApiFuture<AppendRowsResponse>	append(ProtoRows rows, long offset) Schedules the writing of rows at given offset.
void	close() Close the stream writer.
static long	getApiMaxRequestBytes() The maximum size of one request.
long	getInflightWaitSeconds() Returns the wait of a request in Client side before sending to the Server.
String	getStreamName()
TableSchema	getUpdatedSchema() Thread-safe getter of updated TableSchema
String	getWriterId()
static StreamWriter.Builder	newBuilder(String streamName) Constructs a new StreamWriterV2.Builder using the given stream.
static StreamWriter.Builder	newBuilder(String streamName, BigQueryWriteClient client) Constructs a new StreamWriterV2.Builder using the given stream and client.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getApiMaxRequestBytes

public static long getApiMaxRequestBytes()

The maximum size of one request. Defined by the API.

append

public com.google.api.core.ApiFuture<AppendRowsResponse> append(ProtoRows rows)

Schedules the writing of rows at the end of current stream.

Parameters:

rows - the rows in serialized format to write to BigQuery.

Returns:

the append response wrapped in a future.

append

public com.google.api.core.ApiFuture<AppendRowsResponse> append(ProtoRows rows,
                                                                long offset)

Schedules the writing of rows at given offset.

Example of writing rows with specific offset.

 ApiFuture<AppendRowsResponse> future = writer.append(rows, 0);
 ApiFutures.addCallback(future, new ApiFutureCallback<AppendRowsResponse>() {
   public void onSuccess(AppendRowsResponse response) {
     if (!response.hasError()) {
       System.out.println("written with offset: " + response.getAppendResult().getOffset());
     } else {
       System.out.println("received an in stream error: " + response.getError().toString());
     }
   }

   public void onFailure(Throwable t) {
     System.out.println("failed to write: " + t);
   }
 }, MoreExecutors.directExecutor());
 

Parameters:

rows - the rows in serialized format to write to BigQuery.

offset - the offset of the first row. Provide -1 to write at the current end of stream.

Returns:

the append response wrapped in a future.

getInflightWaitSeconds

public long getInflightWaitSeconds()

Returns the wait of a request in Client side before sending to the Server. Request could wait in Client because it reached the client side inflight request limit (adjustable when constructing the StreamWriter). The value is the wait time for the last sent request. A constant high wait value indicates a need for more throughput, you can create a new Stream for to increase the throughput in exclusive stream case, or create a new Writer in the default stream case.

getWriterId

public String getWriterId()

Returns:

a unique Id for the writer.

getStreamName

public String getStreamName()

Returns:

name of the Stream that this writer is working on.

close

public void close()

Close the stream writer. Shut down all resources.

Specified by:

close in interface AutoCloseable

newBuilder

public static StreamWriter.Builder newBuilder(String streamName,
                                              BigQueryWriteClient client)

Constructs a new StreamWriterV2.Builder using the given stream and client. AppendRows needs special headers to be added to client, so a passed in client will not work. This should be used by test only.

newBuilder

public static StreamWriter.Builder newBuilder(String streamName)

Constructs a new StreamWriterV2.Builder using the given stream.

getUpdatedSchema

public TableSchema getUpdatedSchema()

Thread-safe getter of updated TableSchema