com.google.cloud.bigtable.beam

Class CloudBigtableIO

java.lang.Object

com.google.cloud.bigtable.beam.CloudBigtableIO

@Experimental
public class CloudBigtableIO
extends Object

Utilities to create PTransforms for reading and writing Google Cloud Bigtable entities in a Beam pipeline.

Google Cloud Bigtable offers you a fast, fully managed, massively scalable NoSQL database service that's ideal for web, mobile, and Internet of Things applications requiring terabytes to petabytes of data. Unlike comparable market offerings, Cloud Bigtable doesn't require you to sacrifice speed, scale, or cost efficiency when your applications grow. Cloud Bigtable has been battle-tested at Google for more than 10 years--it's the database driving major applications such as Google Analytics and Gmail.

To use CloudBigtableIO, users must use gcloud to get a credential for Cloud Bigtable:

 $ gcloud auth login
 

To read a PCollection from a table, with an optional Scan, use read(CloudBigtableScanConfiguration):

 PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
 Pipeline p = Pipeline.create(options);
 PCollection<Result> = p.apply(
   Read.from(CloudBigtableIO.read(
      new CloudBigtableScanConfiguration.Builder()
          .withProjectId("project-id")
          .withInstanceId("instance-id")
          .withTableId("table-id")
          .build())));
 

To write a PCollection to a table, use writeToTable(CloudBigtableTableConfiguration):

 PipelineOptions options =
     PipelineOptionsFactory.fromArgs(args).create();
 Pipeline p = Pipeline.create(options);
 PCollection<Mutation> mutationCollection = ...;
 mutationCollection.apply(
   CloudBigtableIO.writeToTable(
      new CloudBigtableScanConfiguration.Builder()
          .withProjectId("project-id")
          .withInstanceId("instance-id")
          .withTableId("table-id")
          .build()));
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CloudBigtableIO.CloudBigtableMultiTableWriteFn A DoFn that can write either a bounded or unbounded PCollection of KV of (String tableName, List of Mutations) to the specified table.
static class	CloudBigtableIO.CloudBigtableSingleTableBufferedWriteFn A DoFn that can write either a bounded or unbounded PCollection of Mutations to a table specified via a CloudBigtableTableConfiguration using the BufferedMutator.
static class	CloudBigtableIO.Source A BoundedSource for a Cloud Bigtable Table, which is potentially filtered by a Scan.
protected static class	CloudBigtableIO.SourceWithKeys A BoundedSource for a Cloud Bigtable Table with a start/stop key range, along with a potential filter via a Scan.

Constructor Summary

Constructors

Constructor and Description
CloudBigtableIO()

Method Summary

Modifier and Type	Method and Description
static BoundedSource<Result>	read(CloudBigtableScanConfiguration config) Creates a BoundedSource for a Cloud Bigtable Table, which is potentially filtered by a Scan.
static PTransform<PCollection<KV<String,Iterable<Mutation>>>,PDone>	writeToMultipleTables(CloudBigtableConfiguration config) Creates a PTransform that can write either a bounded or unbounded PCollection of KV of (String tableName, List of Mutations) to the specified table.
static PTransform<PCollection<Mutation>,PDone>	writeToTable(CloudBigtableTableConfiguration config) Creates a PTransform that can write either a bounded or unbounded PCollection of Mutations to a table specified via a CloudBigtableTableConfiguration.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CloudBigtableIO

public CloudBigtableIO()

Method Detail

writeToTable

public static PTransform<PCollection<Mutation>,PDone> writeToTable(CloudBigtableTableConfiguration config)

Creates a PTransform that can write either a bounded or unbounded PCollection of Mutations to a table specified via a CloudBigtableTableConfiguration.

NOTE: This PTransform will write Puts and Deletes, not Appends and Increments. This limitation exists because if the batch fails partway through, Appends/Increments might be re-run, causing the Mutation to be executed twice, which is never the user's intent. Re-running a Delete will not cause any differences. Re-running a Put isn't normally a problem, but might cause problems in some cases when the number of versions supported by the column family is greater than one. In a case where multiple versions could be a problem, it's best to add a timestamp to the Put.

writeToMultipleTables

public static PTransform<PCollection<KV<String,Iterable<Mutation>>>,PDone> writeToMultipleTables(CloudBigtableConfiguration config)

Creates a PTransform that can write either a bounded or unbounded PCollection of KV of (String tableName, List of Mutations) to the specified table.

NOTE: This PTransform will write Puts and Deletes, not Appends and Increments. This limitation exists because if the batch fails partway through, Appends/Increments might be re-run, causing the Mutation to be executed twice, which is never the user's intent. Re-running a Delete will not cause any differences. Re-running a Put isn't normally a problem, but might cause problems in some cases when the number of versions supported by the column family is greater than one. In a case where multiple versions could be a problem, it's best to add a timestamp to the Put.

read

public static BoundedSource<Result> read(CloudBigtableScanConfiguration config)

Creates a BoundedSource for a Cloud Bigtable Table, which is potentially filtered by a Scan.