org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryUtils

java.lang.Object

org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils

public class BigQueryUtils
extends java.lang.Object

Utility methods for BigQuery related operations.

Example: Writing to BigQuery

 PCollection<Row> rows = ...;

 rows.apply(BigQueryIO.<Row>write()
       .withSchema(BigQueryUtils.toTableSchema(rows))
       .withFormatFunction(BigQueryUtils.toTableRow())
       .to("my-project:my_dataset.my_table"));
 

Constructor Summary

Constructors

Constructor and Description
BigQueryUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.beam.sdk.values.Row	toBeamRow(org.apache.avro.generic.GenericRecord record, org.apache.beam.sdk.schemas.Schema schema)
static org.apache.beam.sdk.transforms.SerializableFunction<SchemaAndRecord,org.apache.beam.sdk.values.Row>	toBeamRow(org.apache.beam.sdk.schemas.Schema schema) Convert SchemaAndRecord to a Beam Row.
static org.apache.beam.sdk.values.Row	toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema, com.google.api.services.bigquery.model.TableSchema bqSchema, com.google.api.services.bigquery.model.TableRow jsonBqRow) Tries to parse the JSON TableRow from BigQuery.
static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,com.google.api.services.bigquery.model.TableRow>	toTableRow() Convert a Beam Row to a BigQuery TableRow.
static com.google.api.services.bigquery.model.TableRow	toTableRow(org.apache.beam.sdk.values.Row row)
static com.google.api.services.bigquery.model.TableSchema	toTableSchema(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.Row> rows) Convert a Beam PCollection to a BigQuery TableSchema.
static com.google.api.services.bigquery.model.TableSchema	toTableSchema(org.apache.beam.sdk.schemas.Schema schema) Convert a Beam Schema to a BigQuery TableSchema.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigQueryUtils

public BigQueryUtils()

Method Detail

toTableSchema

public static com.google.api.services.bigquery.model.TableSchema toTableSchema(org.apache.beam.sdk.schemas.Schema schema)

Convert a Beam Schema to a BigQuery TableSchema.

toTableSchema

public static com.google.api.services.bigquery.model.TableSchema toTableSchema(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.Row> rows)

Convert a Beam PCollection to a BigQuery TableSchema.

toTableRow

public static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,com.google.api.services.bigquery.model.TableRow> toTableRow()

Convert a Beam Row to a BigQuery TableRow.

toBeamRow

public static org.apache.beam.sdk.transforms.SerializableFunction<SchemaAndRecord,org.apache.beam.sdk.values.Row> toBeamRow(org.apache.beam.sdk.schemas.Schema schema)

Convert SchemaAndRecord to a Beam Row.

toBeamRow

public static org.apache.beam.sdk.values.Row toBeamRow(org.apache.avro.generic.GenericRecord record,
                                                       org.apache.beam.sdk.schemas.Schema schema)

toTableRow

public static com.google.api.services.bigquery.model.TableRow toTableRow(org.apache.beam.sdk.values.Row row)

toBeamRow

public static org.apache.beam.sdk.values.Row toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema,
                                                       com.google.api.services.bigquery.model.TableSchema bqSchema,
                                                       com.google.api.services.bigquery.model.TableRow jsonBqRow)

Tries to parse the JSON TableRow from BigQuery.

Only supports basic types and arrays. Doesn't support date types.