org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryUtils

java.lang.Object

org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils

public class BigQueryUtils
extends java.lang.Object

Utility methods for BigQuery related operations.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BigQueryUtils.ConversionOptions Options for how to convert BigQuery data to Beam data.

Constructor Summary

Constructors

Constructor and Description
BigQueryUtils()

Method Summary

Modifier and Type	Method and Description
static java.lang.Object	convertAvroFormat(org.apache.beam.sdk.schemas.Schema.FieldType beamFieldType, java.lang.Object avroValue, BigQueryUtils.ConversionOptions options) Tries to convert an Avro decoded value to a Beam field value based on the target type of the Beam field.
static com.google.api.services.bigquery.model.TableRow	convertGenericRecordToTableRow(org.apache.avro.generic.GenericRecord record, com.google.api.services.bigquery.model.TableSchema tableSchema)
static org.apache.beam.sdk.schemas.Schema	fromTableSchema(com.google.api.services.bigquery.model.TableSchema tableSchema) Convert a BigQuery TableSchema to a Beam Schema.
static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<com.google.api.services.bigquery.model.TableRow>	tableRowFromBeamRow()
static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<com.google.api.services.bigquery.model.TableRow>	tableRowToBeamRow()
static org.apache.beam.sdk.values.Row	toBeamRow(org.apache.avro.generic.GenericRecord record, org.apache.beam.sdk.schemas.Schema schema, BigQueryUtils.ConversionOptions options)
static org.apache.beam.sdk.values.Row	toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema, com.google.api.services.bigquery.model.TableRow jsonBqRow) Tries to convert a JSON TableRow from BigQuery into a Beam Row.
static org.apache.beam.sdk.values.Row	toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema, com.google.api.services.bigquery.model.TableSchema bqSchema, com.google.api.services.bigquery.model.TableRow jsonBqRow) Tries to parse the JSON TableRow from BigQuery.
static org.apache.avro.Schema	toGenericAvroSchema(java.lang.String schemaName, java.util.List<com.google.api.services.bigquery.model.TableFieldSchema> fieldSchemas) Convert a list of BigQuery TableFieldSchema to Avro Schema.
static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,com.google.api.services.bigquery.model.TableRow>	toTableRow() Convert a Beam Row to a BigQuery TableRow.
static com.google.api.services.bigquery.model.TableRow	toTableRow(org.apache.beam.sdk.values.Row row) Convert a BigQuery TableRow to a Beam Row.
static <T> org.apache.beam.sdk.transforms.SerializableFunction<T,com.google.api.services.bigquery.model.TableRow>	toTableRow(org.apache.beam.sdk.transforms.SerializableFunction<T,org.apache.beam.sdk.values.Row> toRow) Convert a Beam schema type to a BigQuery TableRow.
static com.google.api.services.bigquery.model.TableSchema	toTableSchema(org.apache.beam.sdk.schemas.Schema schema) Convert a Beam Schema to a BigQuery TableSchema.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigQueryUtils

public BigQueryUtils()

Method Detail

toTableSchema

@Experimental(value=SCHEMAS)
public static com.google.api.services.bigquery.model.TableSchema toTableSchema(org.apache.beam.sdk.schemas.Schema schema)

Convert a Beam Schema to a BigQuery TableSchema.

fromTableSchema

@Experimental(value=SCHEMAS)
public static org.apache.beam.sdk.schemas.Schema fromTableSchema(com.google.api.services.bigquery.model.TableSchema tableSchema)

Convert a BigQuery TableSchema to a Beam Schema.

toGenericAvroSchema

@Experimental(value=SCHEMAS)
public static org.apache.avro.Schema toGenericAvroSchema(java.lang.String schemaName,
                                                                                      java.util.List<com.google.api.services.bigquery.model.TableFieldSchema> fieldSchemas)

Convert a list of BigQuery TableFieldSchema to Avro Schema.

tableRowToBeamRow

public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<com.google.api.services.bigquery.model.TableRow> tableRowToBeamRow()

tableRowFromBeamRow

public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<com.google.api.services.bigquery.model.TableRow> tableRowFromBeamRow()

toTableRow

public static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,com.google.api.services.bigquery.model.TableRow> toTableRow()

Convert a Beam Row to a BigQuery TableRow.

toTableRow

public static <T> org.apache.beam.sdk.transforms.SerializableFunction<T,com.google.api.services.bigquery.model.TableRow> toTableRow(org.apache.beam.sdk.transforms.SerializableFunction<T,org.apache.beam.sdk.values.Row> toRow)

Convert a Beam schema type to a BigQuery TableRow.

toBeamRow

@Experimental(value=SCHEMAS)
public static org.apache.beam.sdk.values.Row toBeamRow(org.apache.avro.generic.GenericRecord record,
                                                                                    org.apache.beam.sdk.schemas.Schema schema,
                                                                                    BigQueryUtils.ConversionOptions options)

convertGenericRecordToTableRow

public static com.google.api.services.bigquery.model.TableRow convertGenericRecordToTableRow(org.apache.avro.generic.GenericRecord record,
                                                                                             com.google.api.services.bigquery.model.TableSchema tableSchema)

toTableRow

public static com.google.api.services.bigquery.model.TableRow toTableRow(org.apache.beam.sdk.values.Row row)

Convert a BigQuery TableRow to a Beam Row.

toBeamRow

@Experimental(value=SCHEMAS)
public static org.apache.beam.sdk.values.Row toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema,
                                                                                    com.google.api.services.bigquery.model.TableRow jsonBqRow)

Tries to convert a JSON TableRow from BigQuery into a Beam Row.

Only supports basic types and arrays. Doesn't support date types or structs.

toBeamRow

@Experimental(value=SCHEMAS)
public static org.apache.beam.sdk.values.Row toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema,
                                                                                    com.google.api.services.bigquery.model.TableSchema bqSchema,
                                                                                    com.google.api.services.bigquery.model.TableRow jsonBqRow)

Tries to parse the JSON TableRow from BigQuery.

Only supports basic types and arrays. Doesn't support date types.

convertAvroFormat

public static java.lang.Object convertAvroFormat(org.apache.beam.sdk.schemas.Schema.FieldType beamFieldType,
                                                 java.lang.Object avroValue,
                                                 BigQueryUtils.ConversionOptions options)

Tries to convert an Avro decoded value to a Beam field value based on the target type of the Beam field.