com.google.cloud.bigquery.connector.common

Class BigQueryClient

java.lang.Object

com.google.cloud.bigquery.connector.common.BigQueryClient

public class BigQueryClient
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	BigQueryClient.CreateTableOptions
static interface	BigQueryClient.LoadDataOptions
static interface	BigQueryClient.ReadTableOptions

Constructor Summary

Constructors

Constructor and Description

BigQueryClient(com.google.cloud.bigquery.BigQuery bigQuery, Optional<String> materializationProject, Optional<String> materializationDataset, com.google.common.cache.Cache<String,com.google.cloud.bigquery.TableInfo> destinationTableCache, Map<String,String> labels, com.google.cloud.bigquery.QueryJobConfiguration.Priority queryJobPriority, Optional<BigQueryJobCompletionListener> jobCompletionListener, long bigQueryJobTimeoutInMinutes)

Method Summary

Modifier and Type	Method and Description
com.google.cloud.bigquery.Job	appendDestinationWithTemporary(com.google.cloud.bigquery.TableId temporaryTableId, com.google.cloud.bigquery.TableId destinationTableId) Appends all the data from the given temporary table, to the given destination table, transactionally.
long	calculateTableSize(com.google.cloud.bigquery.TableId tableId, Optional<String> filter)
long	calculateTableSize(com.google.cloud.bigquery.TableInfo tableInfo, Optional<String> filter)
com.google.cloud.bigquery.Job	createAndWaitFor(com.google.cloud.bigquery.JobConfiguration.Builder jobConfiguration)
com.google.cloud.bigquery.Job	createAndWaitFor(com.google.cloud.bigquery.JobConfiguration jobConfiguration)
com.google.cloud.bigquery.TableInfo	createTable(com.google.cloud.bigquery.TableId tableId, com.google.cloud.bigquery.Schema schema, BigQueryClient.CreateTableOptions options) Creates an empty table in BigQuery.
void	createTableIfNeeded(com.google.cloud.bigquery.TableId tableId, com.google.cloud.bigquery.Schema bigQuerySchema, BigQueryClient.CreateTableOptions options) Creates the table with the given schema, only if it does not exist yet.
String	createTablePathForBigQueryStorage(com.google.cloud.bigquery.TableId tableId) Creates a String appropriately formatted for BigQuery Storage Write API representing the given table.
com.google.cloud.bigquery.TableInfo	createTempTable(com.google.cloud.bigquery.TableId destinationTableId, com.google.cloud.bigquery.Schema schema) Creates a temporary table with a job to cleanup after application end, and the same location as the destination table; the temporary table will have the same name as the destination table, with the current time in milliseconds appended to it; useful for holding temporary data in order to overwrite the destination table.
com.google.cloud.bigquery.TableInfo	createTempTableAfterCheckingSchema(com.google.cloud.bigquery.TableId destinationTableId, com.google.cloud.bigquery.Schema schema, boolean enableModeCheckForSchemaFields)
com.google.cloud.bigquery.TableId	createTempTableId(com.google.cloud.bigquery.TableId destinationTableId)
boolean	deleteTable(com.google.cloud.bigquery.TableId tableId) Deletes this table in BigQuery.
static String	fullTableName(com.google.cloud.bigquery.TableId tableId)
String	getProjectId()
com.google.cloud.bigquery.Schema	getQueryResultSchema(String querySql, Map<String,String> additionalQueryJobLabels)
com.google.cloud.bigquery.TableInfo	getReadTable(BigQueryClient.ReadTableOptions options)
com.google.cloud.bigquery.Schema	getReadTableSchema(BigQueryClient.ReadTableOptions options) Returns the schema of the table/query/view.
com.google.cloud.bigquery.TableInfo	getTable(com.google.cloud.bigquery.TableId tableId)
Iterable<com.google.cloud.bigquery.Table>	listTables(com.google.cloud.bigquery.DatasetId datasetId, com.google.cloud.bigquery.TableDefinition.Type... types)
com.google.cloud.bigquery.JobStatistics.LoadStatistics	loadDataIntoTable(BigQueryClient.LoadDataOptions options, List<String> sourceUris, com.google.cloud.bigquery.FormatOptions formatOptions, com.google.cloud.bigquery.JobInfo.WriteDisposition writeDisposition, Optional<com.google.cloud.bigquery.Schema> schema, com.google.cloud.bigquery.TableId destinationTable)
com.google.cloud.bigquery.TableInfo	materializeQueryToTable(String querySql, int expirationTimeInMinutes) Runs the provided query on BigQuery and saves the result in a temporary table.
com.google.cloud.bigquery.TableInfo	materializeQueryToTable(String querySql, int expirationTimeInMinutes, Map<String,String> additionalQueryJobLabels) Runs the provided query on BigQuery and saves the result in a temporary table.
com.google.cloud.bigquery.TableInfo	materializeViewToTable(String querySql, com.google.cloud.bigquery.TableId viewId, int expirationTimeInMinutes) Runs the provided query on BigQuery and saves the result in a temporary table.
com.google.cloud.bigquery.Job	overwriteDestinationWithTemporary(com.google.cloud.bigquery.TableId temporaryTableId, com.google.cloud.bigquery.TableId destinationTableId) Overwrites the given destination table, with all the data from the given temporary table, transactionally.
com.google.cloud.bigquery.Job	overwriteDestinationWithTemporaryDynamicPartitons(com.google.cloud.bigquery.TableId temporaryTableId, com.google.cloud.bigquery.TableId destinationTableId) Overwrites the partitions of the destination table, using the partitions from the given temporary table, transactionally.
com.google.cloud.bigquery.TableResult	query(String sql)
static void	runCleanupJobs()
boolean	tableExists(com.google.cloud.bigquery.TableId tableId) Checks whether the requested table exists in BigQuery.
com.google.cloud.bigquery.Table	update(com.google.cloud.bigquery.TableInfo table)
com.google.cloud.bigquery.JobInfo	waitForJob(com.google.cloud.bigquery.Job job) Waits for a BigQuery Job to complete: this is a blocking function.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigQueryClient

public BigQueryClient(com.google.cloud.bigquery.BigQuery bigQuery,
                      Optional<String> materializationProject,
                      Optional<String> materializationDataset,
                      com.google.common.cache.Cache<String,com.google.cloud.bigquery.TableInfo> destinationTableCache,
                      Map<String,String> labels,
                      com.google.cloud.bigquery.QueryJobConfiguration.Priority queryJobPriority,
                      Optional<BigQueryJobCompletionListener> jobCompletionListener,
                      long bigQueryJobTimeoutInMinutes)

Method Detail

runCleanupJobs

public static void runCleanupJobs()

waitForJob

public com.google.cloud.bigquery.JobInfo waitForJob(com.google.cloud.bigquery.Job job)

Waits for a BigQuery Job to complete: this is a blocking function.

Parameters:

job - The Job to keep track of.

getTable

public com.google.cloud.bigquery.TableInfo getTable(com.google.cloud.bigquery.TableId tableId)

tableExists

public boolean tableExists(com.google.cloud.bigquery.TableId tableId)

Checks whether the requested table exists in BigQuery.

Parameters:

tableId - The TableId of the requested table in BigQuery

Returns:

True if the requested table exists in BigQuery, false otherwise.

createTable

public com.google.cloud.bigquery.TableInfo createTable(com.google.cloud.bigquery.TableId tableId,
                                                       com.google.cloud.bigquery.Schema schema,
                                                       BigQueryClient.CreateTableOptions options)

Creates an empty table in BigQuery.

Parameters:

tableId - The TableId of the table to be created.

schema - The Schema of the table to be created.

options - Allows configuring the created table

Returns:

The Table object representing the table that was created.

createTempTable

public com.google.cloud.bigquery.TableInfo createTempTable(com.google.cloud.bigquery.TableId destinationTableId,
                                                           com.google.cloud.bigquery.Schema schema)

Creates a temporary table with a job to cleanup after application end, and the same location as the destination table; the temporary table will have the same name as the destination table, with the current time in milliseconds appended to it; useful for holding temporary data in order to overwrite the destination table.

Parameters:

destinationTableId - The TableId of the eventual destination for the data going into the temporary table.

schema - The Schema of the destination / temporary table.

Returns:

The Table object representing the created temporary table.

createTempTableAfterCheckingSchema

public com.google.cloud.bigquery.TableInfo createTempTableAfterCheckingSchema(com.google.cloud.bigquery.TableId destinationTableId,
                                                                              com.google.cloud.bigquery.Schema schema,
                                                                              boolean enableModeCheckForSchemaFields)
                                                                       throws IllegalArgumentException

Throws:

IllegalArgumentException

createTempTableId

public com.google.cloud.bigquery.TableId createTempTableId(com.google.cloud.bigquery.TableId destinationTableId)

deleteTable

public boolean deleteTable(com.google.cloud.bigquery.TableId tableId)

Deletes this table in BigQuery.

Parameters:

tableId - The TableId of the table to be deleted.

Returns:

True if the operation was successful, false otherwise.

overwriteDestinationWithTemporaryDynamicPartitons

public com.google.cloud.bigquery.Job overwriteDestinationWithTemporaryDynamicPartitons(com.google.cloud.bigquery.TableId temporaryTableId,
                                                                                       com.google.cloud.bigquery.TableId destinationTableId)

Overwrites the partitions of the destination table, using the partitions from the given temporary table, transactionally.

Parameters:

temporaryTableId - The TableId representing the temporary-table.

destinationTableId - The TableId representing the destination table.

Returns:

The Job object representing this operation (which can be tracked to wait until it has finished successfully).

overwriteDestinationWithTemporary

public com.google.cloud.bigquery.Job overwriteDestinationWithTemporary(com.google.cloud.bigquery.TableId temporaryTableId,
                                                                       com.google.cloud.bigquery.TableId destinationTableId)

Overwrites the given destination table, with all the data from the given temporary table, transactionally.

Parameters:

temporaryTableId - The TableId representing the temporary-table.

destinationTableId - The TableId representing the destination table.

Returns:

The Job object representing this operation (which can be tracked to wait until it has finished successfully).

appendDestinationWithTemporary

public com.google.cloud.bigquery.Job appendDestinationWithTemporary(com.google.cloud.bigquery.TableId temporaryTableId,
                                                                    com.google.cloud.bigquery.TableId destinationTableId)

Appends all the data from the given temporary table, to the given destination table, transactionally.

Parameters:

temporaryTableId - The TableId representing the temporary-table.

destinationTableId - The TableId representing the destination table.

Returns:

The Job object representing this operation (which can be tracked to wait until it has finished successfully).

createTablePathForBigQueryStorage

public String createTablePathForBigQueryStorage(com.google.cloud.bigquery.TableId tableId)

Creates a String appropriately formatted for BigQuery Storage Write API representing the given table.

Parameters:

tableId - The TableId representing the given object.

Returns:

The formatted String.

getReadTable

public com.google.cloud.bigquery.TableInfo getReadTable(BigQueryClient.ReadTableOptions options)

getReadTableSchema

public com.google.cloud.bigquery.Schema getReadTableSchema(BigQueryClient.ReadTableOptions options)

Returns the schema of the table/query/view. Uses dryRun to get the query schema instead of materializing it.

Parameters:

options - The ReadTableOptions options for reading the data source.

Returns:

The schema.

getProjectId

public String getProjectId()

listTables

public Iterable<com.google.cloud.bigquery.Table> listTables(com.google.cloud.bigquery.DatasetId datasetId,
                                                            com.google.cloud.bigquery.TableDefinition.Type... types)

update

public com.google.cloud.bigquery.Table update(com.google.cloud.bigquery.TableInfo table)

createAndWaitFor

public com.google.cloud.bigquery.Job createAndWaitFor(com.google.cloud.bigquery.JobConfiguration.Builder jobConfiguration)

createAndWaitFor

public com.google.cloud.bigquery.Job createAndWaitFor(com.google.cloud.bigquery.JobConfiguration jobConfiguration)

query

public com.google.cloud.bigquery.TableResult query(String sql)

fullTableName

public static String fullTableName(com.google.cloud.bigquery.TableId tableId)

calculateTableSize

public long calculateTableSize(com.google.cloud.bigquery.TableId tableId,
                               Optional<String> filter)

calculateTableSize

public long calculateTableSize(com.google.cloud.bigquery.TableInfo tableInfo,
                               Optional<String> filter)

materializeQueryToTable

public com.google.cloud.bigquery.TableInfo materializeQueryToTable(String querySql,
                                                                   int expirationTimeInMinutes)

Runs the provided query on BigQuery and saves the result in a temporary table.

Parameters:

querySql - the query to be run

expirationTimeInMinutes - the time in minutes until the table is expired and auto-deleted

Returns:

a reference to the table

materializeQueryToTable

public com.google.cloud.bigquery.TableInfo materializeQueryToTable(String querySql,
                                                                   int expirationTimeInMinutes,
                                                                   Map<String,String> additionalQueryJobLabels)

Runs the provided query on BigQuery and saves the result in a temporary table.

Parameters:

querySql - the query to be run

expirationTimeInMinutes - the time in minutes until the table is expired and auto-deleted

additionalQueryJobLabels - the labels to insert on the query job

Returns:

a reference to the table

materializeViewToTable

public com.google.cloud.bigquery.TableInfo materializeViewToTable(String querySql,
                                                                  com.google.cloud.bigquery.TableId viewId,
                                                                  int expirationTimeInMinutes)

Runs the provided query on BigQuery and saves the result in a temporary table. This method is intended to be used to materialize views, so the view location (based on its TableId) is taken as a location for the temporary table, removing the need to set the materializationProject and materializationDataset properties

Parameters:

querySql - the query to be run

viewId - the view the query came from

expirationTimeInMinutes - the time in hours until the table is expired and auto-deleted

Returns:

a reference to the table

getQueryResultSchema

public com.google.cloud.bigquery.Schema getQueryResultSchema(String querySql,
                                                             Map<String,String> additionalQueryJobLabels)

loadDataIntoTable

public com.google.cloud.bigquery.JobStatistics.LoadStatistics loadDataIntoTable(BigQueryClient.LoadDataOptions options,
                                                                                List<String> sourceUris,
                                                                                com.google.cloud.bigquery.FormatOptions formatOptions,
                                                                                com.google.cloud.bigquery.JobInfo.WriteDisposition writeDisposition,
                                                                                Optional<com.google.cloud.bigquery.Schema> schema,
                                                                                com.google.cloud.bigquery.TableId destinationTable)

createTableIfNeeded

public void createTableIfNeeded(com.google.cloud.bigquery.TableId tableId,
                                com.google.cloud.bigquery.Schema bigQuerySchema,
                                BigQueryClient.CreateTableOptions options)

Creates the table with the given schema, only if it does not exist yet.