com.google.cloud.bigquery.connector.common

Class BigQueryUtil

java.lang.Object

com.google.cloud.bigquery.connector.common.BigQueryUtil

public class BigQueryUtil
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BIG_NUMERIC_PRECISION
static int	DEFAULT_BIG_NUMERIC_SCALE
static int	DEFAULT_NUMERIC_PRECISION
static int	DEFAULT_NUMERIC_SCALE

Method Summary

Modifier and Type	Method and Description
static com.google.cloud.bigquery.Schema	adjustSchemaIfNeeded(com.google.cloud.bigquery.Schema wantedSchema, com.google.cloud.bigquery.Schema existingTableSchema) Adjusts the wanted schema to properly match the schema of an existing table.
static void	convertAndThrow(com.google.cloud.bigquery.BigQueryError error)
static <T> T	createVerifiedInstance(String fullyQualifiedClassName, Class<T> requiredClass, Object... constructorArgs) Create an instance of the given class name, and verify that it is an instance of the required class
static Optional<String>	emptyIfNeeded(String value)
static boolean	filterLengthInLimit(Optional<String> filter)
static <T> Optional<T>	firstPresent(Optional<T>... optionals)
static String	friendlyTableName(com.google.cloud.bigquery.TableId tableId)
static com.google.common.collect.ImmutableList<String>	getClusteringFields(com.google.cloud.bigquery.TableInfo tableInfo)
static com.google.common.collect.ImmutableList<String>	getPartitionFields(com.google.cloud.bigquery.TableInfo tableInfo)
static List<String>	getStreamNames(com.google.cloud.bigquery.storage.v1.ReadSession readSession) Create a list of read stream names of given read session.
static boolean	isBigLakeManagedTable(com.google.cloud.bigquery.TableInfo table) BigLake Managed tables are not represented by a dedicated type.
static boolean	isBigQueryNativeTable(com.google.cloud.bigquery.TableInfo table) Since StandardTableDefinition (table_type == TableDefinition.Type.TABLE) can represent both BigQuery native tables and BigLake Managed tables, the absence of the "bigLakeConfiguration" field within the StandardTableDefinition StandardTableDefinition.getBigLakeConfiguration().
static boolean	isRetryable(Throwable cause)
static List<String>	optimizeLoadUriList(List<String> uris, String prefixRegex, String suffixRegex) Solving Issue #248.
static com.google.cloud.bigquery.TableId	parseTableId(String rawTable)
static com.google.cloud.bigquery.TableId	parseTableId(String rawTable, Optional<String> dataset, Optional<String> project)
static com.google.cloud.bigquery.TableId	parseTableId(String rawTable, Optional<String> dataset, Optional<String> project, Optional<String> datePartition)
static String	prepareQueryForLog(String query, int maxLength)
static String	sanitizeLabelValue(String value)
static boolean	schemaWritable(com.google.cloud.bigquery.Schema sourceSchema, com.google.cloud.bigquery.Schema destinationSchema, boolean regardFieldOrder, boolean enableModeCheckForSchemaFields) Compares to Schema instances for equality.
static <T> T	verifySerialization(T obj) Verify the given object is Serializable by returning the deserialized version of the serialized instance

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_NUMERIC_PRECISION

public static final int DEFAULT_NUMERIC_PRECISION

See Also:

Constant Field Values

DEFAULT_NUMERIC_SCALE

public static final int DEFAULT_NUMERIC_SCALE

See Also:

Constant Field Values

DEFAULT_BIG_NUMERIC_PRECISION

public static final int DEFAULT_BIG_NUMERIC_PRECISION

See Also:

Constant Field Values

DEFAULT_BIG_NUMERIC_SCALE

public static final int DEFAULT_BIG_NUMERIC_SCALE

See Also:

Constant Field Values

Method Detail

isRetryable

public static boolean isRetryable(Throwable cause)

firstPresent

public static <T> Optional<T> firstPresent(Optional<T>... optionals)

parseTableId

public static com.google.cloud.bigquery.TableId parseTableId(String rawTable)

parseTableId

public static com.google.cloud.bigquery.TableId parseTableId(String rawTable,
                                                             Optional<String> dataset,
                                                             Optional<String> project)

parseTableId

public static com.google.cloud.bigquery.TableId parseTableId(String rawTable,
                                                             Optional<String> dataset,
                                                             Optional<String> project,
                                                             Optional<String> datePartition)

friendlyTableName

public static String friendlyTableName(com.google.cloud.bigquery.TableId tableId)

convertAndThrow

public static void convertAndThrow(com.google.cloud.bigquery.BigQueryError error)

optimizeLoadUriList

public static List<String> optimizeLoadUriList(List<String> uris,
                                               String prefixRegex,
                                               String suffixRegex)

Solving Issue #248. As the bigquery load API can handle wildcard characters, we replace (a) 10 URIs ending in XXX[0-9].suffix with XXX*.suffix; (b) 10 URIs ending in XXX[0-9]*.suffix with XXX*.suffix; This reduces the total number of URIs sent to the BigQuery API

Parameters:

uris - the list of URIs where the data was written to

prefixRegex - the regex to catch the part before the file index.

suffixRegex - the regex to catch the part after the file index.

schemaWritable

public static boolean schemaWritable(com.google.cloud.bigquery.Schema sourceSchema,
                                     com.google.cloud.bigquery.Schema destinationSchema,
                                     boolean regardFieldOrder,
                                     boolean enableModeCheckForSchemaFields)

Compares to Schema instances for equality. Unlike Schema.equals(), here the caller can disregard the schema's field order.

Parameters:

sourceSchema - the first schema to compare

destinationSchema - the second schema to compare

regardFieldOrder - whether to regard the field order in the comparison

Returns:

true is the two schema equal each other, false otherwise

getStreamNames

public static List<String> getStreamNames(com.google.cloud.bigquery.storage.v1.ReadSession readSession)

Create a list of read stream names of given read session.

Parameters:

readSession - BQ read session handle

Returns:

list of stream names, empty if read session is null

emptyIfNeeded

public static Optional<String> emptyIfNeeded(String value)

createVerifiedInstance

public static <T> T createVerifiedInstance(String fullyQualifiedClassName,
                                           Class<T> requiredClass,
                                           Object... constructorArgs)

Create an instance of the given class name, and verify that it is an instance of the required class

verifySerialization

public static <T> T verifySerialization(T obj)

Verify the given object is Serializable by returning the deserialized version of the serialized instance

getPartitionFields

public static com.google.common.collect.ImmutableList<String> getPartitionFields(com.google.cloud.bigquery.TableInfo tableInfo)

getClusteringFields

public static com.google.common.collect.ImmutableList<String> getClusteringFields(com.google.cloud.bigquery.TableInfo tableInfo)

filterLengthInLimit

public static boolean filterLengthInLimit(Optional<String> filter)

adjustSchemaIfNeeded

public static com.google.cloud.bigquery.Schema adjustSchemaIfNeeded(com.google.cloud.bigquery.Schema wantedSchema,
                                                                    com.google.cloud.bigquery.Schema existingTableSchema)

Adjusts the wanted schema to properly match the schema of an existing table.

Parameters:

wantedSchema -

existingTableSchema -

Returns:

the adjusted schema

prepareQueryForLog

public static String prepareQueryForLog(String query,
                                        int maxLength)

sanitizeLabelValue

public static String sanitizeLabelValue(String value)

isBigLakeManagedTable

public static boolean isBigLakeManagedTable(com.google.cloud.bigquery.TableInfo table)

BigLake Managed tables are not represented by a dedicated type. Instead, their presence is indicated by the field 'bigLakeConfiguration' within the StandardTableDefinition StandardTableDefinition.getBigLakeConfiguration().

Parameters:

table - the table to check

Returns:

Returns true if the table is a BigLake Managed table.

isBigQueryNativeTable

public static boolean isBigQueryNativeTable(com.google.cloud.bigquery.TableInfo table)

Since StandardTableDefinition (table_type == TableDefinition.Type.TABLE) can represent both BigQuery native tables and BigLake Managed tables, the absence of the "bigLakeConfiguration" field within the StandardTableDefinition StandardTableDefinition.getBigLakeConfiguration(). indicates a BigQuery native table.

Parameters:

table - the table to check

Returns:

Returns true if the table is a BigQuery Native table.