com.google.cloud.spark.bigquery

Class SparkBigQueryUtil

java.lang.Object

com.google.cloud.spark.bigquery.SparkBigQueryUtil

public class SparkBigQueryUtil
extends Object

Spark related utilities

Constructor Summary

Constructors

Constructor and Description
SparkBigQueryUtil()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.fs.Path	createGcsPath(SparkBigQueryConfig config, org.apache.hadoop.conf.Configuration conf, String applicationId) Checks whether temporaryGcsBucket or persistentGcsBucket parameters are present in the config and creates a org.apache.hadoop.fs.Path object backed by GCS.
static SparkBigQueryConfig	createSparkBigQueryConfig(org.apache.spark.sql.SQLContext sqlContext, scala.collection.immutable.Map<String,String> options, scala.Option<org.apache.spark.sql.types.StructType> schema, DataSourceVersion dataSourceVersion)
static @NotNull com.google.common.collect.ImmutableMap<String,String>	extractJobLabels(org.apache.spark.SparkConf sparkConf)
static com.google.common.collect.ImmutableList<org.apache.spark.sql.sources.Filter>	extractPartitionAndClusteringFilters(com.google.cloud.bigquery.TableInfo table, com.google.common.collect.ImmutableList<org.apache.spark.sql.sources.Filter> filters)
static String	getJobId(org.apache.spark.sql.internal.SQLConf sqlConf)
static String	getTableNameFromOptions(Map<String,String> options)
static Stream<TypeConverter>	getTypeConverterStream()
static boolean	isDataFrameShowMethodInStackTrace()
static boolean	isJson(org.apache.spark.sql.types.Metadata metadata)
static List<String>	optimizeLoadUriListForSpark(List<String> uris) Optimizing the URI list for BigQuery load, using the Spark specific file prefix and suffix patterns, based on BigQueryUtil.optimizeLoadUriList()
static com.google.cloud.bigquery.TableId	parseSimpleTableId(org.apache.spark.sql.SparkSession spark, Map<String,String> options)
static com.google.cloud.bigquery.JobInfo.WriteDisposition	saveModeToWriteDisposition(org.apache.spark.sql.SaveMode saveMode)
static <K,V> com.google.common.collect.ImmutableMap<K,V>	scalaMapToJavaMap(scala.collection.immutable.Map<K,V> map)
static int	sparkDateToBigQuery(Object sparkValue)
static long	sparkTimestampToBigQuery(Object sparkValue)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkBigQueryUtil

public SparkBigQueryUtil()

Method Detail

optimizeLoadUriListForSpark

public static List<String> optimizeLoadUriListForSpark(List<String> uris)

Optimizing the URI list for BigQuery load, using the Spark specific file prefix and suffix patterns, based on BigQueryUtil.optimizeLoadUriList()

Parameters:

uris - A list of URIs to be loaded by BigQuery load

Returns:

an optimized list of URIs

createGcsPath

public static org.apache.hadoop.fs.Path createGcsPath(SparkBigQueryConfig config,
                                                      org.apache.hadoop.conf.Configuration conf,
                                                      String applicationId)

Checks whether temporaryGcsBucket or persistentGcsBucket parameters are present in the config and creates a org.apache.hadoop.fs.Path object backed by GCS. When the indirect write method in dsv1 is used, the data is written first to this GCS path and is then loaded into BigQuery

Parameters:

config - SparkBigQueryConfig

conf - Hadoop configuration parameters

applicationId - A unique identifier for the Spark application

Returns:

org.apache.hadoop.fs.Path object backed by GCS

getJobId

public static String getJobId(org.apache.spark.sql.internal.SQLConf sqlConf)

saveModeToWriteDisposition

public static com.google.cloud.bigquery.JobInfo.WriteDisposition saveModeToWriteDisposition(org.apache.spark.sql.SaveMode saveMode)

parseSimpleTableId

public static com.google.cloud.bigquery.TableId parseSimpleTableId(org.apache.spark.sql.SparkSession spark,
                                                                   Map<String,String> options)

sparkTimestampToBigQuery

public static long sparkTimestampToBigQuery(Object sparkValue)

sparkDateToBigQuery

public static int sparkDateToBigQuery(Object sparkValue)

getTableNameFromOptions

public static String getTableNameFromOptions(Map<String,String> options)

scalaMapToJavaMap

public static <K,V> com.google.common.collect.ImmutableMap<K,V> scalaMapToJavaMap(scala.collection.immutable.Map<K,V> map)

isDataFrameShowMethodInStackTrace

public static boolean isDataFrameShowMethodInStackTrace()

isJson

public static boolean isJson(org.apache.spark.sql.types.Metadata metadata)

extractPartitionAndClusteringFilters

public static com.google.common.collect.ImmutableList<org.apache.spark.sql.sources.Filter> extractPartitionAndClusteringFilters(com.google.cloud.bigquery.TableInfo table,
                                                                                                                                com.google.common.collect.ImmutableList<org.apache.spark.sql.sources.Filter> filters)

getTypeConverterStream

public static Stream<TypeConverter> getTypeConverterStream()

extractJobLabels

@NotNull
public static @NotNull com.google.common.collect.ImmutableMap<String,String> extractJobLabels(org.apache.spark.SparkConf sparkConf)

createSparkBigQueryConfig

public static SparkBigQueryConfig createSparkBigQueryConfig(org.apache.spark.sql.SQLContext sqlContext,
                                                            scala.collection.immutable.Map<String,String> options,
                                                            scala.Option<org.apache.spark.sql.types.StructType> schema,
                                                            DataSourceVersion dataSourceVersion)