com.google.cloud.spark.bigquery.direct

Class BigQueryRDDFactory

java.lang.Object

com.google.cloud.spark.bigquery.direct.BigQueryRDDFactory

public class BigQueryRDDFactory
extends Object

Wrapper class for generating BigQueryRDD. Extracted this logic out so that we can reuse it from 1) Dsv1 buildScan 2) Dsv1 pushdown functionality 3) Dsv2 pushdown functionality

Constructor Summary

Constructors

Constructor and Description
BigQueryRDDFactory(BigQueryClient bigQueryClient, BigQueryClientFactory bigQueryReadClientFactory, BigQueryTracerFactory bigQueryTracerFactory, SparkBigQueryConfig options, org.apache.spark.sql.SQLContext sqlContext)

Method Summary

Modifier and Type	Method and Description
org.apache.spark.rdd.RDD<org.apache.spark.sql.catalyst.InternalRow>	buildScanFromSQL(String sql) Creates RDD from the SQL string that is passed in.
org.apache.spark.rdd.RDD<?>	createRddFromTable(com.google.cloud.bigquery.TableId tableId, ReadSessionCreator readSessionCreator, String[] requiredColumns, String filter)
long	getNumBytes(com.google.cloud.bigquery.TableDefinition tableDefinition)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigQueryRDDFactory

public BigQueryRDDFactory(BigQueryClient bigQueryClient,
                          BigQueryClientFactory bigQueryReadClientFactory,
                          BigQueryTracerFactory bigQueryTracerFactory,
                          SparkBigQueryConfig options,
                          org.apache.spark.sql.SQLContext sqlContext)

Method Detail

buildScanFromSQL

public org.apache.spark.rdd.RDD<org.apache.spark.sql.catalyst.InternalRow> buildScanFromSQL(String sql)

Creates RDD from the SQL string that is passed in. This functionality is invoked from the query pushdown module

createRddFromTable

public org.apache.spark.rdd.RDD<?> createRddFromTable(com.google.cloud.bigquery.TableId tableId,
                                                      ReadSessionCreator readSessionCreator,
                                                      String[] requiredColumns,
                                                      String filter)

getNumBytes

public long getNumBytes(com.google.cloud.bigquery.TableDefinition tableDefinition)