bigquery

Type Members

1. implicit class BigQueryDataFrameReader extends AnyRef

   

2. implicit class BigQueryDataFrameWriter[T] extends AnyRef

   

3. case class BigQueryInsertableRelation(bigQuery: BigQuery, sqlContext: SQLContext, options: SparkBigQueryOptions) extends BaseRelation with InsertableRelation with Product with Serializable

   

   Used only to insert data into BigQuery.

   Used only to insert data into BigQuery. Getting the table metadata (existence, number of records, size, etc.) is done by API calls without reading existing data

4. class BigQueryRelationProvider extends RelationProvider with CreatableRelationProvider with SchemaRelationProvider with DataSourceRegister

   

5. case class BigQueryWriteHelper(bigQuery: BigQuery, sqlContext: SQLContext, saveMode: SaveMode, options: SparkBigQueryOptions, data: DataFrame) extends Product with Serializable

   

6. class DefaultSource extends BigQueryRelationProvider

   

7. case class IntermediateDataCleaner(path: Path, conf: Configuration) extends Thread with Product with Serializable

   

   Responsible for recursively deleting the intermediate path.

   Responsible for recursively deleting the intermediate path. Implementing Thread in order to act as shutdown hook.

   path

   the path to delete

   conf

   the hadoop configuration

8. case class SparkBigQueryOptions(tableId: TableId, parentProject: String, credentials: Option[String] = None, credentialsFile: Option[String] = None, filter: Option[String] = None, schema: Option[StructType] = None, parallelism: Option[Int] = None, temporaryGcsBucket: Option[String] = None, intermediateFormat: FormatOptions = SparkBigQueryOptions.DefaultFormat, combinePushedDownFilters: Boolean = true, viewsEnabled: Boolean = false, viewMaterializationProject: Option[String] = None, viewMaterializationDataset: Option[String] = None, viewExpirationTimeInHours: Int = 24, maxReadRowsRetries: Int = 3) extends Product with Serializable

   

   Options for defining BigQueryRelations

9. case class ToIterator[E](remote: RemoteIterator[E]) extends Iterator[E] with Product with Serializable

   

   Converts HDFS RemoteIterator to Scala iterator