package bigquery
Main package for BigQuery APIs. Import all.
import com.spotify.scio.bigquery._There are two BigQuery dialects,
legacy and
standard.
APIs that take a BigQuery query string as argument, e.g.
com.spotify.scio.bigquery.client.BigQuery.query.rows,
com.spotify.scio.bigquery.client.BigQuery.query.schema,
com.spotify.scio.bigquery.client.BigQuery.getTypedRows and
BigQueryType.fromQuery, automatically
detects the query's dialect. To override this, start the query with either #legacysql or
#standardsql comment line.
- Alphabetic
- By Inheritance
- bigquery
- AllSyntax
- ScioContextSyntax
- SCollectionSyntax
- TableReferenceSyntax
- TableRowSyntax
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Package Members
Type Members
- sealed trait BigQueryIO[T] extends ScioIO[T]
- final case class BigQuerySelect(sqlQuery: Query) extends BigQueryIO[TableRow] with Product with Serializable
Get an SCollection for a BigQuery SELECT query.
Get an SCollection for a BigQuery SELECT query. Both Legacy SQL and Standard SQL dialects are supported. By default the query dialect will be automatically detected. To override this behavior, start the query string with
#legacysqlor#standardsql. - final case class BigQueryStorage(table: Table) extends BigQueryIO[TableRow] with Product with Serializable
Get an IO for a BigQuery table using the storage API.
- final case class BigQueryStorageSelect(sqlQuery: Query) extends BigQueryIO[TableRow] with Product with Serializable
- final case class BigQueryStorageTap(table: Table, readOptions: TableReadOptions) extends Tap[TableRow] with Product with Serializable
Tap for BigQuery tables using storage api.
- final case class BigQueryTable(table: Table) extends BigQueryIO[TableRow] with Product with Serializable
Get an IO for a BigQuery table.
- type BigQueryTag = bigquery.types.BigQueryTag
BigQuery tag for macro generated classes/fields.
- final case class BigQueryTap(table: TableReference) extends Tap[TableRow] with Product with Serializable
Tap for BigQuery tables.
- final case class BigQueryTaps(self: Taps) extends Product with Serializable
- final case class BigQueryTypedSelect[T](reader: TypedRead[T], sqlQuery: Query, fromTableRow: (TableRow) => T)(implicit evidence$2: Coder[T]) extends BigQueryIO[T] with Product with Serializable
- final case class BigQueryTypedTable[T](reader: TypedRead[T], writer: Write[T], table: Table, fn: (TableRow) => T)(implicit evidence$4: Coder[T]) extends BigQueryIO[T] with Product with Serializable
- final case class BigQueryTypedTap[T](table: Table, fn: (TableRow) => T)(implicit evidence$1: Coder[T]) extends Tap[T] with Product with Serializable
- sealed trait ExtendedErrorInfo extends AnyRef
- class MockBigQuery extends AnyRef
Mock BigQuery environment for integration test.
Mock BigQuery environment for integration test.
Use mockTable to feed data into live BigQuery service and queryResult to query them.
- class MockTable extends AnyRef
A BigQuery table being mocked for test.
- final case class Query(underlying: String) extends Source with Product with Serializable
- sealed trait Source extends AnyRef
- sealed trait Table extends Source
- type TableRow = google.api.services.bigquery.model.TableRow
Alias for BigQuery
TableRow. - final case class TableRowJsonIO(path: String) extends ScioIO[TableRow] with Product with Serializable
Get an IO for a BigQuery TableRow JSON file.
- final case class TableRowJsonTap(path: String) extends Tap[TableRow] with Product with Serializable
Tap for BigQuery TableRow JSON files.
- case class TimePartitioning(type: String, field: String = null, expirationMs: Long = 0, requirePartitionFilter: Boolean = false) extends Product with Serializable
Scala wrapper for com.google.api.services.bigquery.model.TimePartitioning.
- type description = bigquery.types.description
Annotation for BigQuery field description.
Value Members
- val BigQueryType: bigquery.types.BigQueryType.type
Typed BigQuery annotations and converters.
- val CREATE_IF_NEEDED: CreateDisposition
Alias for BigQuery
CreateDisposition. - val CREATE_NEVER: CreateDisposition
Alias for BigQuery
CreateDisposition. - val WRITE_APPEND: WriteDisposition
Alias for BigQuery
WriteDisposition. - val WRITE_EMPTY: WriteDisposition
Alias for BigQuery
WriteDisposition. - val WRITE_TRUNCATE: WriteDisposition
Alias for BigQuery
WriteDisposition. - implicit def bigQuerySCollectionBeamSchemaOps[T](sc: SCollection[T])(implicit arg0: ClassTag[T]): SCollectionBeamSchemaOps[T]
- Definition Classes
- SCollectionSyntax
- implicit def bigQuerySCollectionTableRowOps[T <: TableRow](sc: SCollection[T]): SCollectionTableRowOps[T]
- Definition Classes
- SCollectionSyntax
- implicit def bigQuerySCollectionTypedOps[T <: HasAnnotation](sc: SCollection[T]): SCollectionTypedOps[T]
- Definition Classes
- SCollectionSyntax
- implicit def bigQueryScioContextOps(sc: ScioContext): ScioContextOps
- Definition Classes
- ScioContextSyntax
- implicit def bigQueryTableRefOps(r: TableReference): TableReferenceOps
- Definition Classes
- TableReferenceSyntax
- implicit def bigQueryTableRowOps(tr: TableRow): TableRowOps
- Definition Classes
- TableRowSyntax
- object BigQueryIO
- object BigQuerySelect extends Serializable
- object BigQueryStorage extends Serializable
- object BigQuerySysProps extends SysProps
- object BigQueryTable extends Serializable
- object BigQueryTaps extends Serializable
- object BigQueryTyped
- object BigQueryTypedSelect extends Serializable
- object BigQueryTypedTable extends Serializable
- object BigQueryUtil
Utility for BigQuery data types.
- object Date
Utility for BigQuery
DATEtype. - object DateTime
Utility for BigQuery
DATETIMEtype. - object ExtendedErrorInfo
- object MockBigQuery
Companion object for MockBigQuery.
- object Numeric
- object StorageUtil
Utility for BigQuery Storage API.
- object Table
- object TableRow
Create a TableRow with
Map-like syntax.Create a TableRow with
Map-like syntax. For example:val r = TableRow("name" -> "Alice", "score" -> 100)
- object TableRowJsonIO extends Serializable
- object Time
Utility for BigQuery
TIMEtype. - object Timestamp
Utility for BigQuery
TIMESTAMPtype.