A B C D E F G H I J L M N O P Q R S T U V W

A

abort(WriterCommitMessageContext[]) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataSourceWriterContext

If not in WritingMode IGNORE_INPUTS, the BigQuery Storage Write API WriteClient is shut down.

abort() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataWriterContext

abort(WriterCommitMessageContext[]) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryIndirectDataSourceWriterContext

abort(WriterCommitMessageContext[]) - Method in interface com.google.cloud.spark.bigquery.write.context.DataSourceWriterContext

abort() - Method in interface com.google.cloud.spark.bigquery.write.context.DataWriterContext

abort() - Method in class com.google.cloud.spark.bigquery.write.context.NoOpDataWriterContext

arrow(List<String>, ByteString, Optional<StructType>, Optional<BigQueryStorageReadRowsTracer>, ReadSession.TableReadOptions.ResponseCompressionCodec) - Static method in interface com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter

Arrow(List<String>, ByteString, Optional<StructType>, Optional<BigQueryStorageReadRowsTracer>, ReadSession.TableReadOptions.ResponseCompressionCodec) - Constructor for class com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter.Arrow

ArrowBinaryIterator - Class in com.google.cloud.spark.bigquery

ArrowBinaryIterator(List<String>, ByteString, ReadRowsResponse, Optional<StructType>, Optional<BigQueryStorageReadRowsTracer>, ReadSession.TableReadOptions.ResponseCompressionCodec) - Constructor for class com.google.cloud.spark.bigquery.ArrowBinaryIterator

ArrowSchemaConverter - Class in com.google.cloud.spark.bigquery

ArrowSchemaConverter class for accessing values and converting arrow data types to the types supported by big query.

avro(Schema, List<String>, String, Optional<StructType>, Optional<BigQueryStorageReadRowsTracer>, SchemaConvertersConfiguration, ReadSession.TableReadOptions.ResponseCompressionCodec) - Static method in interface com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter

Avro(Schema, List<String>, String, Optional<StructType>, Optional<BigQueryStorageReadRowsTracer>, SchemaConvertersConfiguration, ReadSession.TableReadOptions.ResponseCompressionCodec) - Constructor for class com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter.Avro

AvroBinaryIterator - Class in com.google.cloud.spark.bigquery

AvroBinaryIterator(Schema, List<String>, Schema, ReadRowsResponse, Optional<StructType>, Optional<BigQueryStorageReadRowsTracer>, SchemaConvertersConfiguration, ReadSession.TableReadOptions.ResponseCompressionCodec) - Constructor for class com.google.cloud.spark.bigquery.AvroBinaryIterator

An iterator for scanning over rows serialized in Avro format

AvroIntermediateRecordWriter - Class in com.google.cloud.spark.bigquery.write.context

AvroSchemaConverter - Class in com.google.cloud.spark.bigquery

AvroSchemaConverter() - Constructor for class com.google.cloud.spark.bigquery.AvroSchemaConverter

B

BIGQUERY_JOB_TIMEOUT_IN_MINUTES - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

bigQueryClient - Variable in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

BigQueryConnectorUtils - Class in com.google.cloud.spark.bigquery

BigQueryDataSourceWriterInsertableRelation - Class in com.google.cloud.spark.bigquery.write

BigQueryDataSourceWriterInsertableRelation(BigQueryClient, SQLContext, SparkBigQueryConfig, Injector) - Constructor for class com.google.cloud.spark.bigquery.write.BigQueryDataSourceWriterInsertableRelation

BigQueryDataSourceWriterModule - Class in com.google.cloud.spark.bigquery.write.context

BigQueryDataSourceWriterModule(SparkBigQueryConfig, String, StructType, SaveMode) - Constructor for class com.google.cloud.spark.bigquery.write.context.BigQueryDataSourceWriterModule

BigQueryDeprecatedIndirectInsertableRelation - Class in com.google.cloud.spark.bigquery.write

The original indirect insertable relation, using Spark's write.

BigQueryDeprecatedIndirectInsertableRelation(BigQueryClient, SQLContext, SparkBigQueryConfig) - Constructor for class com.google.cloud.spark.bigquery.write.BigQueryDeprecatedIndirectInsertableRelation

BigQueryDirectDataSourceWriterContext - Class in com.google.cloud.spark.bigquery.write.context

BigQueryDirectDataSourceWriterContext(BigQueryClient, BigQueryClientFactory, TableId, String, SaveMode, StructType, RetrySettings, Optional<String>, boolean, ImmutableMap<String, String>, SchemaConvertersConfiguration, Optional<String>, boolean, PartitionOverwriteMode, SparkContext) - Constructor for class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataSourceWriterContext

BigQueryDirectDataWriterContext - Class in com.google.cloud.spark.bigquery.write.context

BigQueryDirectDataWriterContext(int, long, long, BigQueryClientFactory, String, StructType, ProtoSchema, RetrySettings, Optional<String>, boolean) - Constructor for class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataWriterContext

BigQueryDirectDataWriterContextFactory - Class in com.google.cloud.spark.bigquery.write.context

BigQueryDirectDataWriterContextFactory(BigQueryClientFactory, String, StructType, ProtoSchema, boolean, RetrySettings, Optional<String>, boolean) - Constructor for class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataWriterContextFactory

BigQueryDirectWriterCommitMessageContext - Class in com.google.cloud.spark.bigquery.write.context

BigQueryDirectWriterCommitMessageContext(String, int, long, long, String, long, long) - Constructor for class com.google.cloud.spark.bigquery.write.context.BigQueryDirectWriterCommitMessageContext

BigQueryIndirectDataSourceWriterContext - Class in com.google.cloud.spark.bigquery.write.context

A DataSourceWriter implemented by first writing the DataFrame's data into GCS in an intermediate format, and then triggering a BigQuery load job on this data.

BigQueryIndirectDataSourceWriterContext(BigQueryClient, SparkBigQueryConfig, Configuration, StructType, String, SaveMode, Path, Optional<IntermediateDataCleaner>, SparkContext) - Constructor for class com.google.cloud.spark.bigquery.write.context.BigQueryIndirectDataSourceWriterContext

BigQueryInsertableRelationBase - Class in com.google.cloud.spark.bigquery.write

BigQueryInsertableRelationBase(BigQueryClient, SQLContext, SparkBigQueryConfig) - Constructor for class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

BigQueryJobCompletedEvent - Class in com.google.cloud.spark.bigquery.events

BigQueryJobCompletedEvent(JobInfo, JobConfiguration.Type) - Constructor for class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

BigQueryPartition - Class in com.google.cloud.spark.bigquery.direct

BigQueryPartition(String, int) - Constructor for class com.google.cloud.spark.bigquery.direct.BigQueryPartition

BigQueryRDDFactory - Class in com.google.cloud.spark.bigquery.direct

Wrapper class for generating BigQueryRDD.

BigQueryRDDFactory(BigQueryClient, BigQueryClientFactory, BigQueryTracerFactory, SparkBigQueryConfig, SQLContext) - Constructor for class com.google.cloud.spark.bigquery.direct.BigQueryRDDFactory

BigQueryRelation - Class in com.google.cloud.spark.bigquery

BigQueryRelation(SparkBigQueryConfig, TableInfo, SQLContext) - Constructor for class com.google.cloud.spark.bigquery.BigQueryRelation

BigQueryWriteHelper - Class in com.google.cloud.spark.bigquery.write

BigQueryWriteHelper(BigQueryClient, SQLContext, SaveMode, SparkBigQueryConfig, Dataset<Row>, Optional<TableInfo>) - Constructor for class com.google.cloud.spark.bigquery.write.BigQueryWriteHelper

build() - Method in class com.google.cloud.spark.bigquery.InjectorBuilder

buildDescriptorProtoWithFields(DescriptorProtos.DescriptorProto.Builder, FieldList, int) - Static method in class com.google.cloud.spark.bigquery.ProtobufUtils

buildScan() - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

buildScan(String[]) - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

buildScan(String[], Filter[]) - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

buildScanFromSQL(String) - Method in class com.google.cloud.spark.bigquery.direct.BigQueryRDDFactory

Creates RDD from the SQL string that is passed in.

buildSingleRowMessage(StructType, Descriptors.Descriptor, InternalRow, Optional<Map<Integer, ProtobufUtils.ProtobufSchemaFieldCacheEntry>>, DynamicMessage.Builder) - Static method in class com.google.cloud.spark.bigquery.ProtobufUtils

C

call(Integer, Iterator<Row>) - Method in class com.google.cloud.spark.bigquery.write.DataSourceWriterContextPartitionHandler

close() - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

close() - Method in class com.google.cloud.spark.bigquery.write.context.AvroIntermediateRecordWriter

close() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataWriterContext

close() - Method in class com.google.cloud.spark.bigquery.write.context.NoOpDataWriterContext

com.google.cloud.spark.bigquery - package com.google.cloud.spark.bigquery

com.google.cloud.spark.bigquery.direct - package com.google.cloud.spark.bigquery.direct

com.google.cloud.spark.bigquery.events - package com.google.cloud.spark.bigquery.events

com.google.cloud.spark.bigquery.examples - package com.google.cloud.spark.bigquery.examples

com.google.cloud.spark.bigquery.metrics - package com.google.cloud.spark.bigquery.metrics

com.google.cloud.spark.bigquery.pushdowns - package com.google.cloud.spark.bigquery.pushdowns

com.google.cloud.spark.bigquery.util - package com.google.cloud.spark.bigquery.util

com.google.cloud.spark.bigquery.write - package com.google.cloud.spark.bigquery.write

com.google.cloud.spark.bigquery.write.context - package com.google.cloud.spark.bigquery.write.context

commit(WriterCommitMessageContext[]) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataSourceWriterContext

This function will determine, based on the WritingMode: if in IGNORE_INPUTS mode, no work is to be done; otherwise all streams will be batch committed using the BigQuery Storage Write API, and then: if in OVERWRITE mode, the overwriteDestinationWithTemporary function from BigQueryClient will be called to replace the destination table with all the data from the temporary table; if in OVERWRITE mode with dynamic partitions enabled, overwriteDestinationWithTemporaryDynamicPartitons from BigQueryClient will be called to replace the required partitions;if in ALL_ELSE mode no more work needs to be done.

commit() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataWriterContext

commit(WriterCommitMessageContext[]) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryIndirectDataSourceWriterContext

commit(WriterCommitMessageContext[]) - Method in interface com.google.cloud.spark.bigquery.write.context.DataSourceWriterContext

commit() - Method in interface com.google.cloud.spark.bigquery.write.context.DataWriterContext

commit() - Method in class com.google.cloud.spark.bigquery.write.context.NoOpDataWriterContext

compileFilter(Filter) - Static method in class com.google.cloud.spark.bigquery.SparkFilterUtils

compileFilters(Iterable<Filter>) - Static method in class com.google.cloud.spark.bigquery.SparkFilterUtils

config - Variable in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

configure(Binder) - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorModule

configure(Binder) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDataSourceWriterModule

convert(ReadRowsResponse) - Method in class com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter.Arrow

convert(ReadRowsResponse) - Method in class com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter.Avro

convert(ReadRowsResponse) - Method in interface com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter

convertToInternalRow(Schema, List<String>, GenericRecord, Optional<StructType>) - Method in class com.google.cloud.spark.bigquery.SchemaConverters

convertToRDD(Dataset<Row>) - Method in interface com.google.cloud.spark.bigquery.DataFrameToRDDConverter

CreatableRelationProviderHelper - Class in com.google.cloud.spark.bigquery.write

CreatableRelationProviderHelper() - Constructor for class com.google.cloud.spark.bigquery.write.CreatableRelationProviderHelper

create(Injector, String, StructType, SaveMode, Map<String, String>) - Static method in interface com.google.cloud.spark.bigquery.write.context.DataSourceWriterContext

createBigQueryColumn(StructField, int) - Method in class com.google.cloud.spark.bigquery.SchemaConverters

Converts a single StructField to a BigQuery Field (column).

createBigQueryInsertableRelation(SQLContext, Dataset<Row>, SaveMode, SparkBigQueryConfig) - Method in class com.google.cloud.spark.bigquery.write.CreatableRelationProviderHelper

createCredentials() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

createDataWriterContext(int, long, long) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataWriterContextFactory

If ignoreInputs is true, return a NoOpDataWriter, a stub class that performs no operations upon the call of its methods; otherwise return BigQueryDataWriter.

createDataWriterContext(int, long, long) - Method in interface com.google.cloud.spark.bigquery.write.context.DataWriterContextFactory

createDefault() - Static method in class com.google.cloud.spark.bigquery.SchemaConvertersConfiguration

createExpressionEncoder(StructType) - Method in class org.apache.spark.sql.Scala213SparkSqlUtils

createExpressionEncoder(StructType) - Method in class org.apache.spark.sql.SparkSqlUtils

createGcsPath(SparkBigQueryConfig, Configuration, String) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

Checks whether temporaryGcsBucket or persistentGcsBucket parameters are present in the config and creates a org.apache.hadoop.fs.Path object backed by GCS.

createInjector(StructType, Map<String, String>, boolean) - Static method in class com.google.cloud.spark.bigquery.InjectorFactory

createInjector(SparkSession, StructType, Map<String, String>, boolean) - Static method in class com.google.cloud.spark.bigquery.InjectorFactory

createProtoFieldBuilder(String, DescriptorProtos.FieldDescriptorProto.Label, int, DescriptorProtos.FieldDescriptorProto.Type) - Static method in class com.google.cloud.spark.bigquery.ProtobufUtils

createRddFromTable(TableId, ReadSessionCreator, String[], String) - Method in class com.google.cloud.spark.bigquery.direct.BigQueryRDDFactory

createRelation(SQLContext, SaveMode, Map<String, String>, Dataset<Row>, Map<String, String>) - Method in class com.google.cloud.spark.bigquery.write.CreatableRelationProviderHelper

createSparkBigQueryConfig(SQLContext, Map<String, String>, Option<StructType>, DataSourceVersion) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

createWriterContextFactory() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataSourceWriterContext

createWriterContextFactory() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryIndirectDataSourceWriterContext

createWriterContextFactory() - Method in interface com.google.cloud.spark.bigquery.write.context.DataSourceWriterContext

D

DataFrameToRDDConverter - Interface in com.google.cloud.spark.bigquery

DataOrigin - Enum in com.google.cloud.spark.bigquery.metrics

DataSourceVersion - Enum in com.google.cloud.spark.bigquery

A single place to put data source implementations customizations, in order to avoid a general `if(dataSource==v1) {...} else {...}` throughout the code.

DataSourceWriterContext - Interface in com.google.cloud.spark.bigquery.write.context

DataSourceWriterContextPartitionHandler - Class in com.google.cloud.spark.bigquery.write

DataSourceWriterContextPartitionHandler(DataWriterContextFactory, long) - Constructor for class com.google.cloud.spark.bigquery.write.DataSourceWriterContextPartitionHandler

DataWriterContext<T> - Interface in com.google.cloud.spark.bigquery.write.context

An internal version to Spark DataSource DataWriter interface

DataWriterContextFactory<T> - Interface in com.google.cloud.spark.bigquery.write.context

DATE_PARTITION_PARAM - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

DEFAULT_CACHE_EXPIRATION_IN_MINUTES - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

DEFAULT_JOB_PRIORITY - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

deletePath() - Method in class com.google.cloud.spark.bigquery.write.IntermediateDataCleaner

DirectBigQueryRelation - Class in com.google.cloud.spark.bigquery.direct

DirectBigQueryRelation(SparkBigQueryConfig, TableInfo, BigQueryClient, BigQueryClientFactory, BigQueryTracerFactory, SQLContext) - Constructor for class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

disable(SparkSession) - Method in interface com.google.cloud.spark.bigquery.pushdowns.SparkBigQueryPushdown

disablePushdownSession(SparkSession) - Static method in class com.google.cloud.spark.bigquery.BigQueryConnectorUtils

E

emptyRowRDDsCreated - Static variable in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

enable(SparkSession) - Method in interface com.google.cloud.spark.bigquery.pushdowns.SparkBigQueryPushdown

ENABLE_LIST_INFERENCE - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

enablePushdownSession(SparkSession) - Static method in class com.google.cloud.spark.bigquery.BigQueryConnectorUtils

equals(Object) - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

equals(Object) - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

equals(Object) - Method in class com.google.cloud.spark.bigquery.SchemaConvertersConfiguration

equals(Object) - Method in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

exists() - Method in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

Does this table exist?

extractJobLabels(SparkConf) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

extractPartitionAndClusteringFilters(TableInfo, ImmutableList<Filter>) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

F

forDescription(String) - Static method in enum com.google.cloud.spark.bigquery.SupportedCustomDataType

from(JobInfo) - Static method in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

from(SparkSession, ReadSession, long, DataFormat, DataOrigin, long) - Static method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

from(SchemaConvertersConfiguration) - Static method in class com.google.cloud.spark.bigquery.SchemaConverters

from(SparkBigQueryConfig) - Static method in class com.google.cloud.spark.bigquery.SchemaConvertersConfiguration

from(Map<String, String>, ImmutableMap<String, String>, DataSourceVersion, SparkSession, Optional<StructType>, boolean) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

from(Map<String, String>, ImmutableMap<String, String>, Configuration, ImmutableMap<String, String>, int, SQLConf, String, Optional<StructType>, boolean) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

from(String, String, SQLConf, boolean, boolean) - Static method in enum com.google.cloud.spark.bigquery.SparkBigQueryConfig.IntermediateFormat

from(String) - Static method in enum com.google.cloud.spark.bigquery.SparkBigQueryConfig.WriteMethod

from(Map<String, String>, ImmutableMap<String, String>, Configuration) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

G

GCS_CONFIG_HTTP_CONNECT_TIMEOUT_PROPERTY - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

GCS_CONFIG_HTTP_MAX_RETRY_PROPERTY - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

GCS_CONFIG_HTTP_READ_TIMEOUT_PROPERTY - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

GCS_CONFIG_PROXY_ADDRESS_PROPERTY - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

GCS_CONFIG_PROXY_PASSWORD_PROPERTY - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

GCS_CONFIG_PROXY_USERNAME_PROPERTY - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

getAccessToken() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getAccessTokenProviderConfig() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getAccessTokenProviderFQCN() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getAccumulatorNameForMetric(String, String) - Static method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryConnectorMetricsUtils

getAllowMapTypeConversion() - Method in class com.google.cloud.spark.bigquery.SchemaConvertersConfiguration

getAllowMapTypeConversion() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getArray(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getArrowCompressionCodec() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBatchSizeInBytes(ReadRowsResponse) - Method in class com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter.Arrow

getBatchSizeInBytes(ReadRowsResponse) - Method in class com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter.Avro

getBatchSizeInBytes(ReadRowsResponse) - Method in interface com.google.cloud.spark.bigquery.ReadRowsResponseToInternalRowIteratorConverter

getBigQueryClientConnectTimeout() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBigQueryClientReadTimeout() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBigQueryClientRetrySettings() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBigqueryDataWriteHelperRetrySettings() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBigQueryHttpEndpoint() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBigQueryJobLabels() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBigQueryJobTimeoutInMinutes() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBigQueryProxyConfig() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBigQueryRDDFactory() - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

getBigQueryRDDFactory() - Method in interface com.google.cloud.spark.bigquery.SupportsQueryPushdown

getBigQueryStorageGrpcEndpoint() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBigQueryTableLabels() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getBinary(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getBoolean(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getByte(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getBytesRead() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

getBytesRead() - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

getBytesReadAccumulator() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

getBytesWritten() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectWriterCommitMessageContext

getCacheExpirationTimeInMinutes() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getChannelPoolSize() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getChild(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getClusteredFields() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getCompiledFilter(boolean, DataFormat, Optional<String>, Filter...) - Static method in class com.google.cloud.spark.bigquery.SparkFilterUtils

getConnectorInfo() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorUserAgentProvider

getCreateDisposition() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getCreateReadSessionTimeoutInSeconds() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getCredentialsFile() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getCredentialsKey() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getCustomDataTypeOptional() - Method in class com.google.cloud.spark.bigquery.ProtobufUtils.ProtobufSchemaFieldCacheEntry

getDataSource() - Method in enum com.google.cloud.spark.bigquery.SparkBigQueryConfig.IntermediateFormat

getDecimal(int, int, int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getDecimalTargetTypes() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getDefaultParallelism() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getDescriptionOrCommentOfField(StructField, Optional<SupportedCustomDataType>) - Static method in class com.google.cloud.spark.bigquery.SchemaConverters

getDouble(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getEnableModeCheckForSchemaFields() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getEpochId() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectWriterCommitMessageContext

getError() - Method in interface com.google.cloud.spark.bigquery.write.context.WriterCommitMessageContext

getEtag() - Method in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

getFieldDescriptor() - Method in class com.google.cloud.spark.bigquery.ProtobufUtils.ProtobufSchemaFieldCacheEntry

getFileSuffix() - Method in enum com.google.cloud.spark.bigquery.SparkBigQueryConfig.IntermediateFormat

getFilter() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getFloat(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getFlowControlWindowBytes() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getFormatOptions() - Method in enum com.google.cloud.spark.bigquery.SparkBigQueryConfig.IntermediateFormat

getGeneratedId() - Method in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

getGpn() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getImpersonationServiceAccount() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getImpersonationServiceAccountsForGroups() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getImpersonationServiceAccountsForUsers() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getInstance() - Static method in class org.apache.spark.sql.SparkSqlUtils

getInt(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getIntermediateFormat() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getJobId() - Method in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

getJobId(SQLConf) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

getJobInfo() - Method in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

getKmsKeyName() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getLoadJobConfiguration() - Method in class com.google.cloud.spark.bigquery.events.LoadJobCompletedEvent

getLoadJobStatistics() - Method in class com.google.cloud.spark.bigquery.events.LoadJobCompletedEvent

getLoadSchemaUpdateOptions() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getLoggedInUserGroups() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getLoggedInUserName() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getLong(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getMap(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getMaterializationDataset() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getMaterializationExpirationTimeInMinutes() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getMaterializationProject() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getMaxParallelism() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getMaxReadRowsRetries() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getNestedTypeDescriptor() - Method in class com.google.cloud.spark.bigquery.ProtobufUtils.ProtobufSchemaFieldCacheEntry

getNullable() - Method in class com.google.cloud.spark.bigquery.ProtobufUtils.ProtobufSchemaFieldCacheEntry

getNumBytes(TableDefinition) - Method in class com.google.cloud.spark.bigquery.direct.BigQueryRDDFactory

getNumReadStreams() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

getParentProjectId() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getParseTime() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

getParseTime() - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

getParseTimeAccumulator() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

getPartitionExpirationMs() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPartitionField() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPartitionId() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectWriterCommitMessageContext

getPartitionOverwriteModeValue() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPartitionRange() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPartitionRequireFilter() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPartitionType() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPartitionTypeOrDefault() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPersistentGcsBucket() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPersistentGcsPath() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPreferredMinParallelism() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getProxyPassword() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

getProxyUri() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

getProxyUsername() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

getPushAllFilters() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getPushdownFilters() - Method in interface com.google.cloud.spark.bigquery.SupportsQueryPushdown

getQuery() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getQueryJobConfiguration() - Method in class com.google.cloud.spark.bigquery.events.QueryJobCompletedEvent

getQueryJobPriority() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getQueryJobStatistics() - Method in class com.google.cloud.spark.bigquery.events.QueryJobCompletedEvent

getReadDataFormat() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getResponseCompressionCodec() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getRowCount() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectWriterCommitMessageContext

getRowsRead() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

getRowsRead() - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

getRowsReadAccumulator() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

getScanTime() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

getScanTime() - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

getScanTimeAccumulator() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

getSchema() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getSchemaWithPseudoColumns(TableInfo) - Method in class com.google.cloud.spark.bigquery.SchemaConverters

Retrieves and returns BigQuery Schema from TableInfo.

getSelfLink() - Method in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

getShort(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getSnapshotTimeMillis() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getSparkDataType() - Method in enum com.google.cloud.spark.bigquery.SupportedCustomDataType

getSparkType() - Method in class com.google.cloud.spark.bigquery.ProtobufUtils.ProtobufSchemaFieldCacheEntry

getSqlType() - Method in enum com.google.cloud.spark.bigquery.SupportedCustomDataType

getStatus() - Method in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

getStream() - Method in class com.google.cloud.spark.bigquery.direct.BigQueryPartition

getTableId() - Method in class com.google.cloud.spark.bigquery.BigQueryRelation

getTableId() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getTableId() - Method in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

getTableIdWithoutThePartition() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

Returns the table id, without the added partition id if it exists.

getTableName() - Method in class com.google.cloud.spark.bigquery.BigQueryRelation

getTableNameForLogging() - Method in class com.google.cloud.spark.bigquery.BigQueryRelation

getTableNameFromOptions(Map<String, String>) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

getTablePath() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectWriterCommitMessageContext

getTaskId() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectWriterCommitMessageContext

getTemporaryGcsBucket() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getTimeInSpark() - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

getTraceId() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getTypeConverterOptional() - Method in class com.google.cloud.spark.bigquery.ProtobufUtils.ProtobufSchemaFieldCacheEntry

getTypeConverterStream() - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

getTypeMarker() - Method in enum com.google.cloud.spark.bigquery.SupportedCustomDataType

getUserAgent() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorUserAgentProvider

getUserEmail() - Method in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

getUTF8String(int) - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

getVersion() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorVersionProvider

getWriteMethod() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

getWriteStreamName() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectWriterCommitMessageContext

GPN_ATTRIBUTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

H

handledFilters(boolean, DataFormat, Filter...) - Static method in class com.google.cloud.spark.bigquery.SparkFilterUtils

handledFilters(boolean, DataFormat, Iterable<Filter>) - Static method in class com.google.cloud.spark.bigquery.SparkFilterUtils

hashCode() - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

hashCode() - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

hashCode() - Method in class com.google.cloud.spark.bigquery.SchemaConvertersConfiguration

hashCode() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

hasNext() - Method in class com.google.cloud.spark.bigquery.ArrowBinaryIterator

hasNext() - Method in class com.google.cloud.spark.bigquery.AvroBinaryIterator

hasNext() - Method in class com.google.cloud.spark.bigquery.InternalRowIterator

hasNull() - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

HdfsUtils - Class in com.google.cloud.spark.bigquery.util

HdfsUtils() - Constructor for class com.google.cloud.spark.bigquery.util.HdfsUtils

HTTP_CONNECT_TIMEOUT_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

HTTP_MAX_RETRY_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

HTTP_READ_TIMEOUT_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

I

IMPERSONATION_FOR_GROUP_PREFIX - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

IMPERSONATION_FOR_USER_PREFIX - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

IMPERSONATION_GLOBAL - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

incrementBytesReadAccumulator(long) - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

incrementBytesReadCounter(long) - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

incrementParseTimeAccumulator(long) - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

incrementRowsReadAccumulator(long) - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

incrementRowsReadCounter(long) - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

incrementScanTimeAccumulator(long) - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

index() - Method in class com.google.cloud.spark.bigquery.direct.BigQueryPartition

InjectorBuilder - Class in com.google.cloud.spark.bigquery

InjectorBuilder() - Constructor for class com.google.cloud.spark.bigquery.InjectorBuilder

InjectorFactory - Class in com.google.cloud.spark.bigquery

insert(Dataset<Row>, boolean) - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

insert(Dataset<Row>, boolean) - Method in class com.google.cloud.spark.bigquery.write.BigQueryDataSourceWriterInsertableRelation

insert(Dataset<Row>, boolean) - Method in class com.google.cloud.spark.bigquery.write.BigQueryDeprecatedIndirectInsertableRelation

INTERMEDIATE_FORMAT_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

IntermediateDataCleaner - Class in com.google.cloud.spark.bigquery.write

Responsible for recursively deleting the intermediate path.

IntermediateDataCleaner(Path, Configuration) - Constructor for class com.google.cloud.spark.bigquery.write.IntermediateDataCleaner

IntermediateRecordWriter - Interface in com.google.cloud.spark.bigquery.write.context

InternalRowIterator - Class in com.google.cloud.spark.bigquery

Iterator on InternalRow that wraps the conversion from Avro/Arrow schema to InternalRow

InternalRowIterator(Iterator<ReadRowsResponse>, ReadRowsResponseToInternalRowIteratorConverter, ReadRowsHelper, BigQueryStorageReadRowsTracer) - Constructor for class com.google.cloud.spark.bigquery.InternalRowIterator

isCombinePushedDownFilters() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

isDataFrameShowMethodInStackTrace() - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

isEmpty() - Method in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

Is this table empty? A none-existing table is considered to be empty

isHandled(boolean, Filter, DataFormat) - Static method in class com.google.cloud.spark.bigquery.SparkFilterUtils

isJson(Metadata) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

isOptimizedEmptyProjection() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

isTopLevelFieldHandled(boolean, Filter, DataFormat, Map<String, StructField>) - Static method in class com.google.cloud.spark.bigquery.SparkFilterUtils

isUseAvroLogicalTypes() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

isViewsEnabled() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

isWriteAtLeastOnce() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

J

JavaShakespeare - Class in com.google.cloud.spark.bigquery.examples

JavaShakespeare() - Constructor for class com.google.cloud.spark.bigquery.examples.JavaShakespeare

jobInfo - Variable in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

L

LoadJobCompletedEvent - Class in com.google.cloud.spark.bigquery.events

LoadJobCompletedEvent(JobInfo) - Constructor for class com.google.cloud.spark.bigquery.events.LoadJobCompletedEvent

logEvent() - Method in class com.google.cloud.spark.bigquery.events.BigQueryJobCompletedEvent

logger - Variable in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

M

main(String[]) - Static method in class com.google.cloud.spark.bigquery.examples.JavaShakespeare

MAX_TRACE_ID_LENGTH - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

metricRegistry() - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

MIN_BUFFERED_RESPONSES_PER_STREAM - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

MIN_STREAMS_PER_PARTITION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

N

needConversion() - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

newArrowSchemaConverter(ValueVector, StructField) - Static method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

next() - Method in class com.google.cloud.spark.bigquery.ArrowBinaryIterator

next() - Method in class com.google.cloud.spark.bigquery.AvroBinaryIterator

next() - Method in class com.google.cloud.spark.bigquery.InternalRowIterator

NoOpDataWriterContext - Class in com.google.cloud.spark.bigquery.write.context

NoOpDataWriterContext() - Constructor for class com.google.cloud.spark.bigquery.write.context.NoOpDataWriterContext

numNulls() - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

numReadStreams - Variable in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

O

of(boolean) - Static method in class com.google.cloud.spark.bigquery.SchemaConvertersConfiguration

of(DataType) - Static method in enum com.google.cloud.spark.bigquery.SupportedCustomDataType

onDataWriterCommit(WriterCommitMessageContext) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataSourceWriterContext

onDataWriterCommit(WriterCommitMessageContext) - Method in interface com.google.cloud.spark.bigquery.write.context.DataSourceWriterContext

onJobEnd(SparkListenerJobEnd) - Method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

optimizeLoadUriListForSpark(List<String>) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

Optimizing the URI list for BigQuery load, using the Spark specific file prefix and suffix patterns, based on BigQueryUtil.optimizeLoadUriList()

org.apache.spark.sql - package org.apache.spark.sql

P

parseSimpleTableId(SparkSession, Map<String, String>) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

PartitionOverwriteMode - Enum in com.google.cloud.spark.bigquery

partitionOverwriteModeProperty - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

partitionOverwriteModeValue - Variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

postConnectorVersion(SparkContext, String) - Static method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryConnectorMetricsUtils

postInputFormatEvent(SparkContext) - Static method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryConnectorMetricsUtils

postWriteSessionMetrics(long, SparkBigQueryConfig.WriteMethod, long, Optional<SparkBigQueryConfig.IntermediateFormat>, SparkContext) - Static method in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryConnectorMetricsUtils

ProtobufSchemaFieldCacheEntry(DataType, boolean, Descriptors.Descriptor, Descriptors.FieldDescriptor, Optional<TypeConverter>, Optional<SupportedCustomDataType>) - Constructor for class com.google.cloud.spark.bigquery.ProtobufUtils.ProtobufSchemaFieldCacheEntry

ProtobufUtils - Class in com.google.cloud.spark.bigquery

ProtobufUtils() - Constructor for class com.google.cloud.spark.bigquery.ProtobufUtils

ProtobufUtils.ProtobufSchemaFieldCacheEntry - Class in com.google.cloud.spark.bigquery

provideBigQueryJobCompletionListener() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorModule

provideDataSourceVersion() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorModule

provideDirectDataSourceWriterContext(BigQueryClient, BigQueryClientFactory, SparkSession) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDataSourceWriterModule

provideEnvironmentContext() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorModule

provideIndirectDataSourceWriterContext(BigQueryClient, SparkSession) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDataSourceWriterModule

provideSparkBigQueryConfig() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorModule

provideSparkSession() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorModule

provideUserAgentProvider() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConnectorModule

PROXY_ADDRESS_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

PROXY_PASSWORD_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

PROXY_USERNAME_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryProxyAndHttpConfig

Q

QueryJobCompletedEvent - Class in com.google.cloud.spark.bigquery.events

QueryJobCompletedEvent(JobInfo) - Constructor for class com.google.cloud.spark.bigquery.events.QueryJobCompletedEvent

R

ReadRowsResponseToInternalRowIteratorConverter - Interface in com.google.cloud.spark.bigquery

ReadRowsResponseToInternalRowIteratorConverter.Arrow - Class in com.google.cloud.spark.bigquery

ReadRowsResponseToInternalRowIteratorConverter.Avro - Class in com.google.cloud.spark.bigquery

rowsReadAccumulator - Variable in class com.google.cloud.spark.bigquery.metrics.SparkBigQueryReadSessionMetrics

rowToInternalRow(Row) - Method in class org.apache.spark.sql.Scala213SparkSqlUtils

rowToInternalRow(Row) - Method in class org.apache.spark.sql.SparkSqlUtils

run() - Method in class com.google.cloud.spark.bigquery.write.IntermediateDataCleaner

S

saveModeToWriteDisposition(SaveMode) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

Scala213SparkSqlUtils - Class in org.apache.spark.sql

Scala213SparkSqlUtils() - Constructor for class org.apache.spark.sql.Scala213SparkSqlUtils

scalaMapToJavaMap(Map<K, V>) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

schema() - Method in class com.google.cloud.spark.bigquery.BigQueryRelation

schema() - Method in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

SchemaConverters - Class in com.google.cloud.spark.bigquery

SchemaConvertersConfiguration - Class in com.google.cloud.spark.bigquery

serialize(Object) - Method in enum com.google.cloud.spark.bigquery.SupportedCustomDataType

setTableInfo(TableInfo) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataSourceWriterContext

setTableInfo(TableInfo) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryIndirectDataSourceWriterContext

setTableInfo(TableInfo) - Method in interface com.google.cloud.spark.bigquery.write.context.DataSourceWriterContext

sizeInBytes() - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

sourceName() - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

SparkBigQueryConfig - Class in com.google.cloud.spark.bigquery

SparkBigQueryConfig.IntermediateFormat - Enum in com.google.cloud.spark.bigquery

SparkBigQueryConfig.WriteMethod - Enum in com.google.cloud.spark.bigquery

SparkBigQueryConnectorMetricsUtils - Class in com.google.cloud.spark.bigquery.metrics

SparkBigQueryConnectorMetricsUtils() - Constructor for class com.google.cloud.spark.bigquery.metrics.SparkBigQueryConnectorMetricsUtils

SparkBigQueryConnectorModule - Class in com.google.cloud.spark.bigquery

SparkBigQueryConnectorModule(SparkSession, Map<String, String>, Map<String, String>, Optional<StructType>, DataSourceVersion, boolean, Optional<SparkBigQueryConfig>) - Constructor for class com.google.cloud.spark.bigquery.SparkBigQueryConnectorModule

SparkBigQueryConnectorUserAgentProvider - Class in com.google.cloud.spark.bigquery

Provides the versions of the client environment in an anonymous way.

SparkBigQueryConnectorUserAgentProvider(String, Optional<String>) - Constructor for class com.google.cloud.spark.bigquery.SparkBigQueryConnectorUserAgentProvider

SparkBigQueryConnectorVersionProvider - Class in com.google.cloud.spark.bigquery

SparkBigQueryConnectorVersionProvider(SparkContext) - Constructor for class com.google.cloud.spark.bigquery.SparkBigQueryConnectorVersionProvider

SparkBigQueryProxyAndHttpConfig - Class in com.google.cloud.spark.bigquery

SparkBigQueryPushdown - Interface in com.google.cloud.spark.bigquery.pushdowns

SparkBigQueryReadSessionMetrics - Class in com.google.cloud.spark.bigquery.metrics

SparkBigQueryUtil - Class in com.google.cloud.spark.bigquery

Spark related utilities

SparkBigQueryUtil() - Constructor for class com.google.cloud.spark.bigquery.SparkBigQueryUtil

sparkDateToBigQuery(Object) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

SparkFilterUtils - Class in com.google.cloud.spark.bigquery

SparkMetricsSource - Class in com.google.cloud.spark.bigquery.metrics

SparkMetricsSource() - Constructor for class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

sparkRowToAvroGenericData(InternalRow, StructType, Schema) - Static method in class com.google.cloud.spark.bigquery.AvroSchemaConverter

sparkSchemaToAvroSchema(StructType) - Static method in class com.google.cloud.spark.bigquery.AvroSchemaConverter

SparkSqlUtils - Class in org.apache.spark.sql

SparkSqlUtils() - Constructor for class org.apache.spark.sql.SparkSqlUtils

sparkTimestampToBigQuery(Object) - Static method in class com.google.cloud.spark.bigquery.SparkBigQueryUtil

sparkToProtoValue(Object) - Method in interface com.google.cloud.spark.bigquery.TypeConverter

sqlContext() - Method in class com.google.cloud.spark.bigquery.BigQueryRelation

sqlContext - Variable in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

sqlContext() - Method in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

SupportedCustomDataType - Enum in com.google.cloud.spark.bigquery

supportsBigQueryType(LegacySQLTypeName) - Method in interface com.google.cloud.spark.bigquery.TypeConverter

SupportsQueryPushdown - Interface in com.google.cloud.spark.bigquery

supportsScalaVersion(String) - Method in class org.apache.spark.sql.Scala213SparkSqlUtils

supportsScalaVersion(String) - Method in class org.apache.spark.sql.SparkSqlUtils

supportsSparkType(DataType) - Method in interface com.google.cloud.spark.bigquery.TypeConverter

supportsSparkVersion(String) - Method in interface com.google.cloud.spark.bigquery.pushdowns.SparkBigQueryPushdown

T

table - Variable in class com.google.cloud.spark.bigquery.write.BigQueryInsertableRelationBase

toAttributes(StructType) - Static method in class org.apache.spark.sql.Scala213SparkSqlUtils

toBigQuerySchema(StructType) - Method in class com.google.cloud.spark.bigquery.SchemaConverters

Spark ==> BigQuery Schema Converter utils:

toBigQueryType(DataType, Metadata) - Method in class com.google.cloud.spark.bigquery.SchemaConverters

toBigQueryType(DataType) - Method in interface com.google.cloud.spark.bigquery.TypeConverter

toDescriptor(StructType) - Static method in class com.google.cloud.spark.bigquery.ProtobufUtils

toJavaUtilIterable(RemoteIterator<T>) - Static method in class com.google.cloud.spark.bigquery.util.HdfsUtils

toJavaUtilIterator(RemoteIterator<T>) - Static method in class com.google.cloud.spark.bigquery.util.HdfsUtils

Converts HDFS RemoteIterator to java.util.Iterator

toProtoFieldType(DataType) - Method in interface com.google.cloud.spark.bigquery.TypeConverter

toProtoRows(StructType, InternalRow[]) - Static method in class com.google.cloud.spark.bigquery.ProtobufUtils

Spark Row --> ProtoRows converter utils: To be used by the DataWriters facing the BigQuery Storage Write API

toProtoSchema(Schema) - Static method in class com.google.cloud.spark.bigquery.ProtobufUtils

BigQuery Schema ==> ProtoSchema converter utils:

toProtoSchema(StructType) - Static method in class com.google.cloud.spark.bigquery.ProtobufUtils

toReadSessionCreatorConfig() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

toReadTableOptions() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

toSpark(Schema) - Method in class com.google.cloud.spark.bigquery.SchemaConverters

Convert a BigQuery schema to a Spark schema

toSparkType(LegacySQLTypeName) - Method in interface com.google.cloud.spark.bigquery.TypeConverter

toSqlType(DataType) - Static method in enum com.google.cloud.spark.bigquery.SupportedCustomDataType

toString() - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

toString() - Method in class com.google.cloud.spark.bigquery.SchemaConvertersConfiguration

toString() - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectWriterCommitMessageContext

TypeConverter<T> - Interface in com.google.cloud.spark.bigquery

U

unhandledFilters(Filter[]) - Method in class com.google.cloud.spark.bigquery.direct.DirectBigQueryRelation

unhandledFilters(boolean, DataFormat, Filter...) - Static method in class com.google.cloud.spark.bigquery.SparkFilterUtils

unhandledFilters(boolean, DataFormat, Iterable<Filter>) - Static method in class com.google.cloud.spark.bigquery.SparkFilterUtils

updateOptionsMap(Map<String, String>) - Method in enum com.google.cloud.spark.bigquery.DataSourceVersion

updateParseTime(long) - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

updateScanTime(long) - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

updateTableMetadataIfNeeded(StructType, SparkBigQueryConfig, BigQueryClient) - Static method in class com.google.cloud.spark.bigquery.write.BigQueryWriteHelper

updateTimeInSpark(long) - Method in class com.google.cloud.spark.bigquery.metrics.SparkMetricsSource

USE_AVRO_LOGICAL_TYPES_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

useCommitCoordinator() - Method in interface com.google.cloud.spark.bigquery.write.context.DataSourceWriterContext

useParentProjectForMetadataOperations() - Method in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

V

VALIDATE_SPARK_AVRO_PARAM - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

valueOf(String) - Static method in enum com.google.cloud.spark.bigquery.DataSourceVersion

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum com.google.cloud.spark.bigquery.metrics.DataOrigin

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum com.google.cloud.spark.bigquery.PartitionOverwriteMode

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum com.google.cloud.spark.bigquery.SparkBigQueryConfig.IntermediateFormat

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum com.google.cloud.spark.bigquery.SparkBigQueryConfig.WriteMethod

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum com.google.cloud.spark.bigquery.SupportedCustomDataType

Returns the enum constant of this type with the specified name.

values() - Static method in enum com.google.cloud.spark.bigquery.DataSourceVersion

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum com.google.cloud.spark.bigquery.metrics.DataOrigin

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum com.google.cloud.spark.bigquery.PartitionOverwriteMode

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum com.google.cloud.spark.bigquery.SparkBigQueryConfig.IntermediateFormat

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum com.google.cloud.spark.bigquery.SparkBigQueryConfig.WriteMethod

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum com.google.cloud.spark.bigquery.SupportedCustomDataType

Returns an array containing the constants of this enum type, in the order they are declared.

vector() - Method in class com.google.cloud.spark.bigquery.ArrowSchemaConverter

VIEWS_ENABLED_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

W

withConfig(SparkBigQueryConfig) - Method in class com.google.cloud.spark.bigquery.InjectorBuilder

withCustomDefaults(Map<String, String>) - Method in class com.google.cloud.spark.bigquery.InjectorBuilder

withDataSourceVersion(DataSourceVersion) - Method in class com.google.cloud.spark.bigquery.InjectorBuilder

withOptions(Map<String, String>) - Method in class com.google.cloud.spark.bigquery.InjectorBuilder

withSchema(StructType) - Method in class com.google.cloud.spark.bigquery.InjectorBuilder

withSpark(SparkSession) - Method in class com.google.cloud.spark.bigquery.InjectorBuilder

withTableIsMandatory(boolean) - Method in class com.google.cloud.spark.bigquery.InjectorBuilder

write(GenericRecord) - Method in class com.google.cloud.spark.bigquery.write.context.AvroIntermediateRecordWriter

write(InternalRow) - Method in class com.google.cloud.spark.bigquery.write.context.BigQueryDirectDataWriterContext

write(T) - Method in interface com.google.cloud.spark.bigquery.write.context.DataWriterContext

write(GenericRecord) - Method in interface com.google.cloud.spark.bigquery.write.context.IntermediateRecordWriter

write(InternalRow) - Method in class com.google.cloud.spark.bigquery.write.context.NoOpDataWriterContext

WRITE_AT_LEAST_ONCE_OPTION - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

WRITE_METHOD_PARAM - Static variable in class com.google.cloud.spark.bigquery.SparkBigQueryConfig

writeDataFrameToBigQuery() - Method in class com.google.cloud.spark.bigquery.write.BigQueryWriteHelper

WriterCommitMessageContext - Interface in com.google.cloud.spark.bigquery.write.context

A B C D E F G H I J L M N O P Q R S T U V W