Package org.apache.beam.sdk.io.gcp.bigquery

Interface BigQueryServices.DatasetService

  • All Superinterfaces:
    java.lang.AutoCloseable
    All Known Implementing Classes:
    FakeDatasetService
    Enclosing interface:
    BigQueryServices
    public static interface BigQueryServices.DatasetService
extends java.lang.AutoCloseable
    An interface to get, create and delete Cloud BigQuery datasets and tables.

    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse> commitWriteStreams​(java.lang.String tableUrn, java.lang.Iterable<java.lang.String> writeStreamNames)
      Commit write streams of type PENDING.
      void createDataset​(java.lang.String projectId, java.lang.String datasetId, @Nullable java.lang.String location, @Nullable java.lang.String description, @Nullable java.lang.Long defaultTableExpirationMs)
      Create a Dataset with the given location, description and default expiration time for tables in the dataset (if null, tables don't expire).
      void createTable​(com.google.api.services.bigquery.model.Table table)
      Creates the specified table if it does not exist.
      com.google.cloud.bigquery.storage.v1.WriteStream createWriteStream​(java.lang.String tableUrn, com.google.cloud.bigquery.storage.v1.WriteStream.Type type)
      Create a Write Stream for use with the Storage Write API.
      void deleteDataset​(java.lang.String projectId, java.lang.String datasetId)
      Deletes the dataset specified by the datasetId value.
      void deleteTable​(com.google.api.services.bigquery.model.TableReference tableRef)
      Deletes the table specified by tableId from the dataset.
      com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FinalizeWriteStreamResponse> finalizeWriteStream​(java.lang.String streamName)
      Finalize a write stream.
      com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse> flush​(java.lang.String streamName, long flushOffset)
      Flush a given stream up to the given offset.
      com.google.api.services.bigquery.model.Dataset getDataset​(java.lang.String projectId, java.lang.String datasetId)
      Gets the specified Dataset resource by dataset ID.
      BigQueryServices.StreamAppendClient getStreamAppendClient​(java.lang.String streamName, com.google.protobuf.Descriptors.Descriptor descriptor, boolean useConnectionPool)
      Create an append client for a given Storage API write stream.
      @Nullable com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef)
      Gets the specified Table resource by table ID.
      @Nullable com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef, java.util.List<java.lang.String> selectedFields)  
      @Nullable com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef, java.util.List<java.lang.String> selectedFields, BigQueryServices.DatasetService.TableMetadataView view)  
      <T> long insertAll​(com.google.api.services.bigquery.model.TableReference ref, java.util.List<org.apache.beam.sdk.values.FailsafeValueInSingleWindow<com.google.api.services.bigquery.model.TableRow,​com.google.api.services.bigquery.model.TableRow>> rowList, @Nullable java.util.List<java.lang.String> insertIdList, InsertRetryPolicy retryPolicy, java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<T>> failedInserts, ErrorContainer<T> errorContainer, boolean skipInvalidRows, boolean ignoreUnknownValues, boolean ignoreInsertIds, java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<com.google.api.services.bigquery.model.TableRow>> successfulRows)
      Inserts TableRows with the specified insertIds if not null.
      boolean isTableEmpty​(com.google.api.services.bigquery.model.TableReference tableRef)
      Returns true if the table is empty.
      com.google.api.services.bigquery.model.Table patchTableDescription​(com.google.api.services.bigquery.model.TableReference tableReference, @Nullable java.lang.String tableDescription)
      Patch BigQuery Table description.

      • Methods inherited from interface java.lang.AutoCloseable

        close

    • Method Detail

      • getTable

        @Nullable com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef)
                                                         throws java.lang.InterruptedException,
                                                                java.io.IOException
        Gets the specified Table resource by table ID.

        Returns null if the table is not found.

        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • getTable

        @Nullable com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef,
                                                                java.util.List<java.lang.String> selectedFields)
                                                         throws java.lang.InterruptedException,
                                                                java.io.IOException
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • getTable

        @Nullable com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef,
                                                                java.util.List<java.lang.String> selectedFields,
                                                                BigQueryServices.DatasetService.TableMetadataView view)
                                                         throws java.lang.InterruptedException,
                                                                java.io.IOException
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • createTable

        void createTable​(com.google.api.services.bigquery.model.Table table)
          throws java.lang.InterruptedException,
                 java.io.IOException
        Creates the specified table if it does not exist.
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • deleteTable

        void deleteTable​(com.google.api.services.bigquery.model.TableReference tableRef)
          throws java.io.IOException,
                 java.lang.InterruptedException
        Deletes the table specified by tableId from the dataset. If the table contains data, all the data will be deleted.
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • isTableEmpty

        boolean isTableEmpty​(com.google.api.services.bigquery.model.TableReference tableRef)
              throws java.io.IOException,
                     java.lang.InterruptedException
        Returns true if the table is empty.
        Throws:
        java.io.IOException - if the table is not found.
        java.lang.InterruptedException

      • getDataset

        com.google.api.services.bigquery.model.Dataset getDataset​(java.lang.String projectId,
                                                          java.lang.String datasetId)
                                                   throws java.io.IOException,
                                                          java.lang.InterruptedException
        Gets the specified Dataset resource by dataset ID.
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • createDataset

        void createDataset​(java.lang.String projectId,
                   java.lang.String datasetId,
                   @Nullable java.lang.String location,
                   @Nullable java.lang.String description,
                   @Nullable java.lang.Long defaultTableExpirationMs)
            throws java.io.IOException,
                   java.lang.InterruptedException
        Create a Dataset with the given location, description and default expiration time for tables in the dataset (if null, tables don't expire).
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • deleteDataset

        void deleteDataset​(java.lang.String projectId,
                   java.lang.String datasetId)
            throws java.io.IOException,
                   java.lang.InterruptedException
        Deletes the dataset specified by the datasetId value.

        Before you can delete a dataset, you must delete all its tables.

        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • insertAll

        <T> long insertAll​(com.google.api.services.bigquery.model.TableReference ref,
                   java.util.List<org.apache.beam.sdk.values.FailsafeValueInSingleWindow<com.google.api.services.bigquery.model.TableRow,​com.google.api.services.bigquery.model.TableRow>> rowList,
                   @Nullable java.util.List<java.lang.String> insertIdList,
                   InsertRetryPolicy retryPolicy,
                   java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<T>> failedInserts,
                   ErrorContainer<T> errorContainer,
                   boolean skipInvalidRows,
                   boolean ignoreUnknownValues,
                   boolean ignoreInsertIds,
                   java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<com.google.api.services.bigquery.model.TableRow>> successfulRows)
            throws java.io.IOException,
                   java.lang.InterruptedException
        Inserts TableRows with the specified insertIds if not null.

        If any insert fail permanently according to the retry policy, those rows are added to failedInserts.

        Returns the total bytes count of TableRows.

        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • patchTableDescription

        com.google.api.services.bigquery.model.Table patchTableDescription​(com.google.api.services.bigquery.model.TableReference tableReference,
                                                                   @Nullable java.lang.String tableDescription)
                                                            throws java.io.IOException,
                                                                   java.lang.InterruptedException
        Patch BigQuery Table description.
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • createWriteStream

        com.google.cloud.bigquery.storage.v1.WriteStream createWriteStream​(java.lang.String tableUrn,
                                                                   com.google.cloud.bigquery.storage.v1.WriteStream.Type type)
                                                            throws java.io.IOException,
                                                                   java.lang.InterruptedException
        Create a Write Stream for use with the Storage Write API.
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • getStreamAppendClient

        BigQueryServices.StreamAppendClient getStreamAppendClient​(java.lang.String streamName,
                                                          com.google.protobuf.Descriptors.Descriptor descriptor,
                                                          boolean useConnectionPool)
                                                   throws java.lang.Exception
        Create an append client for a given Storage API write stream. The stream must be created first.
        Throws:
        java.lang.Exception

      • flush

        com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse> flush​(java.lang.String streamName,
                                                                                            long flushOffset)
                                                                                     throws java.io.IOException,
                                                                                            java.lang.InterruptedException
        Flush a given stream up to the given offset. The stream must have type BUFFERED.
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • finalizeWriteStream

        com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FinalizeWriteStreamResponse> finalizeWriteStream​(java.lang.String streamName)
        Finalize a write stream. After finalization, no more records can be appended to the stream.

      • commitWriteStreams

        com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse> commitWriteStreams​(java.lang.String tableUrn,
                                                                                                                       java.lang.Iterable<java.lang.String> writeStreamNames)
        Commit write streams of type PENDING. The streams must be finalized before committing.