com.google.cloud.bigtable.beam

Class CloudBigtableIO.SourceWithKeys

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.sdk.io.BoundedSource<Result>

com.google.cloud.bigtable.beam.CloudBigtableIO.SourceWithKeys

All Implemented Interfaces:

Serializable, HasDisplayData

Enclosing class:

CloudBigtableIO

protected static class CloudBigtableIO.SourceWithKeys
extends BoundedSource<Result>

A BoundedSource for a Cloud Bigtable Table with a start/stop key range, along with a potential filter via a Scan.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.BoundedSource

BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

Source.Reader<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected static long	SIZED_BASED_MAX_SPLIT_COUNT
protected static org.slf4j.Logger	SOURCE_LOG

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	SourceWithKeys(CloudBigtableScanConfiguration configuration, long estimatedSize)

Method Summary

Modifier and Type	Method and Description
protected long	calculateEstimatedSizeBytes(PipelineOptions options)
BoundedSource.BoundedReader<Result>	createReader(PipelineOptions options) Creates a reader that will scan the entire table based on the Scan in the configuration.
protected CloudBigtableScanConfiguration	getConfiguration()
long	getEstimatedSize()
long	getEstimatedSizeBytes(PipelineOptions options) Gets an estimated size based on data returned from getSampleRowKeys().
Coder<Result>	getOutputCoder()
List<com.google.bigtable.repackaged.com.google.cloud.bigtable.data.v2.models.KeyOffset>	getSampleRowKeys() Performs a call to get sample row keys from CloudBigtableServiceImpl.getSampleRowKeys(CloudBigtableTableConfiguration) if they are not yet cached.
protected List<CloudBigtableIO.SourceWithKeys>	getSplits(long desiredBundleSizeBytes)
protected static boolean	isWithinRange(byte[] scanStartKey, byte[] scanEndKey, byte[] startKey, byte[] endKey) Checks if the range of the region is within the range of the scan.
void	populateDisplayData(DisplayData.Builder builder)
protected List<CloudBigtableIO.SourceWithKeys>	split(long regionSize, long desiredBundleSizeBytes, byte[] startKey, byte[] stopKey) Splits the region based on the start and stop key.
List<? extends BoundedSource<Result>>	split(long desiredBundleSizeBytes, PipelineOptions options) Splits the bundle based on the assumption that the data is distributed evenly between startKey and stopKey.
String	toString()
void	validate() Validates the existence of the table in the configuration.

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

SOURCE_LOG

protected static final org.slf4j.Logger SOURCE_LOG

SIZED_BASED_MAX_SPLIT_COUNT

protected static final long SIZED_BASED_MAX_SPLIT_COUNT

See Also:

Constant Field Values

Constructor Detail

SourceWithKeys

protected SourceWithKeys(CloudBigtableScanConfiguration configuration,
                         long estimatedSize)

Method Detail

calculateEstimatedSizeBytes

protected long calculateEstimatedSizeBytes(PipelineOptions options)
                                    throws IOException

Throws:

IOException

getEstimatedSize

public long getEstimatedSize()

split

public List<? extends BoundedSource<Result>> split(long desiredBundleSizeBytes,
                                                   PipelineOptions options)
                                            throws Exception

Splits the bundle based on the assumption that the data is distributed evenly between startKey and stopKey. That assumption may not be correct for any specific start/stop key combination.

This method is called internally by Beam. Do not call it directly.

Specified by:

split in class BoundedSource<Result>

Parameters:

desiredBundleSizeBytes - The desired size for each bundle, in bytes.

options - The pipeline options.

Returns:

A list of sources split into groups.

Throws:

Exception

getOutputCoder

public Coder<Result> getOutputCoder()

toString

public String toString()

Overrides:

toString in class Object

getSplits

protected List<CloudBigtableIO.SourceWithKeys> getSplits(long desiredBundleSizeBytes)
                                                  throws Exception

Throws:

Exception

isWithinRange

protected static boolean isWithinRange(byte[] scanStartKey,
                                       byte[] scanEndKey,
                                       byte[] startKey,
                                       byte[] endKey)

Checks if the range of the region is within the range of the scan.

getSampleRowKeys

@InternalApi(value="For internal usage only")
public List<com.google.bigtable.repackaged.com.google.cloud.bigtable.data.v2.models.KeyOffset> getSampleRowKeys()
                                                                                                                                                       throws IOException

Performs a call to get sample row keys from CloudBigtableServiceImpl.getSampleRowKeys(CloudBigtableTableConfiguration) if they are not yet cached. The sample row keys give information about tablet key boundaries and estimated sizes.

For internal use only - public for technical reasons.

Throws:

IOException

validate

public void validate()

Validates the existence of the table in the configuration.

Overrides:

validate in class Source<Result>

getEstimatedSizeBytes

public long getEstimatedSizeBytes(PipelineOptions options)
                           throws IOException

Gets an estimated size based on data returned from getSampleRowKeys(). The estimate will be high if a Scan is set on the CloudBigtableScanConfiguration; in such cases, the estimate will not take the Scan into account, and will return a larger estimate than what the CloudBigtableIO.Reader will actually read.

Specified by:

getEstimatedSizeBytes in class BoundedSource<Result>

Parameters:

options - The pipeline options.

Returns:

The estimated size of the data, in bytes.

Throws:

IOException

split

protected List<CloudBigtableIO.SourceWithKeys> split(long regionSize,
                                                     long desiredBundleSizeBytes,
                                                     byte[] startKey,
                                                     byte[] stopKey)
                                              throws IOException

Splits the region based on the start and stop key. Uses Bytes.split(byte[], byte[], int) under the covers.

Throws:

IOException

createReader

public BoundedSource.BoundedReader<Result> createReader(PipelineOptions options)

Creates a reader that will scan the entire table based on the Scan in the configuration.

Specified by:

createReader in class BoundedSource<Result>

Returns:

A reader for the table.

getConfiguration

protected CloudBigtableScanConfiguration getConfiguration()

Returns:

the configuration

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class Source<Result>