org.apache.beam.runners.dataflow.internal

Class IsmFormat.IsmRecordCoder<V>

java.lang.Object

org.apache.beam.sdk.coders.Coder<T>

org.apache.beam.sdk.coders.CustomCoder<IsmFormat.IsmRecord<V>>

org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder<V>

All Implemented Interfaces:

Serializable

Enclosing class:

IsmFormat

public static class IsmFormat.IsmRecordCoder<V>
extends CustomCoder<IsmFormat.IsmRecord<V>>

A Coder for IsmFormat.IsmRecords.

Note that this coder standalone will not produce an Ism file. This coder can be used to materialize a PCollection of IsmFormat.IsmRecords. Only when this coder is combined with an IsmSink will one produce an Ism file.

The IsmFormat.IsmRecord encoded format is:

• encoded key component 1 using key component coder 1
• ...
• encoded key component N using key component coder N
• encoded value using value coder

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.coders.Coder

Coder.Context, Coder.NonDeterministicException

Method Summary

Modifier and Type	Method and Description
boolean	consistentWithEquals()
IsmFormat.IsmRecord<V>	decode(InputStream inStream)
void	encode(IsmFormat.IsmRecord<V> value, OutputStream outStream)
<V,T> int	encodeAndHash(List<?> keyComponents, RandomAccessData keyBytesToMutate) Computes the shard id for the given key component(s).
<V,T> int	encodeAndHash(List<?> keyComponents, RandomAccessData keyBytesToMutate, List<Integer> keyComponentByteOffsetsToMutate) Computes the shard id for the given key component(s).
boolean	equals(Object other)
List<Coder<?>>	getCoderArguments()
<T> Coder<T>	getKeyComponentCoder(int index) Returns the key coder at the specified index.
List<Coder<?>>	getKeyComponentCoders() Returns the list of key component coders.
int	getNumberOfShardKeyCoders(List<?> keyComponents)
Coder<V>	getValueCoder() Returns the value coder.
<V,T> int	hash(List<?> keyComponents) Computes the shard id for the given key component(s).
int	hashCode()
static <V> IsmFormat.IsmRecordCoder<V>	of(int numberOfShardKeyCoders, int numberOfMetadataShardKeyCoders, List<Coder<?>> keyComponentCoders, Coder<V> valueCoder) Returns an IsmRecordCoder with the specified key component coders, value coder.
Object	structuralValue(IsmFormat.IsmRecord<V> record)
void	verifyDeterministic()

Methods inherited from class org.apache.beam.sdk.coders.Coder

decode, encode, getEncodedElementByteSize, getEncodedTypeDescriptor, isRegisterByteSizeObserverCheap, registerByteSizeObserver, verifyDeterministic, verifyDeterministic

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Method Detail

of

public static <V> IsmFormat.IsmRecordCoder<V> of(int numberOfShardKeyCoders,
                                                 int numberOfMetadataShardKeyCoders,
                                                 List<Coder<?>> keyComponentCoders,
                                                 Coder<V> valueCoder)

Returns an IsmRecordCoder with the specified key component coders, value coder.

getKeyComponentCoders

public List<Coder<?>> getKeyComponentCoders()

Returns the list of key component coders.

getKeyComponentCoder

public <T> Coder<T> getKeyComponentCoder(int index)

Returns the key coder at the specified index.

getValueCoder

public Coder<V> getValueCoder()

Returns the value coder.

encode

public void encode(IsmFormat.IsmRecord<V> value,
                   OutputStream outStream)
            throws CoderException,
                   IOException

Specified by:

encode in class Coder<IsmFormat.IsmRecord<V>>

Throws:

CoderException

IOException

decode

public IsmFormat.IsmRecord<V> decode(InputStream inStream)
                              throws CoderException,
                                     IOException

Specified by:

decode in class Coder<IsmFormat.IsmRecord<V>>

Throws:

CoderException

IOException

getNumberOfShardKeyCoders

public int getNumberOfShardKeyCoders(List<?> keyComponents)

hash

public <V,T> int hash(List<?> keyComponents)

Computes the shard id for the given key component(s).

The shard keys are encoded into their byte representations and hashed using the 32-bit murmur3 algorithm, x86 variant (little-endian variant), using 1225801234 as the seed value. We ensure that shard ids for metadata keys and normal keys do not overlap.

encodeAndHash

public <V,T> int encodeAndHash(List<?> keyComponents,
                               RandomAccessData keyBytesToMutate)

Computes the shard id for the given key component(s).

Mutates keyBytes such that when returned, contains the encoded version of the key components.

encodeAndHash

public <V,T> int encodeAndHash(List<?> keyComponents,
                               RandomAccessData keyBytesToMutate,
                               List<Integer> keyComponentByteOffsetsToMutate)

Computes the shard id for the given key component(s).

Mutates keyBytes such that when returned, contains the encoded version of the key components. Also, mutates keyComponentByteOffsetsToMutate to store the location where each key component's encoded byte representation ends within keyBytes.

getCoderArguments

public List<Coder<?>> getCoderArguments()

Overrides:

getCoderArguments in class CustomCoder<IsmFormat.IsmRecord<V>>

verifyDeterministic

public void verifyDeterministic()
                         throws Coder.NonDeterministicException

Overrides:

verifyDeterministic in class CustomCoder<IsmFormat.IsmRecord<V>>

Throws:

Coder.NonDeterministicException

consistentWithEquals

public boolean consistentWithEquals()

Overrides:

consistentWithEquals in class Coder<IsmFormat.IsmRecord<V>>

structuralValue

public Object structuralValue(IsmFormat.IsmRecord<V> record)

Overrides:

structuralValue in class Coder<IsmFormat.IsmRecord<V>>

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object