org.apache.beam.runners.dataflow.internal

Class IsmFormat

java.lang.Object

org.apache.beam.runners.dataflow.internal.IsmFormat

public class IsmFormat
extends java.lang.Object

An Ism file is a prefix encoded composite key value file broken into shards. Each composite key is composed of a fixed number of component keys. A fixed number of those sub keys represent the shard key portion; see IsmFormat.IsmRecord and IsmFormat.IsmRecordCoder for further details around the data format. In addition to the data, there is a bloom filter, and multiple indices to allow for efficient retrieval.

An Ism file is composed of these high level sections (in order):

• shard block
• bloom filter (See ScalableBloomFilter for details on encoding format)
• shard index
• footer (See IsmFormat.Footer for details on encoding format)

The shard block is composed of multiple copies of the following:

• data block
• data index

The data block is composed of multiple copies of the following:

• key prefix (See IsmFormat.KeyPrefix for details on encoding format)
• unshared key bytes
• value bytes
• optional 0x00 0x00 bytes followed by metadata bytes (if the following 0x00 0x00 bytes are not present, then there are no metadata bytes)

Each key written into the data block must be in unsigned lexicographically increasing order and also its shard portion of the key must hash to the same shard id as all other keys within the same data block. The hashing function used is the 32-bit murmur3 algorithm, x86 variant (little-endian variant), using 1225801234 as the seed value.

The data index is composed of N copies of the following:

• key prefix (See IsmFormat.KeyPrefix for details on encoding format)
• unshared key bytes
• byte offset to key prefix in data block (variable length long coding)

The shard index is composed of a variable length integer encoding representing the number of shard index records followed by that many shard index records. See IsmFormat.IsmShardCoder for further details as to its encoding scheme.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IsmFormat.Footer The footer stores the relevant information required to locate the index and bloom filter.
static class	IsmFormat.FooterCoder A Coder for IsmFormat.Footer.
static class	IsmFormat.IsmRecord<V> A record containing a composite key and either a value or metadata.
static class	IsmFormat.IsmRecordCoder<V> A Coder for IsmFormat.IsmRecords.
static class	IsmFormat.IsmShard A shard descriptor containing shard id, the data block offset, and the index offset for the given shard.
static class	IsmFormat.IsmShardCoder A coder for IsmFormat.IsmShards.
static class	IsmFormat.KeyPrefix The prefix used before each key which contains the number of shared and unshared bytes from the previous key that was read.
static class	IsmFormat.KeyPrefixCoder A Coder for IsmFormat.KeyPrefix.
static class	IsmFormat.MetadataKeyCoder<K> A coder for metadata key component.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.beam.sdk.coders.Coder<java.util.List<IsmFormat.IsmShard>>	ISM_SHARD_INDEX_CODER A ListCoder wrapping a IsmFormat.IsmShardCoder used to encode the shard index.
static int	SHARD_BITS

Constructor Summary

Constructors

Constructor and Description
IsmFormat()

Method Summary

Modifier and Type	Method and Description
static java.lang.Object	getMetadataKey() An object representing a wild card for a key component.
static boolean	isMetadataKey(java.util.List<?> keyComponents) Returns true if and only if any of the passed in key components represent a metadata key.
static void	validateCoderIsCompatible(IsmFormat.IsmRecordCoder<?> coder) Validates that the key portion of the given coder is deterministic.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SHARD_BITS

public static final int SHARD_BITS

See Also:

Constant Field Values

ISM_SHARD_INDEX_CODER

public static final org.apache.beam.sdk.coders.Coder<java.util.List<IsmFormat.IsmShard>> ISM_SHARD_INDEX_CODER

A ListCoder wrapping a IsmFormat.IsmShardCoder used to encode the shard index. See ListCoder for its encoding specification and IsmFormat.IsmShardCoder for its encoding specification.

Constructor Detail

IsmFormat

public IsmFormat()

Method Detail

validateCoderIsCompatible

public static void validateCoderIsCompatible(IsmFormat.IsmRecordCoder<?> coder)

Validates that the key portion of the given coder is deterministic.

isMetadataKey

public static boolean isMetadataKey(java.util.List<?> keyComponents)

Returns true if and only if any of the passed in key components represent a metadata key.

getMetadataKey

public static java.lang.Object getMetadataKey()

An object representing a wild card for a key component. Encoded using IsmFormat.MetadataKeyCoder.