org.apache.samza.system.hdfs

Class HdfsSystemAdmin

java.lang.Object

org.apache.samza.system.hdfs.HdfsSystemAdmin

All Implemented Interfaces:

org.apache.samza.system.SystemAdmin

public class HdfsSystemAdmin
extends java.lang.Object
implements org.apache.samza.system.SystemAdmin

The HDFS system admin for HdfsSystemConsumer and HdfsSystemProducer A high level overview of the HDFS producer/consumer architecture: ┌──────────────────────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────┤ HDFS │ │ Obtain │ │ │ Partition └──────┬──────────────────────▲──────┬─────────────────────────────────▲───────┘ │ Descriptors │ │ │ │ │ │ │ │ │ │ ┌─────────────▼───────┐ │ │ Filtering/ │ │ │ │ │ └───┐ Grouping └─────┐ │ │ HDFSAvroFileReader │ │ │ │ │ │ │ Persist │ │ │ │ └─────────┬───────────┘ Partition │ │ │ │ │ Descriptors │ ┌──────▼──────────────┐ ┌──────────┴──────────┐ │ │ │ │ │ │ │ │ ┌─────────┴───────────┐ │ │Directory Partitioner│ │ HDFSAvroWriter │ │ │ IFileReader │ │ │ │ │ │ │ │ │ │ └──────┬──────────────┘ └──────────┬──────────┘ │ └─────────┬───────────┘ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌─────────┴───────────┐ ┌─┴──────────┴────────┐ ┌──────────┴──────────┐ │ │ │ │ │ │ │ │ │ HDFSSystemConsumer │ │ HDFSSystemAdmin │ │ HDFSSystemProducer │ └──────────▶ │ │ │ │ │ └─────────┬───────────┘ └───────────┬─────────┘ └──────────┬──────────┘ │ │ │ └────────────────────────────────────┼────────────────────────────────────┘ │ ┌───────────────────────────────────────┴──────────────────────────────────────┐ │ │ │ HDFSSystemFactory │ │ │ └──────────────────────────────────────────────────────────────────────────────┘

Constructor Summary

Constructors

Constructor and Description
HdfsSystemAdmin(java.lang.String systemName, org.apache.samza.config.Config config)

Method Summary

Modifier and Type	Method and Description
java.util.Map<org.apache.samza.system.SystemStreamPartition,java.lang.String>	getOffsetsAfter(java.util.Map<org.apache.samza.system.SystemStreamPartition,java.lang.String> offsets)
java.util.Map<java.lang.String,org.apache.samza.system.SystemStreamMetadata>	getSystemStreamMetadata(java.util.Set<java.lang.String> streamNames) Fetch metadata from hdfs system for a set of streams.
java.lang.Integer	offsetComparator(java.lang.String offset1, java.lang.String offset2) Compare two multi-file style offset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.samza.system.SystemAdmin

clearStream, createStream, deleteMessages, getAllSystemStreams, getSSPMetadata, getSystemStreamPartitionCounts, resolveStartpointToOffset, start, stop, validateStream

Constructor Detail

HdfsSystemAdmin

public HdfsSystemAdmin(java.lang.String systemName,
                       org.apache.samza.config.Config config)

Method Detail

getOffsetsAfter

public java.util.Map<org.apache.samza.system.SystemStreamPartition,java.lang.String> getOffsetsAfter(java.util.Map<org.apache.samza.system.SystemStreamPartition,java.lang.String> offsets)

Specified by:

getOffsetsAfter in interface org.apache.samza.system.SystemAdmin

getSystemStreamMetadata

public java.util.Map<java.lang.String,org.apache.samza.system.SystemStreamMetadata> getSystemStreamMetadata(java.util.Set<java.lang.String> streamNames)

Fetch metadata from hdfs system for a set of streams. This has the potential side effect to persist partition description to the staging directory on hdfs if staging directory is not empty. See getStagingDirectory on HdfsConfig

Specified by:

getSystemStreamMetadata in interface org.apache.samza.system.SystemAdmin

Parameters:

streamNames - The streams to to fetch metadata for.

Returns:

A map from stream name to SystemStreamMetadata for each stream requested in the parameter set.

offsetComparator

public java.lang.Integer offsetComparator(java.lang.String offset1,
                                          java.lang.String offset2)

Compare two multi-file style offset. A multi-file style offset consist of both the file index as well as the offset within that file. And the format of it is: "fileIndex:offsetWithinFile" For example, "2:0", "3:127" Format of the offset within file is defined by the implementation of SingleFileHdfsReader itself.

Specified by:

offsetComparator in interface org.apache.samza.system.SystemAdmin

Parameters:

offset1 - First offset for comparison.

offset2 - Second offset for comparison.

Returns:

-1, if offset1 @lt offset2 0, if offset1 == offset2 1, if offset1 @gt offset2 null, if not comparable