org.apache.samza.system.hdfs

Class HdfsSystemConsumer

java.lang.Object

org.apache.samza.util.BlockingEnvelopeMap

org.apache.samza.system.hdfs.HdfsSystemConsumer

All Implemented Interfaces:

org.apache.samza.system.SystemConsumer

public class HdfsSystemConsumer
extends org.apache.samza.util.BlockingEnvelopeMap

The system consumer for HDFS, extending the BlockingEnvelopeMap. Events will be parsed from HDFS and placed into a blocking queue in BlockingEnvelopeMap. There will be one MultiFileHdfsReader for each SystemStreamPartition, each MultiFileHdfsReader is running within its own thread. ┌───────────────────────────────────────┐ ┌─────────────────────┐ │ │ │ │ │ MultiFileHdfsReader_1 - Thread1 │───────────▶│ SSP1-BlockingQueue ├──────┐ │ │ │ │ │ └───────────────────────────────────────┘ └─────────────────────┘ │ │ ┌───────────────────────────────────────┐ ┌─────────────────────┐ │ │ │ │ │ │ │ MultiFileHdfsReader_2 - Thread2 │───────────▶│ SSP2-BlockingQueue ├──────┤ ┌──────────────────────────┐ │ │ │ │ ├───────▶│ │ └───────────────────────────────────────┘ └─────────────────────┘ └───────▶│ SystemConsumer.poll() │ ┌───────▶│ │ │ └──────────────────────────┘ ... ... │ │ │ ┌───────────────────────────────────────┐ ┌─────────────────────┐ │ │ │ │ │ │ │ MultiFileHdfsReader_N - ThreadN │───────────▶│ SSPN-BlockingQueue ├──────┘ │ │ │ │ └───────────────────────────────────────┘ └─────────────────────┘ Since each thread has only one reader and has its own blocking queue, there are essentially no communication among reader threads. Thread safety between reader threads and Samza main thread is guaranteed by the blocking queues stand in the middle.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HdfsSystemConsumer.HdfsSystemConsumerMetrics

Nested classes/interfaces inherited from class org.apache.samza.util.BlockingEnvelopeMap

org.apache.samza.util.BlockingEnvelopeMap.BlockingEnvelopeMapMetrics, org.apache.samza.util.BlockingEnvelopeMap.BufferGauge, org.apache.samza.util.BlockingEnvelopeMap.BufferSizeGauge

Field Summary

Fields inherited from interface org.apache.samza.system.SystemConsumer

BLOCK_ON_OUTSTANDING_MESSAGES

Constructor Summary

Constructors

Constructor and Description
HdfsSystemConsumer(java.lang.String systemName, org.apache.samza.config.Config config, HdfsSystemConsumer.HdfsSystemConsumerMetrics consumerMetrics)

Method Summary

Modifier and Type	Method and Description
protected java.util.concurrent.BlockingQueue<org.apache.samza.system.IncomingMessageEnvelope>	newBlockingQueue()
java.util.Map<org.apache.samza.system.SystemStreamPartition,java.util.List<org.apache.samza.system.IncomingMessageEnvelope>>	poll(java.util.Set<org.apache.samza.system.SystemStreamPartition> systemStreamPartitions, long timeout)
void	register(org.apache.samza.system.SystemStreamPartition systemStreamPartition, java.lang.String offset)
void	start()
void	stop()

Methods inherited from class org.apache.samza.util.BlockingEnvelopeMap

getMessagesSizeInQueue, getNumMessagesInQueue, isAtHead, put, putAll, setFailureCause, setIsAtHead

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HdfsSystemConsumer

public HdfsSystemConsumer(java.lang.String systemName,
                          org.apache.samza.config.Config config,
                          HdfsSystemConsumer.HdfsSystemConsumerMetrics consumerMetrics)

Method Detail

start

public void start()

stop

public void stop()

newBlockingQueue

protected java.util.concurrent.BlockingQueue<org.apache.samza.system.IncomingMessageEnvelope> newBlockingQueue()

Overrides:

newBlockingQueue in class org.apache.samza.util.BlockingEnvelopeMap

register

public void register(org.apache.samza.system.SystemStreamPartition systemStreamPartition,
                     java.lang.String offset)

Specified by:

register in interface org.apache.samza.system.SystemConsumer

Overrides:

register in class org.apache.samza.util.BlockingEnvelopeMap

poll

public java.util.Map<org.apache.samza.system.SystemStreamPartition,java.util.List<org.apache.samza.system.IncomingMessageEnvelope>> poll(java.util.Set<org.apache.samza.system.SystemStreamPartition> systemStreamPartitions,
                                                                                                                                         long timeout)
                                                                                                                                  throws java.lang.InterruptedException

Specified by:

poll in interface org.apache.samza.system.SystemConsumer

Overrides:

poll in class org.apache.samza.util.BlockingEnvelopeMap

Throws:

java.lang.InterruptedException