org.apache.samza.system.hdfs.partitioner

Class DirectoryPartitioner

java.lang.Object

org.apache.samza.system.hdfs.partitioner.DirectoryPartitioner

public class DirectoryPartitioner
extends java.lang.Object

The partitioner that takes a directory as an input and does 1. Filtering, based on a white list and a black list 2. Grouping, based on grouping pattern And then generate the partition metadata and partition descriptors This class holds the assumption that the directory remains immutable. If the directory does changes: ignore new files showing up in the directory based on an old version of partition descriptor; throw SamzaException if at least one old file doesn't exist anymore

Constructor Summary

Constructors

Constructor and Description
DirectoryPartitioner(java.lang.String whiteList, java.lang.String blackList, java.lang.String groupPattern, FileSystemAdapter fileSystemAdapter)

Method Summary

Modifier and Type	Method and Description
java.util.Map<org.apache.samza.Partition,java.util.List<java.lang.String>>	getPartitionDescriptor(java.lang.String streamName) Get partition descriptors for a stream
java.util.Map<org.apache.samza.Partition,org.apache.samza.system.SystemStreamMetadata.SystemStreamPartitionMetadata>	getPartitionMetadataMap(java.lang.String streamName, java.util.Map<org.apache.samza.Partition,java.util.List<java.lang.String>> existingPartitionDescriptorMap) Get partition metadata for a stream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DirectoryPartitioner

public DirectoryPartitioner(java.lang.String whiteList,
                            java.lang.String blackList,
                            java.lang.String groupPattern,
                            FileSystemAdapter fileSystemAdapter)

Method Detail

getPartitionMetadataMap

public java.util.Map<org.apache.samza.Partition,org.apache.samza.system.SystemStreamMetadata.SystemStreamPartitionMetadata> getPartitionMetadataMap(java.lang.String streamName,
                                                                                                                                                    @Nullable
                                                                                                                                                    java.util.Map<org.apache.samza.Partition,java.util.List<java.lang.String>> existingPartitionDescriptorMap)

Get partition metadata for a stream

Parameters:

streamName - name of the stream; should contain the information about the path of the root directory

existingPartitionDescriptorMap - map of the existing partition descriptor

Returns:

map of SSP metadata

getPartitionDescriptor

public java.util.Map<org.apache.samza.Partition,java.util.List<java.lang.String>> getPartitionDescriptor(java.lang.String streamName)

Get partition descriptors for a stream

Parameters:

streamName - name of the stream; should contain the information about the path of the root directory

Returns:

map of the partition descriptor