io.druid.storage.hdfs

Class HdfsDataSegmentPuller

java.lang.Object

io.druid.storage.hdfs.HdfsDataSegmentPuller

All Implemented Interfaces:

DataSegmentPuller, URIDataPuller

Direct Known Subclasses:

HdfsFileTimestampVersionFinder

public class HdfsDataSegmentPuller
extends Object
implements DataSegmentPuller, URIDataPuller

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HdfsDataSegmentPuller.HdfsIOException FileObject.getLastModified and FileObject.delete don't throw IOException.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	config
static int	DEFAULT_RETRY_COUNT

Constructor Summary

Constructors

Constructor and Description
HdfsDataSegmentPuller(org.apache.hadoop.conf.Configuration config)

Method Summary

Modifier and Type	Method and Description
static FileObject	buildFileObject(URI uri, org.apache.hadoop.conf.Configuration config)
static FileObject	buildFileObject(URI uri, org.apache.hadoop.conf.Configuration config, Boolean overwrite)
InputStream	getInputStream(org.apache.hadoop.fs.Path path)
InputStream	getInputStream(URI uri)
void	getSegmentFiles(DataSegment segment, File dir)
com.metamx.common.FileUtils.FileCopyResult	getSegmentFiles(org.apache.hadoop.fs.Path path, File outDir)
com.metamx.common.FileUtils.FileCopyResult	getSegmentFiles(URI uri, File outDir)
String	getVersion(URI uri) Return the "version" (aka last modified timestamp) of the URI
com.google.common.base.Predicate<Throwable>	shouldRetryPredicate()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_RETRY_COUNT

public static final int DEFAULT_RETRY_COUNT

See Also:

Constant Field Values

config

protected final org.apache.hadoop.conf.Configuration config

Constructor Detail

HdfsDataSegmentPuller

@Inject
public HdfsDataSegmentPuller(org.apache.hadoop.conf.Configuration config)

Method Detail

buildFileObject

public static FileObject buildFileObject(URI uri,
                                         org.apache.hadoop.conf.Configuration config)

buildFileObject

public static FileObject buildFileObject(URI uri,
                                         org.apache.hadoop.conf.Configuration config,
                                         Boolean overwrite)

getSegmentFiles

public void getSegmentFiles(DataSegment segment,
                            File dir)
                     throws SegmentLoadingException

Specified by:

getSegmentFiles in interface DataSegmentPuller

Throws:

SegmentLoadingException

getSegmentFiles

public com.metamx.common.FileUtils.FileCopyResult getSegmentFiles(org.apache.hadoop.fs.Path path,
                                                                  File outDir)
                                                           throws SegmentLoadingException

Throws:

SegmentLoadingException

getSegmentFiles

public com.metamx.common.FileUtils.FileCopyResult getSegmentFiles(URI uri,
                                                                  File outDir)
                                                           throws SegmentLoadingException

Throws:

SegmentLoadingException

getInputStream

public InputStream getInputStream(org.apache.hadoop.fs.Path path)
                           throws IOException

Throws:

IOException

getInputStream

public InputStream getInputStream(URI uri)
                           throws IOException

Specified by:

getInputStream in interface URIDataPuller

Throws:

IOException

getVersion

public String getVersion(URI uri)
                  throws IOException

Return the "version" (aka last modified timestamp) of the URI

Specified by:

getVersion in interface URIDataPuller

Parameters:

uri - The URI of interest

Returns:

The last modified timestamp of the uri in String format

Throws:

IOException

shouldRetryPredicate

public com.google.common.base.Predicate<Throwable> shouldRetryPredicate()

Specified by:

shouldRetryPredicate in interface URIDataPuller