org.apache.druid.data.input.impl

Class AbstractTextFilesFirehoseFactory<T>

java.lang.Object

org.apache.druid.data.input.impl.AbstractTextFilesFirehoseFactory<T>

Type Parameters:

T - object type representing input data

All Implemented Interfaces:

FiniteFirehoseFactory<StringInputRowParser,T>, FirehoseFactory<StringInputRowParser>

Direct Known Subclasses:

PrefetchableTextFilesFirehoseFactory

public abstract class AbstractTextFilesFirehoseFactory<T>
extends Object
implements FiniteFirehoseFactory<StringInputRowParser,T>

This is an abstract class for firehose factory for making firehoses reading text files. It provides an unified connect(StringInputRowParser, File) implementation for its subclasses.

Constructor Summary

Constructors

Constructor and Description
AbstractTextFilesFirehoseFactory()

Method Summary

Modifier and Type	Method and Description
Firehose	connect(StringInputRowParser firehoseParser, File temporaryDirectory) Initialization method that connects up the fire hose.
int	getNumSplits() Returns number of splits returned by FiniteFirehoseFactory.getSplits().
List<T>	getObjects()
Stream<InputSplit<T>>	getSplits() Returns a Stream for InputSplits.
protected void	initializeObjectsIfNeeded()
protected abstract Collection<T>	initObjects() Initialize objects to be read by this firehose.
protected abstract InputStream	openObjectStream(T object) Open an input stream from the given object.
protected abstract InputStream	wrapObjectStream(T object, InputStream stream) Wrap the given input stream if needed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.data.input.FiniteFirehoseFactory

isSplittable, withSplit

Methods inherited from interface org.apache.druid.data.input.FirehoseFactory

connect

Constructor Detail

AbstractTextFilesFirehoseFactory

public AbstractTextFilesFirehoseFactory()

Method Detail

connect

public Firehose connect(StringInputRowParser firehoseParser,
                        File temporaryDirectory)
                 throws IOException

Description copied from interface: FirehoseFactory

Initialization method that connects up the fire hose. If this method returns successfully it should be safe to call hasMore() on the returned Firehose (which might subsequently block).

If this method returns null, then any attempt to call hasMore(), nextRow(), commit() and close() on the return value will throw a surprising NPE. Throwing IOException on connection failure or runtime exception on invalid configuration is preferred over returning null.

Some fire hoses like PrefetchableTextFilesFirehoseFactory may use a temporary directory to cache data in it.

Specified by:

connect in interface FirehoseFactory<StringInputRowParser>

Parameters:

firehoseParser - an input row parser

temporaryDirectory - a directory where temporary files are stored

Throws:

IOException

initializeObjectsIfNeeded

protected void initializeObjectsIfNeeded()
                                  throws IOException

Throws:

IOException

getObjects

public List<T> getObjects()

getSplits

public Stream<InputSplit<T>> getSplits()
                                throws IOException

Description copied from interface: FiniteFirehoseFactory

Returns a Stream for InputSplits. In parallel batch indexing, each InputSplit is processed by a sub task. Listing splits may cause high overhead in some implementations. In this case, InputSplits should be listed lazily so that the listing overhead could be amortized.

Specified by:

getSplits in interface FiniteFirehoseFactory<StringInputRowParser,T>

Throws:

IOException

getNumSplits

public int getNumSplits()
                 throws IOException

Description copied from interface: FiniteFirehoseFactory

Returns number of splits returned by FiniteFirehoseFactory.getSplits().

Specified by:

getNumSplits in interface FiniteFirehoseFactory<StringInputRowParser,T>

Throws:

IOException

initObjects

protected abstract Collection<T> initObjects()
                                      throws IOException

Initialize objects to be read by this firehose. Since firehose factories are constructed whenever org.apache.druid.indexing.common.task.Task objects are deserialized, actual initialization of objects is deferred until connect(StringInputRowParser, File) is called.

Returns:

a collection of initialized objects.

Throws:

IOException

openObjectStream

protected abstract InputStream openObjectStream(T object)
                                         throws IOException

Open an input stream from the given object. If the object is compressed, this method should return a byte stream as it is compressed. The object compression should be handled in wrapObjectStream(Object, InputStream).

Parameters:

object - an object to be read

Returns:

an input stream for the object

Throws:

IOException

wrapObjectStream

protected abstract InputStream wrapObjectStream(T object,
                                                InputStream stream)
                                         throws IOException

Wrap the given input stream if needed. The decompression logic should be applied to the given stream if the object is compressed.

Parameters:

object - an input object

stream - a stream for the object

Returns:

an wrapped input stream

Throws:

IOException