org.apache.druid.segment.realtime.firehose

Class PrefetchSqlFirehoseFactory<T>

java.lang.Object

org.apache.druid.segment.realtime.firehose.PrefetchSqlFirehoseFactory<T>

All Implemented Interfaces:

FiniteFirehoseFactory<InputRowParser<Map<String,Object>>,T>, FirehoseFactory<InputRowParser<Map<String,Object>>>

Direct Known Subclasses:

SqlFirehoseFactory

public abstract class PrefetchSqlFirehoseFactory<T>
extends Object
implements FiniteFirehoseFactory<InputRowParser<Map<String,Object>>,T>

PrefetchSqlFirehoseFactory is an abstract firehose factory for reading prefetched sql resultset data. Regardless of whether prefetching is enabled or not, for each sql object the entire result set is fetched into a file in the local disk. This class defines prefetching as caching the resultsets into local disk in case multiple sql queries are present. When prefetching is enabled, the following functionalities are provided:

- Caching: for the first call of connect(InputRowParser, File), it caches objects in a local disk up to maxCacheCapacityBytes. These caches are NOT deleted until the process terminates, and thus can be used for future reads.
- Fetching: when it reads all cached data, it fetches remaining objects into a local disk and reads data from them. For the performance reason, prefetch technique is used, that is, when the size of remaining fetched data is smaller than FetchConfig.prefetchTriggerBytes, a background prefetch thread automatically starts to fetch remaining objects.

This implementation aims to avoid maintaining a persistent connection to the database by prefetching the resultset into disk.
Prefetching can be turned on/off by setting maxFetchCapacityBytes. Depending on prefetching is enabled or disabled, the behavior of the firehose is different like below.

1. If prefetch is enabled this firehose can fetch input objects in background.
2. When next() is called, it first checks that there are already fetched files in local storage.
2.1 If exists, it simply chooses a fetched file and returns a LineIterator reading that file.
2.2 If there is no fetched files in local storage but some objects are still remained to be read, the firehose fetches one of input objects in background immediately. Finally, the firehose returns an iterator of JsonIterator for deserializing the saved resultset.
3. If prefetch is disabled, the firehose saves the resultset to file and returns an iterator of JsonIterator which directly reads the stream opened by openObjectStream(T, java.io.File). If there is an IOException, it will throw it and the read will fail.

Constructor Summary

Constructors

Constructor and Description
PrefetchSqlFirehoseFactory(Long maxCacheCapacityBytes, Long maxFetchCapacityBytes, Long prefetchTriggerBytes, Long fetchTimeout, com.fasterxml.jackson.databind.ObjectMapper objectMapper)

Method Summary

Modifier and Type	Method and Description
Firehose	connect(InputRowParser<Map<String,Object>> firehoseParser, File temporaryDirectory)
long	getFetchTimeout()
long	getMaxCacheCapacityBytes()
long	getMaxFetchCapacityBytes()
int	getNumSplits(SplitHintSpec splitHintSpec)
List<T>	getObjects()
long	getPrefetchTriggerBytes()
Stream<InputSplit<T>>	getSplits(SplitHintSpec splitHintSpec)
protected void	initializeObjectsIfNeeded()
protected abstract Collection<T>	initObjects()
protected abstract InputStream	openObjectStream(T object, File filename) Open an input stream from the given object.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.data.input.FiniteFirehoseFactory

isSplittable, withSplit

Methods inherited from interface org.apache.druid.data.input.FirehoseFactory

connect, connectForSampler

Constructor Detail

PrefetchSqlFirehoseFactory

public PrefetchSqlFirehoseFactory(Long maxCacheCapacityBytes,
                                  Long maxFetchCapacityBytes,
                                  Long prefetchTriggerBytes,
                                  Long fetchTimeout,
                                  com.fasterxml.jackson.databind.ObjectMapper objectMapper)

Method Detail

getMaxCacheCapacityBytes

public long getMaxCacheCapacityBytes()

getMaxFetchCapacityBytes

public long getMaxFetchCapacityBytes()

getPrefetchTriggerBytes

public long getPrefetchTriggerBytes()

getFetchTimeout

public long getFetchTimeout()

connect

public Firehose connect(InputRowParser<Map<String,Object>> firehoseParser,
                        @Nullable
                        File temporaryDirectory)

Specified by:

connect in interface FirehoseFactory<InputRowParser<Map<String,Object>>>

initializeObjectsIfNeeded

protected void initializeObjectsIfNeeded()

getObjects

public List<T> getObjects()

getSplits

public Stream<InputSplit<T>> getSplits(@Nullable
                                       SplitHintSpec splitHintSpec)

Specified by:

getSplits in interface FiniteFirehoseFactory<InputRowParser<Map<String,Object>>,T>

getNumSplits

public int getNumSplits(@Nullable
                        SplitHintSpec splitHintSpec)

Specified by:

getNumSplits in interface FiniteFirehoseFactory<InputRowParser<Map<String,Object>>,T>

openObjectStream

protected abstract InputStream openObjectStream(T object,
                                                File filename)
                                         throws IOException

Open an input stream from the given object. The object is fetched into the file and an input stream to the file is provided.

Parameters:

object - an object to be read

filename - file to which the object is fetched into

Returns:

an input stream to the file

Throws:

IOException

initObjects

protected abstract Collection<T> initObjects()