Package org.apache.pinot.common.utils.fetcher

Class SegmentFetcherFactory

java.lang.Object

org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory

public class SegmentFetcherFactory
extends Object

Method Summary

Modifier and Type	Method	Description
static void	fetchAndDecryptSegmentToLocal(String uri, File dest, String crypterName)	Fetches a segment from a URI location to a local file and decrypts it if needed
static void	fetchAndDecryptSegmentToLocal(List<URI> uris, File dest, String crypterName)
static File	fetchAndStreamUntarToLocal(String uri, File tempRootDir, long maxStreamRateInByte)	Fetches a segment from URI location to local and untar it in a streamed manner.
static void	fetchSegmentToLocal(String uri, File dest)	Fetches a segment from URI location to local.
static void	fetchSegmentToLocal(URI uri, File dest)	Fetches a segment from URI location to local.
static SegmentFetcherFactory	getInstance()
static SegmentFetcher	getSegmentFetcher(String protocol)	Returns the segment fetcher associated with the given protocol, or the default segment fetcher (HttpSegmentFetcher for "http" and "https", PinotFSSegmentFetcher for other protocols).
static void	init(PinotConfiguration config)	Initializes the segment fetcher factory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getInstance

public static SegmentFetcherFactory getInstance()

init

public static void init(PinotConfiguration config)
                 throws Exception

Initializes the segment fetcher factory. This method should only be called once.

Throws:

Exception

getSegmentFetcher

public static SegmentFetcher getSegmentFetcher(String protocol)

Returns the segment fetcher associated with the given protocol, or the default segment fetcher (HttpSegmentFetcher for "http" and "https", PinotFSSegmentFetcher for other protocols).

fetchSegmentToLocal

public static void fetchSegmentToLocal(URI uri,
                                       File dest)
                                throws Exception

Fetches a segment from URI location to local.

Throws:

Exception

fetchSegmentToLocal

public static void fetchSegmentToLocal(String uri,
                                       File dest)
                                throws Exception

Fetches a segment from URI location to local.

Throws:

Exception

fetchAndStreamUntarToLocal

public static File fetchAndStreamUntarToLocal(String uri,
                                              File tempRootDir,
                                              long maxStreamRateInByte)
                                       throws Exception

Fetches a segment from URI location to local and untar it in a streamed manner.

Parameters:

uri - URI

tempRootDir - Tmp dir to download

maxStreamRateInByte - limit the rate to write download-untar stream to disk, in bytes -1 for no disk write limit, 0 for limit the writing to min(untar, download) rate

Returns:

the untared directory

Throws:

Exception

fetchAndDecryptSegmentToLocal

public static void fetchAndDecryptSegmentToLocal(String uri,
                                                 File dest,
                                                 String crypterName)
                                          throws Exception

Fetches a segment from a URI location to a local file and decrypts it if needed

Parameters:

uri - remote segment location

dest - local file

Throws:

Exception

fetchAndDecryptSegmentToLocal

public static void fetchAndDecryptSegmentToLocal(List<URI> uris,
                                                 File dest,
                                                 String crypterName)
                                          throws Exception

Throws:

Exception