Package org.apache.pinot.common.utils.fetcher

Class BaseSegmentFetcher

java.lang.Object

org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher

All Implemented Interfaces:

SegmentFetcher

Direct Known Subclasses:

HttpSegmentFetcher, PinotFSSegmentFetcher

public abstract class BaseSegmentFetcher
extends Object
implements SegmentFetcher

Base implementation of segment fetcher with the retry logic embedded.

Field Summary

Fields

Modifier and Type	Field	Description
protected AuthProvider	_authProvider
protected org.slf4j.Logger	_logger
protected int	_retryCount
protected int	_retryDelayScaleFactor
protected int	_retryWaitMs
static int	DEFAULT_RETRY_COUNT
static int	DEFAULT_RETRY_DELAY_SCALE_FACTOR
static int	DEFAULT_RETRY_WAIT_MS
static String	RETRY_COUNT_CONFIG_KEY
static String	RETRY_DELAY_SCALE_FACTOR_CONFIG_KEY
static String	RETRY_WAIT_MS_CONFIG_KEY

Constructor Summary

Constructors

Constructor	Description
BaseSegmentFetcher()

Method Summary

Modifier and Type	Method	Description
protected void	doInit(PinotConfiguration config)	Override this for custom initialization.
void	fetchSegmentToLocal(URI uri, File dest)	Fetches a segment from URI location to local.
void	fetchSegmentToLocal(List<URI> uris, File dest)	Fetches a segment to local from any uri in the given list.
protected void	fetchSegmentToLocalWithoutRetry(URI uri, File dest)	Fetches a segment from URI location to local without retry.
File	fetchUntarSegmentToLocalStreamed(URI uri, File dest, long rateLimit)	Fetches a segment from URI location and untar to local in a streamed manner
void	init(PinotConfiguration config)	Initializes the segment fetcher.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

RETRY_COUNT_CONFIG_KEY

public static final String RETRY_COUNT_CONFIG_KEY

See Also:

Constant Field Values

RETRY_WAIT_MS_CONFIG_KEY

public static final String RETRY_WAIT_MS_CONFIG_KEY

See Also:

Constant Field Values

RETRY_DELAY_SCALE_FACTOR_CONFIG_KEY

public static final String RETRY_DELAY_SCALE_FACTOR_CONFIG_KEY

See Also:

Constant Field Values

DEFAULT_RETRY_COUNT

public static final int DEFAULT_RETRY_COUNT

See Also:

Constant Field Values

DEFAULT_RETRY_WAIT_MS

public static final int DEFAULT_RETRY_WAIT_MS

See Also:

Constant Field Values

DEFAULT_RETRY_DELAY_SCALE_FACTOR

public static final int DEFAULT_RETRY_DELAY_SCALE_FACTOR

See Also:

Constant Field Values

_logger

protected final org.slf4j.Logger _logger

_retryCount

protected int _retryCount

_retryWaitMs

protected int _retryWaitMs

_retryDelayScaleFactor

protected int _retryDelayScaleFactor

_authProvider

protected AuthProvider _authProvider

Constructor Detail

BaseSegmentFetcher

public BaseSegmentFetcher()

Method Detail

init

public void init(PinotConfiguration config)

Description copied from interface: SegmentFetcher

Initializes the segment fetcher.

Specified by:

init in interface SegmentFetcher

doInit

protected void doInit(PinotConfiguration config)

Override this for custom initialization.

fetchSegmentToLocal

public void fetchSegmentToLocal(URI uri,
                                File dest)
                         throws Exception

Description copied from interface: SegmentFetcher

Fetches a segment from URI location to local.

Specified by:

fetchSegmentToLocal in interface SegmentFetcher

Throws:

Exception

fetchSegmentToLocal

public void fetchSegmentToLocal(List<URI> uris,
                                File dest)
                         throws Exception

Description copied from interface: SegmentFetcher

Fetches a segment to local from any uri in the given list.

Specified by:

fetchSegmentToLocal in interface SegmentFetcher

Throws:

Exception

fetchUntarSegmentToLocalStreamed

public File fetchUntarSegmentToLocalStreamed(URI uri,
                                             File dest,
                                             long rateLimit)
                                      throws Exception

Description copied from interface: SegmentFetcher

Fetches a segment from URI location and untar to local in a streamed manner

Specified by:

fetchUntarSegmentToLocalStreamed in interface SegmentFetcher

Throws:

Exception

fetchSegmentToLocalWithoutRetry

protected void fetchSegmentToLocalWithoutRetry(URI uri,
                                               File dest)
                                        throws Exception

Fetches a segment from URI location to local without retry. Sub-class should override this or fetchSegmentToLocal(URI, File).

Throws:

Exception