Class BaseSegmentFetcher
- java.lang.Object
-
- org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher
-
- All Implemented Interfaces:
SegmentFetcher
- Direct Known Subclasses:
HttpSegmentFetcher,PinotFSSegmentFetcher
public abstract class BaseSegmentFetcher extends Object implements SegmentFetcher
Base implementation of segment fetcher with the retry logic embedded.
-
-
Field Summary
Fields Modifier and Type Field Description protected AuthProvider_authProviderprotected org.slf4j.Logger_loggerprotected int_retryCountprotected int_retryDelayScaleFactorprotected int_retryWaitMsstatic intDEFAULT_RETRY_COUNTstatic intDEFAULT_RETRY_DELAY_SCALE_FACTORstatic intDEFAULT_RETRY_WAIT_MSstatic StringRETRY_COUNT_CONFIG_KEYstatic StringRETRY_DELAY_SCALE_FACTOR_CONFIG_KEYstatic StringRETRY_WAIT_MS_CONFIG_KEY
-
Constructor Summary
Constructors Constructor Description BaseSegmentFetcher()
-
Method Summary
Modifier and Type Method Description protected voiddoInit(PinotConfiguration config)Override this for custom initialization.voidfetchSegmentToLocal(URI uri, File dest)Fetches a segment from URI location to local.voidfetchSegmentToLocal(List<URI> uris, File dest)Fetches a segment to local from any uri in the given list.protected voidfetchSegmentToLocalWithoutRetry(URI uri, File dest)Fetches a segment from URI location to local without retry.FilefetchUntarSegmentToLocalStreamed(URI uri, File dest, long rateLimit)Fetches a segment from URI location and untar to local in a streamed mannervoidinit(PinotConfiguration config)Initializes the segment fetcher.
-
-
-
Field Detail
-
RETRY_COUNT_CONFIG_KEY
public static final String RETRY_COUNT_CONFIG_KEY
- See Also:
- Constant Field Values
-
RETRY_WAIT_MS_CONFIG_KEY
public static final String RETRY_WAIT_MS_CONFIG_KEY
- See Also:
- Constant Field Values
-
RETRY_DELAY_SCALE_FACTOR_CONFIG_KEY
public static final String RETRY_DELAY_SCALE_FACTOR_CONFIG_KEY
- See Also:
- Constant Field Values
-
DEFAULT_RETRY_COUNT
public static final int DEFAULT_RETRY_COUNT
- See Also:
- Constant Field Values
-
DEFAULT_RETRY_WAIT_MS
public static final int DEFAULT_RETRY_WAIT_MS
- See Also:
- Constant Field Values
-
DEFAULT_RETRY_DELAY_SCALE_FACTOR
public static final int DEFAULT_RETRY_DELAY_SCALE_FACTOR
- See Also:
- Constant Field Values
-
_logger
protected final org.slf4j.Logger _logger
-
_retryCount
protected int _retryCount
-
_retryWaitMs
protected int _retryWaitMs
-
_retryDelayScaleFactor
protected int _retryDelayScaleFactor
-
_authProvider
protected AuthProvider _authProvider
-
-
Method Detail
-
init
public void init(PinotConfiguration config)
Description copied from interface:SegmentFetcherInitializes the segment fetcher.- Specified by:
initin interfaceSegmentFetcher
-
doInit
protected void doInit(PinotConfiguration config)
Override this for custom initialization.
-
fetchSegmentToLocal
public void fetchSegmentToLocal(URI uri, File dest) throws Exception
Description copied from interface:SegmentFetcherFetches a segment from URI location to local.- Specified by:
fetchSegmentToLocalin interfaceSegmentFetcher- Throws:
Exception
-
fetchSegmentToLocal
public void fetchSegmentToLocal(List<URI> uris, File dest) throws Exception
Description copied from interface:SegmentFetcherFetches a segment to local from any uri in the given list.- Specified by:
fetchSegmentToLocalin interfaceSegmentFetcher- Throws:
Exception
-
fetchUntarSegmentToLocalStreamed
public File fetchUntarSegmentToLocalStreamed(URI uri, File dest, long rateLimit) throws Exception
Description copied from interface:SegmentFetcherFetches a segment from URI location and untar to local in a streamed manner- Specified by:
fetchUntarSegmentToLocalStreamedin interfaceSegmentFetcher- Throws:
Exception
-
fetchSegmentToLocalWithoutRetry
protected void fetchSegmentToLocalWithoutRetry(URI uri, File dest) throws Exception
Fetches a segment from URI location to local without retry. Sub-class should override this orfetchSegmentToLocal(URI, File).- Throws:
Exception
-
-