-
- All Implemented Interfaces:
-
ai.platon.pulsar.common.config.Parameterized,java.lang.AutoCloseable
public class WebDriverPoolManager extends PreemptChannelSupport implements Parameterized, AutoCloseable
The web driver pool manager.
The web driver pool manager provides web drivers to run web page fetch tasks.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classWebDriverPoolManager.Companion
-
Field Summary
-
Constructor Summary
Constructors Constructor Description WebDriverPoolManager(BrowserManager browserManager, WebDriverFactory driverFactory, ImmutableConfig immutableConfig, Boolean suppressMetrics)
-
Method Summary
Modifier and Type Method Description final BooleangetIsActive()final BooleangetIsReady()final WebDriverSettingsgetDriverSettings()final DurationgetIdleTimeout()final Map<BrowserId, LoadingWebDriverPool>getWorkingDriverPools()final Map<BrowserId, LoadingWebDriverPool>getRetiredDriverPools()final Set<BrowserId>getClosedDriverPools()final InstantgetStartTime()final InstantgetLastActiveTime()final UnitsetLastActiveTime(Instant lastActiveTime)final DurationgetIdleTime()final BooleangetIsIdle()final IntegergetNumWaitingTasks()final IntegergetNumStandbyDrivers()final IntegergetNumWorkingDrivers()final IntegergetNumAvailableDriverSlots()final IntegergetNumActiveDrivers()final IntegergetNumDyingDrivers()final IntegergetNumClosedDrivers()final IntegergetMaxAllowedDyingDrivers()final MetergetNumReset()final MetergetNumTimeout()final Map<String, Gauge<out Object>>getGauges()final BrowserManagergetBrowserManager()final WebDriverFactorygetDriverFactory()final ImmutableConfiggetImmutableConfig()final BooleangetSuppressMetrics()final BooleangetHasEvent()final BooleangetIsNormal()final BooleangetIsPreempted()final StringgetName()final FetchResultrun(FetchTask task, SuspendFunction1<WebDriver, FetchResult> browseFun)Run the task. final FetchResultrun(BrowserId browserId, FetchTask task, SuspendFunction1<WebDriver, FetchResult> browseFun)TODO: consider pro-actor model insteadreactor: tell me if you can do this job pro-actor: here is a job, tell me if you finished it final Unitrun(Integer driverId, FetchTask task, SuspendFunction1<WebDriver, FetchResult> browseFun)Run the task with the driver specified. final FetchResultrun(WebDriverTask task)Run the task. final FetchResultrunCancelable(WebDriverTask task, WebDriver driver)Run the task and save the execution state, so it can be canceled by cancel and cancelAll. final LoadingWebDriverPoolcreateUnmanagedDriverPool(BrowserId browserId, Integer priority)Create a driver pool, but the driver pool is not added to workingDriverPools. final IntegerpromisedDriverCount(BrowserId browserId)Get the number of drivers which can serve tasks in the pool. final BooleanhasDriverPromise(BrowserId browserId)Check if the pool has at least one driver to serve. final BooleanisFullCapacity(BrowserId browserId)Check if the pool is on full capacity. final WebDriversubscribeDriver(BrowserId browserId)Subscribe a web driver in the pool specified by browserId, the other subscriber should not use the driver. final WebDriversubscribeDriver()Subscribe a web driver, the other subscriber should not use the driver. final BooleanisRetiredPool(BrowserId browserId)Check if a pool is retired. final WebDrivercancel(String url)Cancel the fetch task specified by url remotely. final WebDrivercancel(BrowserId browserId, String url)Cancel the fetch task specified by url remotely. final UnitcancelAll()Cancel all the fetch tasks, stop loading all pages. final UnitcancelAll(BrowserId browserId)Cancel all the fetch tasks, stop loading all pages. final UnitcloseDriverPoolGracefully(BrowserId browserId, Duration timeToWait)Find the driver pool with browserId, cancel all running tasks in it, and close all drivers in it. final Unitmaintain()Maintain all the driver pools, check and report inconsistency, illness, idleness, etc., close bad pools if necessary. final StringtakeSnapshot(BrowserId browserId, Boolean verbose)Take a snapshot about the state of the driver pools. final StringtakeSnapshot(Boolean verbose)Take a snapshot about the state of the driver pools. Unitclose()Close the web driver pool manager. StringtoString()Return a string to represent the snapshot of the status. -
Methods inherited from class ai.platon.pulsar.protocol.browser.driver.WebDriverPoolManager
formatPreemptChannelStatus, getParams, preempt, releaseLocks, whenNormal, whenNormalDeferred -
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
-
Constructor Detail
-
WebDriverPoolManager
WebDriverPoolManager(BrowserManager browserManager, WebDriverFactory driverFactory, ImmutableConfig immutableConfig, Boolean suppressMetrics)
-
-
Method Detail
-
getIsActive
final Boolean getIsActive()
-
getIsReady
final Boolean getIsReady()
-
getDriverSettings
final WebDriverSettings getDriverSettings()
-
getIdleTimeout
final Duration getIdleTimeout()
-
getWorkingDriverPools
final Map<BrowserId, LoadingWebDriverPool> getWorkingDriverPools()
-
getRetiredDriverPools
final Map<BrowserId, LoadingWebDriverPool> getRetiredDriverPools()
-
getClosedDriverPools
final Set<BrowserId> getClosedDriverPools()
-
getStartTime
final Instant getStartTime()
-
getLastActiveTime
final Instant getLastActiveTime()
-
setLastActiveTime
final Unit setLastActiveTime(Instant lastActiveTime)
-
getIdleTime
final Duration getIdleTime()
-
getNumWaitingTasks
final Integer getNumWaitingTasks()
-
getNumStandbyDrivers
final Integer getNumStandbyDrivers()
-
getNumWorkingDrivers
final Integer getNumWorkingDrivers()
-
getNumAvailableDriverSlots
final Integer getNumAvailableDriverSlots()
-
getNumActiveDrivers
final Integer getNumActiveDrivers()
-
getNumDyingDrivers
final Integer getNumDyingDrivers()
-
getNumClosedDrivers
final Integer getNumClosedDrivers()
-
getMaxAllowedDyingDrivers
final Integer getMaxAllowedDyingDrivers()
-
getNumReset
final Meter getNumReset()
-
getNumTimeout
final Meter getNumTimeout()
-
getBrowserManager
final BrowserManager getBrowserManager()
-
getDriverFactory
final WebDriverFactory getDriverFactory()
-
getImmutableConfig
final ImmutableConfig getImmutableConfig()
-
getSuppressMetrics
final Boolean getSuppressMetrics()
-
getHasEvent
final Boolean getHasEvent()
-
getIsNormal
final Boolean getIsNormal()
-
getIsPreempted
final Boolean getIsPreempted()
-
run
final FetchResult run(FetchTask task, SuspendFunction1<WebDriver, FetchResult> browseFun)
Run the task.
-
run
final FetchResult run(BrowserId browserId, FetchTask task, SuspendFunction1<WebDriver, FetchResult> browseFun)
TODO: consider pro-actor model instead
reactor: tell me if you can do this job pro-actor: here is a job, tell me if you finished it
-
run
final Unit run(Integer driverId, FetchTask task, SuspendFunction1<WebDriver, FetchResult> browseFun)
Run the task with the driver specified.
-
run
final FetchResult run(WebDriverTask task)
Run the task.
-
runCancelable
final FetchResult runCancelable(WebDriverTask task, WebDriver driver)
-
createUnmanagedDriverPool
final LoadingWebDriverPool createUnmanagedDriverPool(BrowserId browserId, Integer priority)
Create a driver pool, but the driver pool is not added to workingDriverPools.
-
promisedDriverCount
final Integer promisedDriverCount(BrowserId browserId)
Get the number of drivers which can serve tasks in the pool.
-
hasDriverPromise
final Boolean hasDriverPromise(BrowserId browserId)
Check if the pool has at least one driver to serve.
-
isFullCapacity
final Boolean isFullCapacity(BrowserId browserId)
Check if the pool is on full capacity.
-
subscribeDriver
@Beta() final WebDriver subscribeDriver(BrowserId browserId)
Subscribe a web driver in the pool specified by browserId, the other subscriber should not use the driver.
-
subscribeDriver
@Beta() final WebDriver subscribeDriver()
Subscribe a web driver, the other subscriber should not use the driver.
-
isRetiredPool
final Boolean isRetiredPool(BrowserId browserId)
Check if a pool is retired.
-
cancel
final WebDriver cancel(String url)
Cancel the fetch task specified by url remotely. NOTE: A cancel request should run immediately not waiting for any browser task return.
-
cancel
final WebDriver cancel(BrowserId browserId, String url)
Cancel the fetch task specified by url remotely. NOTE: A cancel request should run immediately not waiting for any browser task return.
-
cancelAll
final Unit cancelAll(BrowserId browserId)
Cancel all the fetch tasks, stop loading all pages.
-
closeDriverPoolGracefully
final Unit closeDriverPoolGracefully(BrowserId browserId, Duration timeToWait)
Find the driver pool with browserId, cancel all running tasks in it, and close all drivers in it.
When we close a driver pool, all threads that are trying to get a driver should wait.
-
maintain
final Unit maintain()
Maintain all the driver pools, check and report inconsistency, illness, idleness, etc., close bad pools if necessary.
If "takeDriverPoolSnapshot" is in file AppPaths.PATH_LOCAL_COMMAND, perform the action.
If the tmp dir is the default one, run the following command to take snapshot once: echo takeDriverPoolSnapshot /tmp/pulsar/pulsar-commands
-
takeSnapshot
final String takeSnapshot(BrowserId browserId, Boolean verbose)
Take a snapshot about the state of the driver pools.
-
takeSnapshot
final String takeSnapshot(Boolean verbose)
Take a snapshot about the state of the driver pools.
-
close
Unit close()
Close the web driver pool manager. All deferred tasks will be canceled, the pool of driver pools will be cleared and the browser manager will be closed.
It happens when the program exits.
The closing call stack:
PrivacyContextManager.close -> PrivacyContext.close -> WebDriverContext.close -> WebDriverPoolManager.close -> BrowserManager.close -> Browser.close -> WebDriver.close |-> LoadingWebDriverPool.close
-
-
-
-