Package org.apache.pulsar.broker.service

Class BrokerService

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    public class BrokerService
extends java.lang.Object
implements java.io.Closeable

    • Field Detail

      • lookupRequestSemaphore

        protected final java.util.concurrent.atomic.AtomicReference<java.util.concurrent.Semaphore> lookupRequestSemaphore

      • topicLoadRequestSemaphore

        protected final java.util.concurrent.atomic.AtomicReference<java.util.concurrent.Semaphore> topicLoadRequestSemaphore

      • brokerPublishRateLimiter

        protected volatile PublishRateLimiter brokerPublishRateLimiter

      • PRODUCER_NAME_GENERATOR_PATH

        public static final java.lang.String PRODUCER_NAME_GENERATOR_PATH
        See Also:
        Constant Field Values

      • MANAGED_LEDGER_PATH_ZNODE

        public static final java.lang.String MANAGED_LEDGER_PATH_ZNODE
        See Also:
        Constant Field Values

      • maxUnackedMsgsPerDispatcher

        public final int maxUnackedMsgsPerDispatcher

    • Constructor Detail

      • BrokerService

        public BrokerService​(PulsarService pulsar,
                     io.netty.channel.EventLoopGroup eventLoopGroup)
              throws java.lang.Exception
        Throws:
        java.lang.Exception

    • Method Detail

      • startProtocolHandlers

        public void startProtocolHandlers​(java.util.Map<java.lang.String,​java.util.Map<java.net.InetSocketAddress,​io.netty.channel.ChannelInitializer<io.netty.channel.socket.SocketChannel>>> protocolHandlers)

      • start

        public void start()
           throws java.lang.Exception
        Throws:
        java.lang.Exception

      • startStatsUpdater

        protected void startStatsUpdater​(int statsUpdateInitialDelayInSecs,
                                 int statsUpdateFrequencyInSecs)

      • startDeduplicationSnapshotMonitor

        protected void startDeduplicationSnapshotMonitor()

      • startInactivityMonitor

        protected void startInactivityMonitor()

      • startMessageExpiryMonitor

        protected void startMessageExpiryMonitor()

      • startCheckReplicationPolicies

        protected void startCheckReplicationPolicies()

      • startCompactionMonitor

        protected void startCompactionMonitor()

      • startConsumedLedgersMonitor

        protected void startConsumedLedgersMonitor()

      • startBacklogQuotaChecker

        protected void startBacklogQuotaChecker()

      • setupTopicPublishRateLimiterMonitor

        public void setupTopicPublishRateLimiterMonitor()
        Schedules and monitors publish-throttling for all owned topics that has publish-throttling configured. It also disables and shutdowns publish-rate-limiter monitor task if broker disables it.

      • setupBrokerPublishRateLimiterMonitor

        public void setupBrokerPublishRateLimiterMonitor()
        Schedules and monitors publish-throttling for broker that has publish-throttling configured. It also disables and shutdowns publish-rate-limiter monitor for broker task if broker disables it.

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Throws:
        java.io.IOException

      • closeAndRemoveReplicationClient

        public java.util.concurrent.CompletableFuture<java.lang.Void> closeAndRemoveReplicationClient​(java.lang.String clusterName)

      • closeAsync

        public java.util.concurrent.CompletableFuture<java.lang.Void> closeAsync()

      • unloadNamespaceBundlesGracefully

        public void unloadNamespaceBundlesGracefully()
        It unloads all owned namespacebundles gracefully.
        • First it makes current broker unavailable and isolates from the clusters so, it will not serve any new requests.
        • Second it starts unloading namespace bundle one by one without closing the connection in order to avoid disruption for other namespacebundles which are sharing the same connection from the same client.

      • unloadNamespaceBundlesGracefully

        public void unloadNamespaceBundlesGracefully​(int maxConcurrentUnload,
                                             boolean closeWithoutWaitingClientDisconnect)

      • getTopicIfExists

        public java.util.concurrent.CompletableFuture<java.util.Optional<Topic>> getTopicIfExists​(java.lang.String topic)

      • getOrCreateTopic

        public java.util.concurrent.CompletableFuture<Topic> getOrCreateTopic​(java.lang.String topic)

      • getTopic

        public java.util.concurrent.CompletableFuture<java.util.Optional<Topic>> getTopic​(java.lang.String topic,
                                                                                  boolean createIfMissing)

      • getTopic

        public java.util.concurrent.CompletableFuture<java.util.Optional<Topic>> getTopic​(java.lang.String topic,
                                                                                  boolean createIfMissing,
                                                                                  java.util.Map<java.lang.String,​java.lang.String> properties)

      • deleteSchemaStorage

        public java.util.concurrent.CompletableFuture<org.apache.pulsar.common.protocol.schema.SchemaVersion> deleteSchemaStorage​(java.lang.String topic)

      • deleteTopic

        public java.util.concurrent.CompletableFuture<java.lang.Void> deleteTopic​(java.lang.String topic,
                                                                          boolean forceDelete)

      • deleteTopic

        public java.util.concurrent.CompletableFuture<java.lang.Void> deleteTopic​(java.lang.String topic,
                                                                          boolean forceDelete,
                                                                          boolean deleteSchema)

      • deleteTopicAuthenticationWithRetry

        public void deleteTopicAuthenticationWithRetry​(java.lang.String topic,
                                               java.util.concurrent.CompletableFuture<java.lang.Void> future,
                                               int count)

      • getReplicationClient

        public org.apache.pulsar.client.api.PulsarClient getReplicationClient​(java.lang.String cluster,
                                                                      java.util.Optional<org.apache.pulsar.common.policies.data.ClusterData> clusterDataOp)

      • getClusterPulsarAdmin

        public org.apache.pulsar.client.admin.PulsarAdmin getClusterPulsarAdmin​(java.lang.String cluster,
                                                                        java.util.Optional<org.apache.pulsar.common.policies.data.ClusterData> clusterDataOp)

      • loadOrCreatePersistentTopic

        protected java.util.concurrent.CompletableFuture<java.util.Optional<Topic>> loadOrCreatePersistentTopic​(java.lang.String topic,
                                                                                                        boolean createIfMissing,
                                                                                                        java.util.Map<java.lang.String,​java.lang.String> properties)
                                                                                                 throws java.lang.RuntimeException
        It creates a topic async and returns CompletableFuture. It also throttles down configured max-concurrent topic loading and puts them into queue once in-process topics are created.
        Parameters:
        topic - persistent-topic name
        Returns:
        CompletableFuture
        Throws:
        java.lang.RuntimeException

      • getManagedLedgerConfig

        public java.util.concurrent.CompletableFuture<org.apache.bookkeeper.mledger.ManagedLedgerConfig> getManagedLedgerConfig​(org.apache.pulsar.common.naming.TopicName topicName)

      • refreshTopicToStatsMaps

        public void refreshTopicToStatsMaps​(NamespaceBundle oldBundle)

      • getOfflineTopicStat

        public org.apache.pulsar.common.policies.data.PersistentOfflineTopicStats getOfflineTopicStat​(org.apache.pulsar.common.naming.TopicName topicName)

      • cacheOfflineTopicStats

        public void cacheOfflineTopicStats​(org.apache.pulsar.common.naming.TopicName topicName,
                                   org.apache.pulsar.common.policies.data.PersistentOfflineTopicStats offlineTopicStats)

      • invalidateOfflineTopicStatCache

        public void invalidateOfflineTopicStatCache​(org.apache.pulsar.common.naming.TopicName topicName)

      • getTopicReference

        public java.util.Optional<Topic> getTopicReference​(java.lang.String topic)
        Get a reference to a topic that is currently loaded in the broker. This method will not make the broker attempt to load the topic if it's not already.

      • updateRates

        public void updateRates()

      • getDimensionMetrics

        public void getDimensionMetrics​(java.util.function.Consumer<io.netty.buffer.ByteBuf> consumer)

      • getTopicMetrics

        public java.util.List<org.apache.pulsar.common.stats.Metrics> getTopicMetrics()

      • getBundleStats

        public java.util.Map<java.lang.String,​org.apache.pulsar.policies.data.loadbalancer.NamespaceBundleStats> getBundleStats()

      • getLookupRequestSemaphore

        public java.util.concurrent.Semaphore getLookupRequestSemaphore()

      • checkGC

        public void checkGC()

      • checkMessageExpiry

        public void checkMessageExpiry()

      • checkReplicationPolicies

        public void checkReplicationPolicies()

      • checkCompaction

        public void checkCompaction()

      • checkMessageDeduplicationInfo

        public void checkMessageDeduplicationInfo()

      • checkInactiveSubscriptions

        public void checkInactiveSubscriptions()

      • checkTopicPublishThrottlingRate

        public void checkTopicPublishThrottlingRate()

      • checkBrokerPublishThrottlingRate

        public void checkBrokerPublishThrottlingRate()

      • forEachTopic

        public void forEachTopic​(java.util.function.Consumer<Topic> consumer)
        Iterates over all loaded topics in the broker.

      • monitorBacklogQuota

        public void monitorBacklogQuota()

      • isTopicNsOwnedByBroker

        public boolean isTopicNsOwnedByBroker​(org.apache.pulsar.common.naming.TopicName topicName)

      • checkTopicNsOwnership

        public java.util.concurrent.CompletableFuture<java.lang.Void> checkTopicNsOwnership​(java.lang.String topic)

      • unloadServiceUnit

        public java.util.concurrent.CompletableFuture<java.lang.Integer> unloadServiceUnit​(NamespaceBundle serviceUnit,
                                                                                   boolean closeWithoutWaitingClientDisconnect,
                                                                                   long timeout,
                                                                                   java.util.concurrent.TimeUnit unit)

      • cleanUnloadedTopicFromCache

        public void cleanUnloadedTopicFromCache​(NamespaceBundle serviceUnit)

      • getAuthorizationService

        public org.apache.pulsar.broker.authorization.AuthorizationService getAuthorizationService()

      • removeTopicFromCache

        public java.util.concurrent.CompletableFuture<java.lang.Void> removeTopicFromCache​(java.lang.String topicName)

      • removeTopicFromCache

        public java.util.concurrent.CompletableFuture<java.lang.Void> removeTopicFromCache​(Topic topic)

      • getNumberOfNamespaceBundles

        public int getNumberOfNamespaceBundles()

      • getTopics

        public org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<java.lang.String,​java.util.concurrent.CompletableFuture<java.util.Optional<Topic>>> getTopics()

      • executor

        public io.netty.channel.EventLoopGroup executor()

      • getReplicationClients

        public org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<java.lang.String,​org.apache.pulsar.client.api.PulsarClient> getReplicationClients()

      • isAuthenticationEnabled

        public boolean isAuthenticationEnabled()

      • isAuthorizationEnabled

        public boolean isAuthorizationEnabled()

      • getKeepAliveIntervalSeconds

        public int getKeepAliveIntervalSeconds()

      • generateUniqueProducerName

        public java.lang.String generateUniqueProducerName()

      • getTopicStats

        public java.util.Map<java.lang.String,​org.apache.pulsar.common.policies.data.stats.TopicStatsImpl> getTopicStats()

      • getAuthenticationService

        public org.apache.pulsar.broker.authentication.AuthenticationService getAuthenticationService()

      • getAllTopicsFromNamespaceBundle

        public java.util.List<Topic> getAllTopicsFromNamespaceBundle​(java.lang.String namespace,
                                                             java.lang.String bundle)

      • registerConfigurationListener

        public <T> void registerConfigurationListener​(java.lang.String configKey,
                                              java.util.function.Consumer<T> listener)
        Allows a listener to listen on update of ServiceConfiguration change, so listener can take appropriate action if any specific config-field value has been changed. On notification, listener should first check if config value has been changed and after taking appropriate action, listener should update config value with new value if it has been changed (so, next time listener can compare values on configMap change).
        Type Parameters:
        T -
        Parameters:
        configKey - : configuration field name
        listener - : listener which takes appropriate action on config-value change

      • getDynamicConfiguration

        public static java.util.List<java.lang.String> getDynamicConfiguration()

      • getRuntimeConfiguration

        public java.util.Map<java.lang.String,​java.lang.String> getRuntimeConfiguration()

      • isDynamicConfiguration

        public static boolean isDynamicConfiguration​(java.lang.String key)

      • validateDynamicConfiguration

        public static boolean validateDynamicConfiguration​(java.lang.String key,
                                                   java.lang.String value)

      • fetchPartitionedTopicMetadataCheckAllowAutoCreationAsync

        public java.util.concurrent.CompletableFuture<org.apache.pulsar.common.partition.PartitionedTopicMetadata> fetchPartitionedTopicMetadataCheckAllowAutoCreationAsync​(org.apache.pulsar.common.naming.TopicName topicName)

      • fetchPartitionedTopicMetadataAsync

        public java.util.concurrent.CompletableFuture<org.apache.pulsar.common.partition.PartitionedTopicMetadata> fetchPartitionedTopicMetadataAsync​(org.apache.pulsar.common.naming.TopicName topicName)

      • getTopicOrderedExecutor

        public org.apache.bookkeeper.common.util.OrderedExecutor getTopicOrderedExecutor()

      • getMultiLayerTopicMap

        public org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<java.lang.String,​org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<java.lang.String,​org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<java.lang.String,​Topic>>> getMultiLayerTopicMap()

      • addUnAckedMessages

        public void addUnAckedMessages​(PersistentDispatcherMultipleConsumers dispatcher,
                               int numberOfMessages)
        If per-broker unacked message reached to limit then it blocks dispatcher if its unacked message limit has been reached to maxUnackedMsgsPerDispatcher.
        Parameters:
        dispatcher -
        numberOfMessages -

      • checkUnAckMessageDispatching

        public void checkUnAckMessageDispatching()
        Adds given dispatcher's unackMessage count to broker-unack message count and if it reaches to the maxUnackedMessages then it blocks all the dispatchers which has unack-messages higher than maxUnackedMsgsPerDispatcher. It unblocks all dispatchers once broker-unack message counts decreased to (maxUnackedMessages/2)

      • isBrokerDispatchingBlocked

        public boolean isBrokerDispatchingBlocked()

      • extractTopic

        public static java.util.Optional<Topic> extractTopic​(java.util.concurrent.CompletableFuture<java.util.Optional<Topic>> topicFuture)
        Safely extract optional topic instance from a future, in a way to avoid unchecked exceptions and race conditions.

      • getListenPort

        public java.util.Optional<java.lang.Integer> getListenPort()

      • getListenPortTls

        public java.util.Optional<java.lang.Integer> getListenPortTls()

      • isAllowAutoTopicCreation

        public boolean isAllowAutoTopicCreation​(java.lang.String topic)

      • isAllowAutoTopicCreation

        public boolean isAllowAutoTopicCreation​(org.apache.pulsar.common.naming.TopicName topicName)

      • isAllowAutoTopicCreation

        public boolean isAllowAutoTopicCreation​(org.apache.pulsar.common.naming.TopicName topicName,
                                        java.util.Optional<org.apache.pulsar.common.policies.data.Policies> policies)

      • isDefaultTopicTypePartitioned

        public boolean isDefaultTopicTypePartitioned​(org.apache.pulsar.common.naming.TopicName topicName,
                                             java.util.Optional<org.apache.pulsar.common.policies.data.Policies> policies)

      • getDefaultNumPartitions

        public int getDefaultNumPartitions​(org.apache.pulsar.common.naming.TopicName topicName,
                                   java.util.Optional<org.apache.pulsar.common.policies.data.Policies> policies)

      • isAllowAutoSubscriptionCreation

        public boolean isAllowAutoSubscriptionCreation​(java.lang.String topic)

      • isAllowAutoSubscriptionCreation

        public boolean isAllowAutoSubscriptionCreation​(org.apache.pulsar.common.naming.TopicName topicName)

      • isSystemTopic

        public boolean isSystemTopic​(java.lang.String topic)

      • isSystemTopic

        public boolean isSystemTopic​(org.apache.pulsar.common.naming.TopicName topicName)

      • getTopicPolicies

        public java.util.Optional<org.apache.pulsar.common.policies.data.TopicPolicies> getTopicPolicies​(org.apache.pulsar.common.naming.TopicName topicName)
        Get TopicPolicies for the parameterized topic.
        Parameters:
        topicName -
        Returns:
        TopicPolicies, if they exist. Otherwise, the value will not be present.

      • deleteTopicPolicies

        public java.util.concurrent.CompletableFuture<java.lang.Void> deleteTopicPolicies​(org.apache.pulsar.common.naming.TopicName topicName)

      • getBrokerEntryMetadataInterceptors

        public java.util.Set<org.apache.pulsar.common.intercept.BrokerEntryMetadataInterceptor> getBrokerEntryMetadataInterceptors()

      • isBrokerEntryMetadataEnabled

        public boolean isBrokerEntryMetadataEnabled()

      • isBrokerPayloadProcessorEnabled

        public boolean isBrokerPayloadProcessorEnabled()

      • pausedConnections

        public void pausedConnections​(int numberOfConnections)

      • resumedConnections

        public void resumedConnections​(int numberOfConnections)

      • getPausedConnections

        public long getPausedConnections()