Package org.apache.pulsar.broker.service.persistent

Class PersistentTopic

java.lang.Object
org.apache.pulsar.broker.service.AbstractTopic
org.apache.pulsar.broker.service.persistent.PersistentTopic
All Implemented Interfaces:
org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback, Topic, TopicPolicyListener<org.apache.pulsar.common.policies.data.TopicPolicies>
Direct Known Subclasses:
SystemTopic
public class PersistentTopic extends AbstractTopic implements Topic, org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback

  • Field Details

    • ledger

      protected final org.apache.bookkeeper.mledger.ManagedLedger ledger

    • msgChunkPublished

      public boolean msgChunkPublished

    • MESSAGE_RATE_BACKOFF_MS

      public static final int MESSAGE_RATE_BACKOFF_MS
      See Also:

    • messageDeduplication

      protected final MessageDeduplication messageDeduplication

    • transactionBuffer

      protected final TransactionBuffer transactionBuffer

  • Constructor Details

    • PersistentTopic

      public PersistentTopic(String topic, org.apache.bookkeeper.mledger.ManagedLedger ledger, BrokerService brokerService)

  • Method Details

    • initialize

      public CompletableFuture<Void> initialize()
      Specified by:
      initialize in interface Topic

    • publishMessage

      public void publishMessage(io.netty.buffer.ByteBuf headersAndPayload, Topic.PublishContext publishContext)
      Specified by:
      publishMessage in interface Topic

    • updateSubscribeRateLimiter

      public void updateSubscribeRateLimiter()

    • asyncReadEntry

      public void asyncReadEntry(org.apache.bookkeeper.mledger.impl.PositionImpl position, org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback callback, Object ctx)

    • getPositionAfterN

      public org.apache.bookkeeper.mledger.impl.PositionImpl getPositionAfterN(org.apache.bookkeeper.mledger.impl.PositionImpl startPosition, long n) throws org.apache.bookkeeper.mledger.ManagedLedgerException
      Throws:
      org.apache.bookkeeper.mledger.ManagedLedgerException

    • getFirstPosition

      public org.apache.bookkeeper.mledger.impl.PositionImpl getFirstPosition() throws org.apache.bookkeeper.mledger.ManagedLedgerException
      Throws:
      org.apache.bookkeeper.mledger.ManagedLedgerException

    • getNumberOfEntries

      public long getNumberOfEntries()

    • addComplete

      public void addComplete(org.apache.bookkeeper.mledger.Position pos, io.netty.buffer.ByteBuf entryData, Object ctx)
      Specified by:
      addComplete in interface org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback

    • addFailed

      public void addFailed(org.apache.bookkeeper.mledger.ManagedLedgerException exception, Object ctx)
      Specified by:
      addFailed in interface org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback

    • addProducer

      public CompletableFuture<Optional<Long>> addProducer(Producer producer, CompletableFuture<Void> producerQueuedFuture)
      Description copied from interface: Topic
      Tries to add a producer to the topic. Several validations will be performed.
      Specified by:
      addProducer in interface Topic
      Overrides:
      addProducer in class AbstractTopic
      producerQueuedFuture - a future that will be triggered if the producer is being queued up prior of getting established
      Returns:
      the "topic epoch" if there is one or empty

    • checkIfTransactionBufferRecoverCompletely

      public CompletableFuture<Void> checkIfTransactionBufferRecoverCompletely(boolean isTxnEnabled)
      Description copied from interface: Topic
      Wait TransactionBuffer Recovers completely. Take snapshot after TB Recovers completely.
      Specified by:
      checkIfTransactionBufferRecoverCompletely in interface Topic
      Returns:
      a future which has completely if isTxn = false. Or a future return by takeSnapshot.

    • incrementTopicEpoch

      protected CompletableFuture<Long> incrementTopicEpoch(Optional<Long> currentEpoch)
      Specified by:
      incrementTopicEpoch in class AbstractTopic

    • setTopicEpoch

      protected CompletableFuture<Long> setTopicEpoch(long newEpoch)
      Specified by:
      setTopicEpoch in class AbstractTopic

    • startReplProducers

      public CompletableFuture<Void> startReplProducers()

    • stopReplProducers

      public CompletableFuture<Void> stopReplProducers()

    • handleProducerRemoved

      protected void handleProducerRemoved(Producer producer)
      Overrides:
      handleProducerRemoved in class AbstractTopic

    • subscribe

      public CompletableFuture<Consumer> subscribe(SubscriptionOption option)
      Description copied from interface: Topic
      Subscribe a topic.
      Specified by:
      subscribe in interface Topic
      Returns:

    • subscribe

      public CompletableFuture<Consumer> subscribe(TransportCnx cnx, String subscriptionName, long consumerId, org.apache.pulsar.common.api.proto.CommandSubscribe.SubType subType, int priorityLevel, String consumerName, boolean isDurable, org.apache.pulsar.client.api.MessageId startMessageId, Map<String,String> metadata, boolean readCompacted, org.apache.pulsar.common.api.proto.CommandSubscribe.InitialPosition initialPosition, long startMessageRollbackDurationSec, boolean replicatedSubscriptionStateArg, org.apache.pulsar.common.api.proto.KeySharedMeta keySharedMeta)
      Specified by:
      subscribe in interface Topic

    • createSubscription

      public CompletableFuture<Subscription> createSubscription(String subscriptionName, org.apache.pulsar.common.api.proto.CommandSubscribe.InitialPosition initialPosition, boolean replicateSubscriptionState, Map<String,String> subscriptionProperties)
      Specified by:
      createSubscription in interface Topic

    • unsubscribe

      public CompletableFuture<Void> unsubscribe(String subscriptionName)
      Delete the cursor ledger for a given subscription.
      Specified by:
      unsubscribe in interface Topic
      Parameters:
      subscriptionName - Subscription for which the cursor ledger is to be deleted
      Returns:
      Completable future indicating completion of unsubscribe operation Completed exceptionally with: ManagedLedgerException if cursor ledger delete fails

    • delete

      public CompletableFuture<Void> delete()
      Delete the managed ledger associated with this topic.
      Specified by:
      delete in interface Topic
      Returns:
      Completable future indicating completion of delete operation Completed exceptionally with: IllegalStateException if topic is still active ManagedLedgerException if ledger delete operation fails

    • deleteForcefully

      public CompletableFuture<Void> deleteForcefully()
      Forcefully close all producers/consumers/replicators and deletes the topic. this function is used when local cluster is removed from global-namespace replication list. Because broker doesn't allow lookup if local cluster is not part of replication cluster list.
      Specified by:
      deleteForcefully in interface Topic
      Returns:

    • close

      public CompletableFuture<Void> close()

    • close

      public CompletableFuture<Void> close(boolean closeWithoutWaitingClientDisconnect)
      Close this topic - close all producers and subscriptions associated with this topic.
      Specified by:
      close in interface Topic
      Parameters:
      closeWithoutWaitingClientDisconnect - don't wait for client disconnect and forcefully close managed-ledger
      Returns:
      Completable future indicating completion of close operation

    • checkDeduplicationStatus

      public CompletableFuture<Void> checkDeduplicationStatus()

    • checkReplication

      public CompletableFuture<Void> checkReplication()
      Specified by:
      checkReplication in interface Topic

    • checkMessageExpiry

      public void checkMessageExpiry()
      Specified by:
      checkMessageExpiry in interface Topic

    • checkMessageDeduplicationInfo

      public void checkMessageDeduplicationInfo()
      Specified by:
      checkMessageDeduplicationInfo in interface Topic

    • isCompactionEnabled

      public boolean isCompactionEnabled()

    • checkCompaction

      public void checkCompaction()

    • preCreateSubscriptionForCompactionIfNeeded

      public CompletableFuture<Void> preCreateSubscriptionForCompactionIfNeeded()

    • addReplicationCluster

      protected CompletableFuture<Void> addReplicationCluster(String remoteCluster, org.apache.bookkeeper.mledger.ManagedCursor cursor, String localCluster)

    • isDeduplicationEnabled

      public boolean isDeduplicationEnabled()

    • getNumberOfConsumers

      public int getNumberOfConsumers()
      Specified by:
      getNumberOfConsumers in class AbstractTopic

    • getNumberOfSameAddressConsumers

      public int getNumberOfSameAddressConsumers(String clientAddress)
      Specified by:
      getNumberOfSameAddressConsumers in class AbstractTopic

    • getSubscriptions

      public org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,PersistentSubscription> getSubscriptions()
      Specified by:
      getSubscriptions in interface Topic

    • getSubscription

      public PersistentSubscription getSubscription(String subscriptionName)
      Specified by:
      getSubscription in interface Topic

    • getReplicators

      public org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,Replicator> getReplicators()
      Specified by:
      getReplicators in interface Topic

    • getPersistentReplicator

      public Replicator getPersistentReplicator(String remoteCluster)

    • getManagedLedger

      public org.apache.bookkeeper.mledger.ManagedLedger getManagedLedger()

    • updateRates

      public void updateRates(NamespaceStats nsStats, org.apache.pulsar.policies.data.loadbalancer.NamespaceBundleStats bundleStats, StatsOutputStream topicStatsStream, ClusterReplicationMetrics replStats, String namespace, boolean hydratePublishers)
      Specified by:
      updateRates in interface Topic

    • getLastUpdatedAvgPublishRateInMsg

      public double getLastUpdatedAvgPublishRateInMsg()

    • getLastUpdatedAvgPublishRateInByte

      public double getLastUpdatedAvgPublishRateInByte()

    • getStats

      public org.apache.pulsar.common.policies.data.stats.TopicStatsImpl getStats(boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean getEarliestTimeInBacklog)
      Specified by:
      getStats in interface Topic

    • asyncGetStats

      public CompletableFuture<org.apache.pulsar.common.policies.data.stats.TopicStatsImpl> asyncGetStats(boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean getEarliestTimeInBacklog)
      Specified by:
      asyncGetStats in interface Topic

    • getInternalStats

      public CompletableFuture<org.apache.pulsar.common.policies.data.PersistentTopicInternalStats> getInternalStats(boolean includeLedgerMetadata)
      Specified by:
      getInternalStats in interface Topic

    • getCompactedTopicContext

      public Optional<CompactedTopicContext> getCompactedTopicContext()

    • getBacklogSize

      public long getBacklogSize()

    • isActive

      public boolean isActive(org.apache.pulsar.common.policies.data.InactiveTopicDeleteMode deleteMode)

    • checkGC

      public void checkGC()
      Specified by:
      checkGC in interface Topic

    • checkInactiveSubscriptions

      public void checkInactiveSubscriptions()
      Specified by:
      checkInactiveSubscriptions in interface Topic

    • checkBackloggedCursors

      public void checkBackloggedCursors()
      Description copied from interface: Topic
      Activate cursors those caught up backlog-threshold entries and deactivate slow cursors which are creating backlog.
      Specified by:
      checkBackloggedCursors in interface Topic

    • checkInactiveLedgers

      public void checkInactiveLedgers()

    • checkCursorsToCacheEntries

      public void checkCursorsToCacheEntries()
      Specified by:
      checkCursorsToCacheEntries in interface Topic

    • checkDeduplicationSnapshot

      public void checkDeduplicationSnapshot()
      Specified by:
      checkDeduplicationSnapshot in interface Topic

    • onLocalPoliciesUpdate

      public CompletableFuture<Void> onLocalPoliciesUpdate()

    • updateDispatchRateLimiter

      public void updateDispatchRateLimiter()
      Overrides:
      updateDispatchRateLimiter in class AbstractTopic

    • onPoliciesUpdate

      public CompletableFuture<Void> onPoliciesUpdate(org.apache.pulsar.common.policies.data.Policies data)
      Specified by:
      onPoliciesUpdate in interface Topic

    • getBacklogQuota

      public org.apache.pulsar.common.policies.data.BacklogQuota getBacklogQuota(org.apache.pulsar.common.policies.data.BacklogQuota.BacklogQuotaType backlogQuotaType)
      Specified by:
      getBacklogQuota in interface Topic
      Returns:
      Backlog quota for topic

    • checkBacklogQuotaExceeded

      public CompletableFuture<Void> checkBacklogQuotaExceeded(String producerName, org.apache.pulsar.common.policies.data.BacklogQuota.BacklogQuotaType backlogQuotaType)
      Specified by:
      checkBacklogQuotaExceeded in interface Topic
      Returns:
      quota exceeded status for blocking producer creation

    • isSizeBacklogExceeded

      public boolean isSizeBacklogExceeded()
      Returns:
      determine if backlog quota enforcement needs to be done for topic based on size limit

    • checkTimeBacklogExceeded

      public CompletableFuture<Boolean> checkTimeBacklogExceeded()
      Returns:
      determine if backlog quota enforcement needs to be done for topic based on time limit

    • isReplicated

      public boolean isReplicated()
      Specified by:
      isReplicated in interface Topic

    • terminate

      public CompletableFuture<org.apache.pulsar.client.api.MessageId> terminate()

    • isOldestMessageExpired

      public boolean isOldestMessageExpired(org.apache.bookkeeper.mledger.ManagedCursor cursor, int messageTTLInSeconds)

    • clearBacklog

      public CompletableFuture<Void> clearBacklog()
      Clears backlog for all cursors in the topic.
      Returns:

    • clearBacklog

      public CompletableFuture<Void> clearBacklog(String cursorName)
      Clears backlog for a given cursor in the topic.

      Note: For a replication cursor, just provide the remote cluster name

      Parameters:
      cursorName -
      Returns:

    • getDispatchRateLimiter

      public Optional<DispatchRateLimiter> getDispatchRateLimiter()
      Specified by:
      getDispatchRateLimiter in interface Topic

    • getBrokerDispatchRateLimiter

      public Optional<DispatchRateLimiter> getBrokerDispatchRateLimiter()
      Specified by:
      getBrokerDispatchRateLimiter in interface Topic

    • getSubscribeRateLimiter

      public Optional<SubscribeRateLimiter> getSubscribeRateLimiter()
      Specified by:
      getSubscribeRateLimiter in interface Topic

    • getLastPublishedSequenceId

      public long getLastPublishedSequenceId(String producerName)

    • getLastPosition

      public org.apache.bookkeeper.mledger.Position getLastPosition()
      Specified by:
      getLastPosition in interface Topic

    • getLastMessageId

      public CompletableFuture<org.apache.pulsar.client.api.MessageId> getLastMessageId()
      Specified by:
      getLastMessageId in interface Topic

    • triggerCompaction

      public void triggerCompaction() throws org.apache.pulsar.broker.PulsarServerException, BrokerServiceException.AlreadyRunningException
      Throws:
      org.apache.pulsar.broker.PulsarServerException
      BrokerServiceException.AlreadyRunningException

    • compactionStatus

      public org.apache.pulsar.client.admin.LongRunningProcessStatus compactionStatus()

    • triggerOffload

      public void triggerOffload(org.apache.pulsar.client.impl.MessageIdImpl messageId) throws BrokerServiceException.AlreadyRunningException
      Throws:
      BrokerServiceException.AlreadyRunningException

    • offloadStatus

      public org.apache.pulsar.client.admin.OffloadProcessStatus offloadStatus()

    • addSchemaIfIdleOrCheckCompatible

      public CompletableFuture<Void> addSchemaIfIdleOrCheckCompatible(org.apache.pulsar.common.protocol.schema.SchemaData schema)
      Description copied from interface: Topic
      If the topic is idle (no producers, no entries, no subscribers and no existing schema), add the passed schema to the topic. Otherwise, check that the passed schema is compatible with what the topic already has.
      Specified by:
      addSchemaIfIdleOrCheckCompatible in interface Topic

    • checkReplicatedSubscriptionControllerState

      public void checkReplicatedSubscriptionControllerState()

    • getReplicatedSubscriptionController

      public Optional<ReplicatedSubscriptionsController> getReplicatedSubscriptionController()

    • getCompactedTopic

      public CompactedTopic getCompactedTopic()

    • isSystemTopic

      public boolean isSystemTopic()
      Specified by:
      isSystemTopic in interface Topic

    • isPersistent

      public boolean isPersistent()
      Specified by:
      isPersistent in interface Topic

    • publishTxnMessage

      public void publishTxnMessage(org.apache.pulsar.client.api.transaction.TxnID txnID, io.netty.buffer.ByteBuf headersAndPayload, Topic.PublishContext publishContext)
      Description copied from interface: Topic
      Publish Transaction message to this Topic's TransactionBuffer.
      Specified by:
      publishTxnMessage in interface Topic
      Parameters:
      txnID - Transaction Id
      headersAndPayload - Message data
      publishContext - Publish context

    • endTxn

      public CompletableFuture<Void> endTxn(org.apache.pulsar.client.api.transaction.TxnID txnID, int txnAction, long lowWaterMark)
      Description copied from interface: Topic
      End the transaction in this topic.
      Specified by:
      endTxn in interface Topic
      Parameters:
      txnID - Transaction id
      txnAction - Transaction action.
      lowWaterMark - low water mark of this tc
      Returns:

    • truncate

      public CompletableFuture<Void> truncate()
      Description copied from interface: Topic
      Truncate a topic. The truncate operation will move all cursors to the end of the topic and delete all inactive ledgers.
      Specified by:
      truncate in interface Topic
      Returns:

    • getDelayedDeliveryTickTimeMillis

      public long getDelayedDeliveryTickTimeMillis()

    • isDelayedDeliveryEnabled

      public boolean isDelayedDeliveryEnabled()

    • getMaxUnackedMessagesOnSubscription

      public int getMaxUnackedMessagesOnSubscription()

    • onUpdate

      public void onUpdate(org.apache.pulsar.common.policies.data.TopicPolicies policies)
      Specified by:
      onUpdate in interface TopicPolicyListener<org.apache.pulsar.common.policies.data.TopicPolicies>

    • initTopicPolicy

      protected CompletableFuture<Void> initTopicPolicy()

    • getMessageDeduplication

      public MessageDeduplication getMessageDeduplication()

    • checkSubscriptionTypesEnable

      public boolean checkSubscriptionTypesEnable(org.apache.pulsar.common.api.proto.CommandSubscribe.SubType subType)

    • getTransactionBufferStats

      public org.apache.pulsar.common.policies.data.TransactionBufferStats getTransactionBufferStats(boolean lowWaterMarks)

    • getTransactionPendingAckStats

      public org.apache.pulsar.common.policies.data.TransactionPendingAckStats getTransactionPendingAckStats(String subName, boolean lowWaterMarks)

    • getMaxReadPosition

      public org.apache.bookkeeper.mledger.impl.PositionImpl getMaxReadPosition()

    • isTxnAborted

      public boolean isTxnAborted(org.apache.pulsar.client.api.transaction.TxnID txnID)

    • getTransactionInBufferStats

      public org.apache.pulsar.common.policies.data.TransactionInBufferStats getTransactionInBufferStats(org.apache.pulsar.client.api.transaction.TxnID txnID)

    • isTerminated

      protected boolean isTerminated()
      Specified by:
      isTerminated in class AbstractTopic

    • getTransactionInPendingAckStats

      public org.apache.pulsar.common.policies.data.TransactionInPendingAckStats getTransactionInPendingAckStats(org.apache.pulsar.client.api.transaction.TxnID txnID, String subName)

    • getPendingAckManagedLedger

      public CompletableFuture<org.apache.bookkeeper.mledger.ManagedLedger> getPendingAckManagedLedger(String subName)

    • getLastDataMessagePublishedTimestamp

      public long getLastDataMessagePublishedTimestamp()