Package org.apache.pulsar.broker.service.persistent

Class PersistentTopic

  • All Implemented Interfaces:
    org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback, Topic, TopicPolicyListener<org.apache.pulsar.common.policies.data.TopicPolicies>
    Direct Known Subclasses:
    SystemTopic
    public class PersistentTopic
extends AbstractTopic
implements Topic, org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback, TopicPolicyListener<org.apache.pulsar.common.policies.data.TopicPolicies>

    • Field Detail

      • ledger

        protected final org.apache.bookkeeper.mledger.ManagedLedger ledger

      • msgChunkPublished

        public boolean msgChunkPublished

      • delayedDeliveryTickTimeMillis

        public volatile long delayedDeliveryTickTimeMillis

      • delayedDeliveryEnabled

        public volatile boolean delayedDeliveryEnabled

      • MESSAGE_RATE_BACKOFF_MS

        public static final int MESSAGE_RATE_BACKOFF_MS
        See Also:
        Constant Field Values

    • Constructor Detail

      • PersistentTopic

        public PersistentTopic​(java.lang.String topic,
                       org.apache.bookkeeper.mledger.ManagedLedger ledger,
                       BrokerService brokerService)

    • Method Detail

      • initialize

        public java.util.concurrent.CompletableFuture<java.lang.Void> initialize()
        Specified by:
        initialize in interface Topic

      • asyncReadEntry

        public void asyncReadEntry​(org.apache.bookkeeper.mledger.impl.PositionImpl position,
                           org.apache.bookkeeper.mledger.AsyncCallbacks.ReadEntryCallback callback,
                           java.lang.Object ctx)

      • getPositionAfterN

        public org.apache.bookkeeper.mledger.impl.PositionImpl getPositionAfterN​(org.apache.bookkeeper.mledger.impl.PositionImpl startPosition,
                                                                         long n)
                                                                  throws org.apache.bookkeeper.mledger.ManagedLedgerException
        Throws:
        org.apache.bookkeeper.mledger.ManagedLedgerException

      • getFirstPosition

        public org.apache.bookkeeper.mledger.impl.PositionImpl getFirstPosition()
                                                                 throws org.apache.bookkeeper.mledger.ManagedLedgerException
        Throws:
        org.apache.bookkeeper.mledger.ManagedLedgerException

      • getNumberOfEntries

        public long getNumberOfEntries()

      • addComplete

        public void addComplete​(org.apache.bookkeeper.mledger.Position pos,
                        io.netty.buffer.ByteBuf entryData,
                        java.lang.Object ctx)
        Specified by:
        addComplete in interface org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback

      • addFailed

        public void addFailed​(org.apache.bookkeeper.mledger.ManagedLedgerException exception,
                      java.lang.Object ctx)
        Specified by:
        addFailed in interface org.apache.bookkeeper.mledger.AsyncCallbacks.AddEntryCallback

      • addProducer

        public java.util.concurrent.CompletableFuture<java.util.Optional<java.lang.Long>> addProducer​(Producer producer,
                                                                                              java.util.concurrent.CompletableFuture<java.lang.Void> producerQueuedFuture)
        Description copied from interface: Topic
        Tries to add a producer to the topic. Several validations will be performed.
        Specified by:
        addProducer in interface Topic
        Overrides:
        addProducer in class AbstractTopic
        producerQueuedFuture - a future that will be triggered if the producer is being queued up prior of getting established
        Returns:
        the "topic epoch" if there is one or empty

      • checkIfTransactionBufferRecoverCompletely

        public java.util.concurrent.CompletableFuture<java.lang.Void> checkIfTransactionBufferRecoverCompletely​(boolean isTxnEnabled)
        Description copied from interface: Topic
        Wait TransactionBuffer Recovers completely. Take snapshot after TB Recovers completely.
        Specified by:
        checkIfTransactionBufferRecoverCompletely in interface Topic
        Returns:
        a future which has completely if isTxn = false. Or a future return by takeSnapshot.

      • incrementTopicEpoch

        protected java.util.concurrent.CompletableFuture<java.lang.Long> incrementTopicEpoch​(java.util.Optional<java.lang.Long> currentEpoch)
        Specified by:
        incrementTopicEpoch in class AbstractTopic

      • setTopicEpoch

        protected java.util.concurrent.CompletableFuture<java.lang.Long> setTopicEpoch​(long newEpoch)
        Specified by:
        setTopicEpoch in class AbstractTopic

      • startReplProducers

        public java.util.concurrent.CompletableFuture<java.lang.Void> startReplProducers()

      • stopReplProducers

        public java.util.concurrent.CompletableFuture<java.lang.Void> stopReplProducers()

      • subscribe

        public java.util.concurrent.CompletableFuture<Consumer> subscribe​(TransportCnx cnx,
                                                                  java.lang.String subscriptionName,
                                                                  long consumerId,
                                                                  org.apache.pulsar.common.api.proto.CommandSubscribe.SubType subType,
                                                                  int priorityLevel,
                                                                  java.lang.String consumerName,
                                                                  boolean isDurable,
                                                                  org.apache.pulsar.client.api.MessageId startMessageId,
                                                                  java.util.Map<java.lang.String,​java.lang.String> metadata,
                                                                  boolean readCompacted,
                                                                  org.apache.pulsar.common.api.proto.CommandSubscribe.InitialPosition initialPosition,
                                                                  long startMessageRollbackDurationSec,
                                                                  boolean replicatedSubscriptionStateArg,
                                                                  org.apache.pulsar.common.api.proto.KeySharedMeta keySharedMeta)
        Specified by:
        subscribe in interface Topic

      • updateUnackedMessagesAppliedOnSubscription

        public void updateUnackedMessagesAppliedOnSubscription​(org.apache.pulsar.common.policies.data.Policies policies)

      • createSubscription

        public java.util.concurrent.CompletableFuture<Subscription> createSubscription​(java.lang.String subscriptionName,
                                                                               org.apache.pulsar.common.api.proto.CommandSubscribe.InitialPosition initialPosition,
                                                                               boolean replicateSubscriptionState)
        Specified by:
        createSubscription in interface Topic

      • unsubscribe

        public java.util.concurrent.CompletableFuture<java.lang.Void> unsubscribe​(java.lang.String subscriptionName)
        Delete the cursor ledger for a given subscription.
        Specified by:
        unsubscribe in interface Topic
        Parameters:
        subscriptionName - Subscription for which the cursor ledger is to be deleted
        Returns:
        Completable future indicating completion of unsubscribe operation Completed exceptionally with: ManagedLedgerException if cursor ledger delete fails

      • delete

        public java.util.concurrent.CompletableFuture<java.lang.Void> delete()
        Delete the managed ledger associated with this topic.
        Specified by:
        delete in interface Topic
        Returns:
        Completable future indicating completion of delete operation Completed exceptionally with: IllegalStateException if topic is still active ManagedLedgerException if ledger delete operation fails

      • deleteForcefully

        public java.util.concurrent.CompletableFuture<java.lang.Void> deleteForcefully()
        Forcefully close all producers/consumers/replicators and deletes the topic. this function is used when local cluster is removed from global-namespace replication list. Because broker doesn't allow lookup if local cluster is not part of replication cluster list.
        Specified by:
        deleteForcefully in interface Topic
        Returns:

      • close

        public java.util.concurrent.CompletableFuture<java.lang.Void> close()

      • close

        public java.util.concurrent.CompletableFuture<java.lang.Void> close​(boolean closeWithoutWaitingClientDisconnect)
        Close this topic - close all producers and subscriptions associated with this topic.
        Specified by:
        close in interface Topic
        Parameters:
        closeWithoutWaitingClientDisconnect - don't wait for client disconnect and forcefully close managed-ledger
        Returns:
        Completable future indicating completion of close operation

      • checkDeduplicationStatus

        public java.util.concurrent.CompletableFuture<java.lang.Void> checkDeduplicationStatus()

      • checkReplication

        public java.util.concurrent.CompletableFuture<java.lang.Void> checkReplication()
        Specified by:
        checkReplication in interface Topic

      • isCompactionEnabled

        public java.util.concurrent.CompletableFuture<java.lang.Boolean> isCompactionEnabled()

      • checkCompaction

        public void checkCompaction()

      • preCreateSubscriptionForCompactionIfNeeded

        public java.util.concurrent.CompletableFuture<java.lang.Void> preCreateSubscriptionForCompactionIfNeeded()

      • addReplicationCluster

        protected java.util.concurrent.CompletableFuture<java.lang.Void> addReplicationCluster​(java.lang.String remoteCluster,
                                                                                       org.apache.bookkeeper.mledger.ManagedCursor cursor,
                                                                                       java.lang.String localCluster)

      • isDeduplicationEnabled

        public boolean isDeduplicationEnabled()

      • getReplicators

        public org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<java.lang.String,​Replicator> getReplicators()
        Specified by:
        getReplicators in interface Topic

      • getPersistentReplicator

        public Replicator getPersistentReplicator​(java.lang.String remoteCluster)

      • getManagedLedger

        public org.apache.bookkeeper.mledger.ManagedLedger getManagedLedger()

      • getLastUpdatedAvgPublishRateInMsg

        public double getLastUpdatedAvgPublishRateInMsg()

      • getLastUpdatedAvgPublishRateInByte

        public double getLastUpdatedAvgPublishRateInByte()

      • getStats

        public org.apache.pulsar.common.policies.data.stats.TopicStatsImpl getStats​(boolean getPreciseBacklog,
                                                                            boolean subscriptionBacklogSize)
        Specified by:
        getStats in interface Topic

      • getInternalStats

        public java.util.concurrent.CompletableFuture<org.apache.pulsar.common.policies.data.PersistentTopicInternalStats> getInternalStats​(boolean includeLedgerMetadata)
        Specified by:
        getInternalStats in interface Topic

      • getBacklogSize

        public long getBacklogSize()

      • isActive

        public boolean isActive​(org.apache.pulsar.common.policies.data.InactiveTopicDeleteMode deleteMode)

      • checkGC

        public void checkGC()
        Specified by:
        checkGC in interface Topic

      • checkBackloggedCursors

        public void checkBackloggedCursors()
        Description copied from interface: Topic
        Activate cursors those caught up backlog-threshold entries and deactivate slow cursors which are creating backlog.
        Specified by:
        checkBackloggedCursors in interface Topic

      • onPoliciesUpdate

        public java.util.concurrent.CompletableFuture<java.lang.Void> onPoliciesUpdate​(org.apache.pulsar.common.policies.data.Policies data)
        Specified by:
        onPoliciesUpdate in interface Topic

      • getBacklogQuota

        public org.apache.pulsar.common.policies.data.BacklogQuota getBacklogQuota​(org.apache.pulsar.common.policies.data.BacklogQuota.BacklogQuotaType backlogQuotaType)
        Specified by:
        getBacklogQuota in interface Topic
        Returns:
        Backlog quota for topic

      • isBacklogQuotaExceeded

        public boolean isBacklogQuotaExceeded​(java.lang.String producerName,
                                      org.apache.pulsar.common.policies.data.BacklogQuota.BacklogQuotaType backlogQuotaType)
        Specified by:
        isBacklogQuotaExceeded in interface Topic
        Returns:
        quota exceeded status for blocking producer creation

      • isSizeBacklogExceeded

        public boolean isSizeBacklogExceeded()
        Returns:
        determine if backlog quota enforcement needs to be done for topic based on size limit

      • isTimeBacklogExceeded

        public boolean isTimeBacklogExceeded()
        Returns:
        determine if backlog quota enforcement needs to be done for topic based on time limit

      • isReplicated

        public boolean isReplicated()
        Specified by:
        isReplicated in interface Topic

      • terminate

        public java.util.concurrent.CompletableFuture<org.apache.pulsar.client.api.MessageId> terminate()

      • isOldestMessageExpired

        public boolean isOldestMessageExpired​(org.apache.bookkeeper.mledger.ManagedCursor cursor,
                                      int messageTTLInSeconds)

      • clearBacklog

        public java.util.concurrent.CompletableFuture<java.lang.Void> clearBacklog()
        Clears backlog for all cursors in the topic.
        Returns:

      • clearBacklog

        public java.util.concurrent.CompletableFuture<java.lang.Void> clearBacklog​(java.lang.String cursorName)
        Clears backlog for a given cursor in the topic.

        Note: For a replication cursor, just provide the remote cluster name

        Parameters:
        cursorName -
        Returns:

      • getSubscribeRateLimiter

        public java.util.Optional<SubscribeRateLimiter> getSubscribeRateLimiter()

      • getLastPublishedSequenceId

        public long getLastPublishedSequenceId​(java.lang.String producerName)

      • getLastPosition

        public org.apache.bookkeeper.mledger.Position getLastPosition()
        Specified by:
        getLastPosition in interface Topic

      • getLastMessageId

        public java.util.concurrent.CompletableFuture<org.apache.pulsar.client.api.MessageId> getLastMessageId()
        Specified by:
        getLastMessageId in interface Topic

      • compactionStatus

        public org.apache.pulsar.client.admin.LongRunningProcessStatus compactionStatus()

      • offloadStatus

        public org.apache.pulsar.client.admin.OffloadProcessStatus offloadStatus()

      • addSchemaIfIdleOrCheckCompatible

        public java.util.concurrent.CompletableFuture<java.lang.Void> addSchemaIfIdleOrCheckCompatible​(org.apache.pulsar.common.protocol.schema.SchemaData schema)
        Description copied from interface: Topic
        If the topic is idle (no producers, no entries, no subscribers and no existing schema), add the passed schema to the topic. Otherwise, check that the passed schema is compatible with what the topic already has.
        Specified by:
        addSchemaIfIdleOrCheckCompatible in interface Topic

      • checkReplicatedSubscriptionControllerState

        public void checkReplicatedSubscriptionControllerState()

      • isSystemTopic

        public boolean isSystemTopic()
        Specified by:
        isSystemTopic in interface Topic

      • isPersistent

        public boolean isPersistent()
        Specified by:
        isPersistent in interface Topic

      • publishTxnMessage

        public void publishTxnMessage​(org.apache.pulsar.client.api.transaction.TxnID txnID,
                              io.netty.buffer.ByteBuf headersAndPayload,
                              Topic.PublishContext publishContext)
        Description copied from interface: Topic
        Publish Transaction message to this Topic's TransactionBuffer.
        Specified by:
        publishTxnMessage in interface Topic
        Parameters:
        txnID - Transaction Id
        headersAndPayload - Message data
        publishContext - Publish context

      • endTxn

        public java.util.concurrent.CompletableFuture<java.lang.Void> endTxn​(org.apache.pulsar.client.api.transaction.TxnID txnID,
                                                                     int txnAction,
                                                                     long lowWaterMark)
        Description copied from interface: Topic
        End the transaction in this topic.
        Specified by:
        endTxn in interface Topic
        Parameters:
        txnID - Transaction id
        txnAction - Transaction action.
        lowWaterMark - low water mark of this tc
        Returns:

      • truncate

        public java.util.concurrent.CompletableFuture<java.lang.Void> truncate()
        Description copied from interface: Topic
        Truncate a topic. The truncate operation will move all cursors to the end of the topic and delete all inactive ledgers.
        Specified by:
        truncate in interface Topic
        Returns:

      • getDelayedDeliveryTickTimeMillis

        public long getDelayedDeliveryTickTimeMillis()

      • getMaxUnackedMessagesOnConsumer

        public int getMaxUnackedMessagesOnConsumer()

      • isDelayedDeliveryEnabled

        public boolean isDelayedDeliveryEnabled()

      • getMaxUnackedMessagesOnSubscription

        public int getMaxUnackedMessagesOnSubscription()

      • onUpdate

        public void onUpdate​(org.apache.pulsar.common.policies.data.TopicPolicies policies)
        Specified by:
        onUpdate in interface TopicPolicyListener<org.apache.pulsar.common.policies.data.TopicPolicies>

      • initTopicPolicy

        protected java.util.concurrent.CompletableFuture<java.lang.Void> initTopicPolicy()

      • checkSubscriptionTypesEnable

        public boolean checkSubscriptionTypesEnable​(org.apache.pulsar.common.api.proto.CommandSubscribe.SubType subType)
                                     throws java.lang.Exception
        Throws:
        java.lang.Exception

      • getTransactionBufferStats

        public org.apache.pulsar.common.policies.data.TransactionBufferStats getTransactionBufferStats()

      • getTransactionPendingAckStats

        public org.apache.pulsar.common.policies.data.TransactionPendingAckStats getTransactionPendingAckStats​(java.lang.String subName)

      • getMaxReadPosition

        public org.apache.bookkeeper.mledger.impl.PositionImpl getMaxReadPosition()

      • isTxnAborted

        public boolean isTxnAborted​(org.apache.pulsar.client.api.transaction.TxnID txnID)

      • getTransactionInBufferStats

        public org.apache.pulsar.common.policies.data.TransactionInBufferStats getTransactionInBufferStats​(org.apache.pulsar.client.api.transaction.TxnID txnID)

      • getTransactionInPendingAckStats

        public org.apache.pulsar.common.policies.data.TransactionInPendingAckStats getTransactionInPendingAckStats​(org.apache.pulsar.client.api.transaction.TxnID txnID,
                                                                                                           java.lang.String subName)

      • getPendingAckManagedLedger

        public java.util.concurrent.CompletableFuture<org.apache.bookkeeper.mledger.ManagedLedger> getPendingAckManagedLedger​(java.lang.String subName)

      • getLastDataMessagePublishedTimestamp

        public long getLastDataMessagePublishedTimestamp()