Package org.apache.bookkeeper.mledger

Interface LedgerOffloader

  • All Known Implementing Classes:
    NullLedgerOffloader
    @LimitedPrivate
@Evolving
public interface LedgerOffloader
    Interface for offloading ledgers to long-term storage.

    • Method Summary

      All Methods Instance Methods Abstract Methods Default Methods 
      Modifier and Type Method Description
      void close()
      Close the resources if necessary
      java.util.concurrent.CompletableFuture<java.lang.Void> deleteOffloaded​(long ledgerId, java.util.UUID uid, java.util.Map<java.lang.String,​java.lang.String> offloadDriverMetadata)
      Delete a ledger from long term storage.
      default java.util.concurrent.CompletableFuture<java.lang.Void> deleteOffloaded​(java.util.UUID uid, java.util.Map<java.lang.String,​java.lang.String> offloadDriverMetadata)  
      default java.util.Map<java.lang.String,​java.lang.String> getOffloadDriverMetadata()
      Get offload driver metadata.
      java.lang.String getOffloadDriverName()
      Get offload driver name.
      org.apache.pulsar.common.policies.data.OffloadPoliciesImpl getOffloadPolicies()
      Get offload policies of this LedgerOffloader
      java.util.concurrent.CompletableFuture<java.lang.Void> offload​(org.apache.bookkeeper.client.api.ReadHandle ledger, java.util.UUID uid, java.util.Map<java.lang.String,​java.lang.String> extraMetadata)
      Offload the passed in ledger to longterm storage.
      java.util.concurrent.CompletableFuture<org.apache.bookkeeper.client.api.ReadHandle> readOffloaded​(long ledgerId, java.util.UUID uid, java.util.Map<java.lang.String,​java.lang.String> offloadDriverMetadata)
      Create a ReadHandle which can be used to read a ledger back from longterm storage.
      default java.util.concurrent.CompletableFuture<org.apache.bookkeeper.client.api.ReadHandle> readOffloaded​(long ledgerId, MLDataFormats.OffloadContext ledgerContext, java.util.Map<java.lang.String,​java.lang.String> offloadDriverMetadata)  
      default java.util.concurrent.CompletableFuture<LedgerOffloader.OffloadHandle> streamingOffload​(ManagedLedger ml, java.util.UUID uid, long beginLedger, long beginEntry, java.util.Map<java.lang.String,​java.lang.String> driverMetadata)
      Begin offload the passed in ledgers to longterm storage, it will finish when a segment reached it's size or time.

    • Field Detail

      • METADATA_SOFTWARE_VERSION_KEY

        static final java.lang.String METADATA_SOFTWARE_VERSION_KEY
        See Also:
        Constant Field Values

      • METADATA_SOFTWARE_GITSHA_KEY

        static final java.lang.String METADATA_SOFTWARE_GITSHA_KEY
        See Also:
        Constant Field Values

    • Method Detail

      • getOffloadDriverName

        java.lang.String getOffloadDriverName()
        Get offload driver name.
        Returns:
        offload driver name.

      • getOffloadDriverMetadata

        default java.util.Map<java.lang.String,​java.lang.String> getOffloadDriverMetadata()
        Get offload driver metadata.

        The driver metadata will be recorded as part of the metadata of the original ledger.

        Returns:
        offload driver metadata.

      • offload

        java.util.concurrent.CompletableFuture<java.lang.Void> offload​(org.apache.bookkeeper.client.api.ReadHandle ledger,
                                                               java.util.UUID uid,
                                                               java.util.Map<java.lang.String,​java.lang.String> extraMetadata)
        Offload the passed in ledger to longterm storage. Metadata passed in is for inspection purposes only and should be stored alongside the ledger data. When the returned future completes, the ledger has been persisted to the longterm storage, so it is safe to delete the original copy in bookkeeper. The uid is used to identify an attempt to offload. The implementation should use this to deterministically generate a unique name for the offloaded object. This uid will be stored in the managed ledger metadata before attempting the call to offload(). If a subsequent or concurrent call to offload() finds a uid in the metadata, it will attempt to cleanup this attempt with a call to #deleteOffloaded(ReadHandle,UUID). Once the offload attempt completes, the managed ledger will update its metadata again, to record the completion, ensuring that subsequent calls will not attempt to offload the same ledger again.
        Parameters:
        ledger - the ledger to offload
        uid - unique id to identity this offload attempt
        extraMetadata - metadata to be stored with the offloaded ledger for informational purposes
        Returns:
        a future, which when completed, denotes that the offload has been successful.

      • streamingOffload

        default java.util.concurrent.CompletableFuture<LedgerOffloader.OffloadHandle> streamingOffload​(ManagedLedger ml,
                                                                                               java.util.UUID uid,
                                                                                               long beginLedger,
                                                                                               long beginEntry,
                                                                                               java.util.Map<java.lang.String,​java.lang.String> driverMetadata)
        Begin offload the passed in ledgers to longterm storage, it will finish when a segment reached it's size or time. Should only be called once for a LedgerOffloader instance. Metadata passed in is for inspection purposes only and should be stored alongside the segment data. When the returned OffloaderHandle.getOffloadResultAsync completes, the corresponding ledgers has been persisted to the longterm storage, so it is safe to delete the original copy in bookkeeper. The uid is used to identify an attempt to offload. The implementation should use this to deterministically generate a unique name for the offloaded object. This uid will be stored in the managed ledger metadata before attempting the call to streamingOffload(). If a subsequent or concurrent call to streamingOffload() finds a uid in the metadata, it will attempt to cleanup this attempt with a call to #deleteOffloaded(ReadHandle,UUID). Once the offload attempt completes, the managed ledger will update its metadata again, to record the completion, ensuring that subsequent calls will not attempt to offload the same ledger again.
        Returns:
        an OffloaderHandle, which when `completeFuture()` completed, denotes that the offload has been successful.

      • readOffloaded

        java.util.concurrent.CompletableFuture<org.apache.bookkeeper.client.api.ReadHandle> readOffloaded​(long ledgerId,
                                                                                                  java.util.UUID uid,
                                                                                                  java.util.Map<java.lang.String,​java.lang.String> offloadDriverMetadata)
        Create a ReadHandle which can be used to read a ledger back from longterm storage. The passed uid, will be match the uid of a previous successful call to #offload(ReadHandle,UUID,Map).
        Parameters:
        ledgerId - the ID of the ledger to load from longterm storage
        uid - unique ID for previous successful offload attempt
        offloadDriverMetadata - offload driver metadata
        Returns:
        a future, which when completed, returns a ReadHandle

      • deleteOffloaded

        java.util.concurrent.CompletableFuture<java.lang.Void> deleteOffloaded​(long ledgerId,
                                                                       java.util.UUID uid,
                                                                       java.util.Map<java.lang.String,​java.lang.String> offloadDriverMetadata)
        Delete a ledger from long term storage. The passed uid, will be match the uid of a previous call to #offload(ReadHandle,UUID,Map), which may or may not have been successful.
        Parameters:
        ledgerId - the ID of the ledger to delete from longterm storage
        uid - unique ID for previous offload attempt
        offloadDriverMetadata - offload driver metadata
        Returns:
        a future, which when completed, signifies that the ledger has been deleted

      • readOffloaded

        default java.util.concurrent.CompletableFuture<org.apache.bookkeeper.client.api.ReadHandle> readOffloaded​(long ledgerId,
                                                                                                          MLDataFormats.OffloadContext ledgerContext,
                                                                                                          java.util.Map<java.lang.String,​java.lang.String> offloadDriverMetadata)

      • deleteOffloaded

        default java.util.concurrent.CompletableFuture<java.lang.Void> deleteOffloaded​(java.util.UUID uid,
                                                                               java.util.Map<java.lang.String,​java.lang.String> offloadDriverMetadata)

      • getOffloadPolicies

        org.apache.pulsar.common.policies.data.OffloadPoliciesImpl getOffloadPolicies()
        Get offload policies of this LedgerOffloader
        Returns:
        offload policies

      • close

        void close()
        Close the resources if necessary