Package org.apache.bookkeeper.mledger

Interface LedgerOffloader

All Known Implementing Classes:
NullLedgerOffloader
@LimitedPrivate @Evolving public interface LedgerOffloader
Interface for offloading ledgers to long-term storage.

  • Field Details

  • Method Details

    • getOffloadDriverName

      String getOffloadDriverName()
      Get offload driver name.
      Returns:
      offload driver name.

    • getOffloadDriverMetadata

      default Map<String,String> getOffloadDriverMetadata()
      Get offload driver metadata.

      The driver metadata will be recorded as part of the metadata of the original ledger.

      Returns:
      offload driver metadata.

    • offload

      CompletableFuture<Void> offload(org.apache.bookkeeper.client.api.ReadHandle ledger, UUID uid, Map<String,String> extraMetadata)
      Offload the passed in ledger to longterm storage. Metadata passed in is for inspection purposes only and should be stored alongside the ledger data. When the returned future completes, the ledger has been persisted to the longterm storage, so it is safe to delete the original copy in bookkeeper. The uid is used to identify an attempt to offload. The implementation should use this to deterministically generate a unique name for the offloaded object. This uid will be stored in the managed ledger metadata before attempting the call to offload(). If a subsequent or concurrent call to offload() finds a uid in the metadata, it will attempt to cleanup this attempt with a call to #deleteOffloaded(ReadHandle,UUID). Once the offload attempt completes, the managed ledger will update its metadata again, to record the completion, ensuring that subsequent calls will not attempt to offload the same ledger again.
      Parameters:
      ledger - the ledger to offload
      uid - unique id to identity this offload attempt
      extraMetadata - metadata to be stored with the offloaded ledger for informational purposes
      Returns:
      a future, which when completed, denotes that the offload has been successful.

    • streamingOffload

      default CompletableFuture<LedgerOffloader.OffloadHandle> streamingOffload(ManagedLedger ml, UUID uid, long beginLedger, long beginEntry, Map<String,String> driverMetadata)
      Begin offload the passed in ledgers to longterm storage, it will finish when a segment reached it's size or time. Should only be called once for a LedgerOffloader instance. Metadata passed in is for inspection purposes only and should be stored alongside the segment data. When the returned OffloaderHandle.getOffloadResultAsync completes, the corresponding ledgers has been persisted to the longterm storage, so it is safe to delete the original copy in bookkeeper. The uid is used to identify an attempt to offload. The implementation should use this to deterministically generate a unique name for the offloaded object. This uid will be stored in the managed ledger metadata before attempting the call to streamingOffload(). If a subsequent or concurrent call to streamingOffload() finds a uid in the metadata, it will attempt to cleanup this attempt with a call to #deleteOffloaded(ReadHandle,UUID). Once the offload attempt completes, the managed ledger will update its metadata again, to record the completion, ensuring that subsequent calls will not attempt to offload the same ledger again.
      Returns:
      an OffloaderHandle, which when `completeFuture()` completed, denotes that the offload has been successful.

    • readOffloaded

      CompletableFuture<org.apache.bookkeeper.client.api.ReadHandle> readOffloaded(long ledgerId, UUID uid, Map<String,String> offloadDriverMetadata)
      Create a ReadHandle which can be used to read a ledger back from longterm storage. The passed uid, will be match the uid of a previous successful call to #offload(ReadHandle,UUID,Map).
      Parameters:
      ledgerId - the ID of the ledger to load from longterm storage
      uid - unique ID for previous successful offload attempt
      offloadDriverMetadata - offload driver metadata
      Returns:
      a future, which when completed, returns a ReadHandle

    • deleteOffloaded

      CompletableFuture<Void> deleteOffloaded(long ledgerId, UUID uid, Map<String,String> offloadDriverMetadata)
      Delete a ledger from long term storage. The passed uid, will be match the uid of a previous call to #offload(ReadHandle,UUID,Map), which may or may not have been successful.
      Parameters:
      ledgerId - the ID of the ledger to delete from longterm storage
      uid - unique ID for previous offload attempt
      offloadDriverMetadata - offload driver metadata
      Returns:
      a future, which when completed, signifies that the ledger has been deleted

    • readOffloaded

      default CompletableFuture<org.apache.bookkeeper.client.api.ReadHandle> readOffloaded(long ledgerId, MLDataFormats.OffloadContext ledgerContext, Map<String,String> offloadDriverMetadata)

    • deleteOffloaded

      default CompletableFuture<Void> deleteOffloaded(UUID uid, Map<String,String> offloadDriverMetadata)

    • getOffloadPolicies

      OffloadPolicies getOffloadPolicies()
      Get offload policies of this LedgerOffloader.
      Returns:
      offload policies

    • close

      void close()
      Close the resources if necessary.

    • scanLedgers

      default void scanLedgers(OffloadedLedgerMetadataConsumer consumer, Map<String,String> offloadDriverMetadata) throws ManagedLedgerException
      Scans all the ManagedLedgers stored on this Offloader (usually a Bucket). The callback should not modify/delete the ledgers.
      Parameters:
      consumer - receives the
      offloadDriverMetadata - additional metadata
      Throws:
      ManagedLedgerException