Package org.elasticsearch.repositories.blobstore

Class BlobStoreRepository

  • All Implemented Interfaces:
    Closeable, AutoCloseable, LifecycleComponent, Releasable, Repository
    Direct Known Subclasses:
    FsRepository
    public abstract class BlobStoreRepository
extends AbstractLifecycleComponent
implements Repository
    BlobStore - based implementation of Snapshot Repository

    This repository works with any BlobStore implementation. The blobStore could be (and preferred) lazy initialized in createBlobStore().

    BlobStoreRepository maintains the following structure in the blob store 

     
   STORE_ROOT
   |- index-N           - JSON serialized {@link RepositoryData} containing a list of all snapshot ids and the indices belonging to
   |                      each snapshot, N is the generation of the file
   |- index.latest      - contains the numeric value of the latest generation of the index file (i.e. N from above)
   |- incompatible-snapshots - list of all snapshot ids that are no longer compatible with the current version of the cluster
   |- snap-20131010.dat - SMILE serialized {@link SnapshotInfo} for snapshot "20131010"
   |- meta-20131010.dat - SMILE serialized {@link MetaData} for snapshot "20131010" (includes only global metadata)
   |- snap-20131011.dat - SMILE serialized {@link SnapshotInfo} for snapshot "20131011"
   |- meta-20131011.dat - SMILE serialized {@link MetaData} for snapshot "20131011"
   .....
   |- indices/ - data for all indices
      |- Ac1342-B_x/ - data for index "foo" which was assigned the unique id of Ac1342-B_x in the repository
      |  |- meta-20131010.dat - JSON Serialized {@link IndexMetaData} for index "foo"
      |  |- 0/ - data for shard "0" of index "foo"
      |  |  |- __1                      \  (files with numeric names were created by older ES versions)
      |  |  |- __2                      |
      |  |  |- __VPO5oDMVT5y4Akv8T_AO_A |- files from different segments see snap-* for their mappings to real segment files
      |  |  |- __1gbJy18wS_2kv1qI7FgKuQ |
      |  |  |- __R8JvZAHlSMyMXyZc2SS8Zg /
      |  |  .....
      |  |  |- snap-20131010.dat - SMILE serialized {@link BlobStoreIndexShardSnapshot} for snapshot "20131010"
      |  |  |- snap-20131011.dat - SMILE serialized {@link BlobStoreIndexShardSnapshot} for snapshot "20131011"
      |  |  |- index-123 - SMILE serialized {@link BlobStoreIndexShardSnapshots} for the shard
      |  |
      |  |- 1/ - data for shard "1" of index "foo"
      |  |  |- __1
      |  |  .....
      |  |
      |  |-2/
      |  ......
      |
      |- 1xB0D8_B3y/ - data for index "bar" which was assigned the unique id of 1xB0D8_B3y in the repository
      ......

    • Constructor Detail

      • BlobStoreRepository

        protected BlobStoreRepository​(RepositoryMetaData metadata,
                              boolean compress,
                              NamedXContentRegistry namedXContentRegistry,
                              ThreadPool threadPool)
        Constructs new BlobStoreRepository
        Parameters:
        metadata - The metadata for this repository including name and settings
        threadPool - Threadpool to run long running repository manipulations on asynchronously

    • Method Detail

      • getBlobStore

        protected BlobStore getBlobStore()

      • blobStore

        public BlobStore blobStore()
        Maintains single lazy instance of BlobStore. Public for testing.

      • createBlobStore

        protected abstract BlobStore createBlobStore()
                                      throws Exception
        Creates new BlobStore to read and write data.
        Throws:
        Exception

      • basePath

        public abstract BlobPath basePath()
        Returns base path of the repository

      • isCompress

        protected final boolean isCompress()
        Returns true if metadata and snapshot files should be compressed
        Returns:
        true if compression is needed

      • chunkSize

        protected ByteSizeValue chunkSize()
        Returns data file chunk size.

        This method should return null if no chunking is needed.

        Returns:
        chunk size

      • initializeSnapshot

        public void initializeSnapshot​(SnapshotId snapshotId,
                               List<IndexId> indices,
                               MetaData clusterMetaData)
        Description copied from interface: Repository
        Starts snapshotting process
        Specified by:
        initializeSnapshot in interface Repository
        Parameters:
        snapshotId - snapshot id
        indices - list of indices to be snapshotted
        clusterMetaData - cluster metadata

      • deleteSnapshot

        public void deleteSnapshot​(SnapshotId snapshotId,
                           long repositoryStateId,
                           ActionListener<Void> listener)
        Description copied from interface: Repository
        Deletes snapshot
        Specified by:
        deleteSnapshot in interface Repository
        Parameters:
        snapshotId - snapshot id
        repositoryStateId - the unique id identifying the state of the repository when the snapshot deletion began
        listener - completion listener

      • finalizeSnapshot

        public void finalizeSnapshot​(SnapshotId snapshotId,
                             List<IndexId> indices,
                             long startTime,
                             String failure,
                             int totalShards,
                             List<SnapshotShardFailure> shardFailures,
                             long repositoryStateId,
                             boolean includeGlobalState,
                             MetaData clusterMetaData,
                             Map<String,​Object> userMetadata,
                             ActionListener<SnapshotInfo> listener)
        Description copied from interface: Repository
        Finalizes snapshotting process

        This method is called on master after all shards are snapshotted.

        Specified by:
        finalizeSnapshot in interface Repository
        Parameters:
        snapshotId - snapshot id
        indices - list of indices in the snapshot
        startTime - start time of the snapshot
        failure - global failure reason or null
        totalShards - total number of shards
        shardFailures - list of shard failures
        repositoryStateId - the unique id identifying the state of the repository when the snapshot began
        includeGlobalState - include cluster global state
        listener - listener to be called on completion of the snapshot

      • getSnapshotInfo

        public SnapshotInfo getSnapshotInfo​(SnapshotId snapshotId)
        Description copied from interface: Repository
        Reads snapshot description from repository.
        Specified by:
        getSnapshotInfo in interface Repository
        Parameters:
        snapshotId - snapshot id
        Returns:
        information about snapshot

      • getSnapshotGlobalMetaData

        public MetaData getSnapshotGlobalMetaData​(SnapshotId snapshotId)
        Description copied from interface: Repository
        Returns global metadata associated with the snapshot.
        Specified by:
        getSnapshotGlobalMetaData in interface Repository
        Parameters:
        snapshotId - the snapshot id to load the global metadata from
        Returns:
        the global metadata about the snapshot

      • getSnapshotIndexMetaData

        public IndexMetaData getSnapshotIndexMetaData​(SnapshotId snapshotId,
                                              IndexId index)
                                       throws IOException
        Description copied from interface: Repository
        Returns the index metadata associated with the snapshot.
        Specified by:
        getSnapshotIndexMetaData in interface Repository
        Parameters:
        snapshotId - the snapshot id to load the index metadata from
        index - the IndexId to load the metadata from
        Returns:
        the index metadata about the given index for the given snapshot
        Throws:
        IOException

      • assertSnapshotOrGenericThread

        protected void assertSnapshotOrGenericThread()

      • startVerification

        public String startVerification()
        Description copied from interface: Repository
        Verifies repository on the master node and returns the verification token.

        If the verification token is not null, it's passed to all data nodes for verification. If it's null - no additional verification is required

        Specified by:
        startVerification in interface Repository
        Returns:
        verification token that should be passed to all Index Shard Repositories for additional verification or null

      • endVerification

        public void endVerification​(String seed)
        Description copied from interface: Repository
        Called at the end of repository verification process.

        This method should perform all necessary cleanup of the temporary files created in the repository

        Specified by:
        endVerification in interface Repository
        Parameters:
        seed - verification request generated by Repository.startVerification() command

      • isReadOnly

        public boolean isReadOnly()
        Description copied from interface: Repository
        Returns true if the repository supports only read operations
        Specified by:
        isReadOnly in interface Repository
        Returns:
        true if the repository is read/only

      • restoreShard

        public void restoreShard​(Store store,
                         SnapshotId snapshotId,
                         Version version,
                         IndexId indexId,
                         ShardId snapshotShardId,
                         RecoveryState recoveryState)
        Description copied from interface: Repository
        Restores snapshot of the shard.

        The index can be renamed on restore, hence different shardId and snapshotShardId are supplied.

        Specified by:
        restoreShard in interface Repository
        Parameters:
        store - the store to restore the index into
        snapshotId - snapshot id
        version - version of elasticsearch that created this snapshot
        indexId - id of the index in the repository from which the restore is occurring
        snapshotShardId - shard id (in the snapshot)
        recoveryState - recovery state

      • getShardSnapshotStatus

        public IndexShardSnapshotStatus getShardSnapshotStatus​(SnapshotId snapshotId,
                                                       Version version,
                                                       IndexId indexId,
                                                       ShardId shardId)
        Description copied from interface: Repository
        Retrieve shard snapshot status for the stored snapshot
        Specified by:
        getShardSnapshotStatus in interface Repository
        Parameters:
        snapshotId - snapshot id
        version - version of elasticsearch that created this snapshot
        indexId - the snapshotted index id for the shard to get status for
        shardId - shard id
        Returns:
        snapshot status