Package com.netflix.genie.web.data.services.impl.jpa

Class JpaPersistenceServiceImpl

java.lang.Object
com.netflix.genie.web.data.services.impl.jpa.JpaPersistenceServiceImpl
All Implemented Interfaces:
PersistenceService
@Transactional(rollbackFor={com.netflix.genie.common.exceptions.GenieException.class,com.netflix.genie.common.internal.exceptions.checked.GenieCheckedException.class,com.netflix.genie.common.internal.exceptions.unchecked.GenieRuntimeException.class,javax.validation.ConstraintViolationException.class}) public class JpaPersistenceServiceImpl extends Object implements PersistenceService
Implementation of PersistenceService using JPA.
Since:
4.0.0

  • Constructor Summary

    Constructors
    Constructor
    Description
    JpaPersistenceServiceImpl(javax.persistence.EntityManager entityManager, JpaRepositories jpaRepositories, com.netflix.genie.common.internal.tracing.brave.BraveTracingComponents tracingComponents)
    Constructor.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    addApplicationsForCommand(@NotBlank String id, @NotEmpty List<@NotBlank String> applicationIds)
    Add applications for the command.
    void
    addClusterCriterionForCommand(String id, @Valid com.netflix.genie.common.internal.dtos.Criterion criterion)
    Add a new Criterion to the existing list of cluster criteria for the command identified by id.
    void
    addClusterCriterionForCommand(String id, @Valid com.netflix.genie.common.internal.dtos.Criterion criterion, @javax.validation.constraints.Min(0L) int priority)
    Add a new Criterion to the existing list of cluster criteria for the command identified by id.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    addConfigsToResource(@NotBlank String id, Set<@Size(max=1024) String> configs, Class<R> resourceClass)
    Add configuration files to the existing set for a resource.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    addDependenciesToResource(@NotBlank String id, Set<@Size(max=1024) String> dependencies, Class<R> resourceClass)
    Add dependency files to the existing set for a resource.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    addTagsToResource(@NotBlank String id, Set<@Size(max=255) String> tags, Class<R> resourceClass)
    Add tags to the existing set for a resource.
    void
    claimJob(@NotBlank String id, @Valid com.netflix.genie.common.internal.dtos.AgentClientMetadata agentClientMetadata)
    Set a job identified by id to be owned by the agent identified by agentClientMetadata.
    void
    deleteAllApplications()
    Delete all applications from the system.
    void
    deleteAllClusters()
    Delete all clusters from database.
    void
    deleteAllCommands()
    Delete all commands from the system.
    void
    deleteApplication(@NotBlank String id)
    Delete an Application from the system.
    void
    deleteCluster(@NotBlank String id)
    Delete a Cluster by id.
    void
    deleteCommand(@NotBlank String id)
    Delete a Command from system.
    long
    deleteJobsCreatedBefore(@NotNull Instant creationThreshold, @NotNull Set<com.netflix.genie.common.internal.dtos.JobStatus> excludeStatuses, @javax.validation.constraints.Min(1L) int batchSize)
    This method will delete a chunk of jobs whose creation time is earlier than the given date.
    long
    deleteUnusedApplications(Instant createdThreshold, int batchSize)
    Delete any unused applications that were created before the given time.
    long
    deleteUnusedClusters(Set<com.netflix.genie.common.internal.dtos.ClusterStatus> deleteStatuses, Instant clusterCreatedThreshold, int batchSize)
    Delete all clusters that are in one of the given states, aren't attached to any jobs and were created before the given time.
    long
    deleteUnusedCommands(Set<com.netflix.genie.common.internal.dtos.CommandStatus> deleteStatuses, Instant commandCreatedThreshold, int batchSize)
    Bulk delete commands from the database where their status is in deleteStatuses they were created before commandCreatedThreshold and they aren't attached to any jobs still in the database.
    long
    deleteUnusedFiles(@NotNull Instant createdThresholdLowerBound, @NotNull Instant createdThresholdUpperBound, @javax.validation.constraints.Min(1L) int batchSize)
    Delete all files from the database that aren't referenced which were created before the supplied created threshold.
    long
    deleteUnusedTags(@NotNull Instant createdThreshold, @javax.validation.constraints.Min(1L) int batchSize)
    Delete all tags from the database that aren't referenced which were created before the supplied created threshold.
    org.springframework.data.domain.Page<com.netflix.genie.common.internal.dtos.Application>
    findApplications(String name, String user, Set<com.netflix.genie.common.internal.dtos.ApplicationStatus> statuses, Set<String> tags, String type, org.springframework.data.domain.Pageable page)
    Find applications which match the given filter criteria.
    org.springframework.data.domain.Page<com.netflix.genie.common.internal.dtos.Cluster>
    findClusters(String name, Set<com.netflix.genie.common.internal.dtos.ClusterStatus> statuses, Set<String> tags, Instant minUpdateTime, Instant maxUpdateTime, org.springframework.data.domain.Pageable page)
    Find and Clusters that match the given parameters.
    Set<com.netflix.genie.common.internal.dtos.Cluster>
    findClustersMatchingAnyCriterion(@NotEmpty Set<@Valid com.netflix.genie.common.internal.dtos.Criterion> criteria, boolean addDefaultStatus)
    Find all the Cluster's that match any of the given Criterion.
    Set<com.netflix.genie.common.internal.dtos.Cluster>
    findClustersMatchingCriterion(@Valid com.netflix.genie.common.internal.dtos.Criterion criterion, boolean addDefaultStatus)
    Find all the Cluster's that match the given Criterion.
    org.springframework.data.domain.Page<com.netflix.genie.common.internal.dtos.Command>
    findCommands(String name, String user, Set<com.netflix.genie.common.internal.dtos.CommandStatus> statuses, Set<String> tags, org.springframework.data.domain.Pageable page)
    Find commands matching the given filter criteria.
    Set<com.netflix.genie.common.internal.dtos.Command>
    findCommandsMatchingCriterion(@Valid com.netflix.genie.common.internal.dtos.Criterion criterion, boolean addDefaultStatus)
    Find all the Command's that match the given Criterion.
    org.springframework.data.domain.Page<com.netflix.genie.common.dto.search.JobSearchResult>
    findJobs(String id, String name, String user, Set<com.netflix.genie.common.dto.JobStatus> statuses, Set<String> tags, String clusterName, String clusterId, String commandName, String commandId, Instant minStarted, Instant maxStarted, Instant minFinished, Instant maxFinished, String grouping, String groupingInstance, @NotNull org.springframework.data.domain.Pageable page)
    Find jobs which match the given filter criteria.
    long
    getActiveJobCountForUser(@NotBlank String user)
    Get the count of 'active' jobs for a given user across all instances.
    Set<String>
    getActiveJobs()
    Get the set of active jobs.
    com.netflix.genie.common.internal.dtos.Application
    getApplication(@NotBlank String id)
    Get the application metadata for given id.
    List<com.netflix.genie.common.internal.dtos.Application>
    getApplicationsForCommand(String id)
    Get the applications for a given command.
    com.netflix.genie.common.internal.dtos.Cluster
    getCluster(@NotBlank String id)
    Get the Cluster identified by the given id.
    List<com.netflix.genie.common.internal.dtos.Criterion>
    getClusterCriteriaForCommand(String id)
    For the given command id return the Cluster Criterion in priority order that is currently associated with this command if any.
    Set<com.netflix.genie.common.internal.dtos.Cluster>
    getClustersForCommand(@NotBlank String id, Set<com.netflix.genie.common.internal.dtos.ClusterStatus> statuses)
    Get all the clusters the command with given id is associated with.
    com.netflix.genie.common.internal.dtos.Command
    getCommand(@NotBlank String id)
    Get the metadata for the Command identified by the id.
    Set<com.netflix.genie.common.internal.dtos.Command>
    getCommandsForApplication(@NotBlank String id, Set<com.netflix.genie.common.internal.dtos.CommandStatus> statuses)
    Get all the commands the application with given id is associated with.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    Set<String>
    getConfigsForResource(@NotBlank String id, Class<R> resourceClass)
    Get the set of configuration files associated with the resource with the given id.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    Set<String>
    getDependenciesForResource(@NotBlank String id, Class<R> resourceClass)
    Get the set of dependency files associated with the resource with the given id.
    com.netflix.genie.common.internal.dtos.FinishedJob
    getFinishedJob(@NotBlank String id)
    Get a DTO representing a finished job.
    JobInfoAggregate
    getHostJobInformation(@NotBlank String hostname)
    Get all the aggregate metadata information about jobs running on a given hostname.
    com.netflix.genie.common.dto.Job
    getJob(@NotBlank String id)
    Get job information for given job id.
    List<com.netflix.genie.common.internal.dtos.Application>
    getJobApplications(@NotBlank String id)
    Get the applications the job used or is currently using.
    Optional<String>
    getJobArchiveLocation(@NotBlank String id)
    Get the location a job directory was archived to if at all.
    com.netflix.genie.common.internal.dtos.ArchiveStatus
    getJobArchiveStatus(@NotBlank String id)
    Get the archive status for a job with the given id.
    com.netflix.genie.common.internal.dtos.Cluster
    getJobCluster(@NotBlank String id)
    Get the cluster the job used or is using.
    com.netflix.genie.common.internal.dtos.Command
    getJobCommand(@NotBlank String id)
    Get the command the job used or is using.
    com.netflix.genie.common.dto.JobExecution
    getJobExecution(@NotBlank String id)
    Get job execution for given job id.
    com.netflix.genie.common.dto.JobMetadata
    getJobMetadata(@NotBlank String id)
    Get the metadata about a job.
    com.netflix.genie.common.internal.dtos.JobRequest
    getJobRequest(@NotBlank String id)
    Get the original request for a job.
    Optional<com.netflix.genie.common.internal.dtos.JobSpecification>
    getJobSpecification(@NotBlank String id)
    Get the saved job specification for the given job.
    com.netflix.genie.common.internal.dtos.JobStatus
    getJobStatus(@NotBlank String id)
    Get the status for a job with the given id.
    Set<String>
    getJobsWithStatusAndArchiveStatusUpdatedBefore(@NotEmpty Set<com.netflix.genie.common.internal.dtos.JobStatus> statuses, @NotEmpty Set<com.netflix.genie.common.internal.dtos.ArchiveStatus> archiveStatuses, @NotNull Instant updated)
    Get the set of jobs (agent only) whose state is in statuses and archive status is in archiveStatuses and last updated before updated.
    com.fasterxml.jackson.databind.JsonNode
    getLauncherExt(@NotBlank String id)
    Get the job requested launcher extension.
    com.fasterxml.jackson.databind.JsonNode
    getRequestedLauncherExt(@NotBlank String id)
    Get the command the job used or is using.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    Set<String>
    getTagsForResource(@NotBlank String id, Class<R> resourceClass)
    Get the set of tags associated with the resource with the given id.
    Set<String>
    getUnclaimedJobs()
    Get the set of jobs in that have not reached CLAIMED state.
    long
    getUsedMemoryOnHost(@NotBlank String hostname)
    Get the amount of memory currently used on the given host by Genie jobs in any of the following states.
    Map<String,com.netflix.genie.common.dto.UserResourcesSummary>
    getUserResourcesSummaries(Set<com.netflix.genie.common.internal.dtos.JobStatus> statuses, boolean api)
    Get a map of summaries of resources usage for each user with at least one active job.
    boolean
    isApiJob(@NotBlank String id)
    Get whether the job with the given ID was submitted via the REST API or other mechanism.
    void
    removeAllClusterCriteriaForCommand(String id)
    Remove all the Criterion currently associated with the command identified by id.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    removeAllConfigsForResource(@NotBlank String id, Class<R> resourceClass)
    Remove all configuration files from the resource.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    removeAllDependenciesForResource(@NotBlank String id, Class<R> resourceClass)
    Remove all dependency files from the resource.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    removeAllTagsForResource(@NotBlank String id, Class<R> resourceClass)
    Remove all tags from the resource.
    void
    removeApplicationForCommand(@NotBlank String id, @NotBlank String appId)
    Remove the application from the command.
    void
    removeApplicationsForCommand(@NotBlank String id)
    Remove all the applications from the command.
    void
    removeClusterCriterionForCommand(String id, @javax.validation.constraints.Min(0L) int priority)
    Remove the Criterion with the given priority from the current list of cluster criteria associated with the command identified by id.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    removeConfigForResource(@NotBlank String id, @NotBlank String config, Class<R> resourceClass)
    Remove a configuration file from the given resource.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    removeDependencyForResource(@NotBlank String id, @NotBlank String dependency, Class<R> resourceClass)
    Remove a dependency file from the given resource.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    removeTagForResource(@NotBlank String id, @NotBlank String tag, Class<R> resourceClass)
    Remove a tag from the given resource.
    String
    saveApplication(@Valid com.netflix.genie.common.internal.dtos.ApplicationRequest applicationRequest)
    Save a new application.
    String
    saveCluster(@Valid com.netflix.genie.common.internal.dtos.ClusterRequest clusterRequest)
    Save a Cluster.
    String
    saveCommand(@Valid com.netflix.genie.common.internal.dtos.CommandRequest commandRequest)
    Save a Command in the system based on the given CommandRequest.
    String
    saveJobSubmission(@Valid JobSubmission jobSubmission)
    Save the given job submission information in the underlying data store.
    void
    saveResolvedJob(@NotBlank String id, @Valid ResolvedJob resolvedJob)
    Save the given resolved details for a job.
    void
    setApplicationsForCommand(@NotBlank String id, @NotNull List<@NotBlank String> applicationIds)
    Set the applications for the command.
    void
    setClusterCriteriaForCommand(String id, List<@Valid com.netflix.genie.common.internal.dtos.Criterion> clusterCriteria)
    For the command identified by id reset the entire list of cluster criteria to match the contents of clusterCriteria.
    void
    updateApplication(@NotBlank String id, @Valid com.netflix.genie.common.internal.dtos.Application updateApp)
    Update an Application.
    void
    updateCluster(@NotBlank String id, @Valid com.netflix.genie.common.internal.dtos.Cluster updateCluster)
    Update a Cluster with the given information.
    void
    updateCommand(@NotBlank String id, @Valid com.netflix.genie.common.internal.dtos.Command updateCommand)
    Update a Command.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    updateConfigsForResource(@NotBlank String id, Set<@Size(max=1024) String> configs, Class<R> resourceClass)
    Update the set of configuration files associated with the resource with the given id.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    updateDependenciesForResource(@NotBlank String id, Set<@Size(max=1024) String> dependencies, Class<R> resourceClass)
    Update the set of dependency files associated with the resource with the given id.
    void
    updateJobArchiveStatus(@NotBlank(message="No job id entered. Unable to update.") String id, @NotNull(message="Status cannot be null.") com.netflix.genie.common.internal.dtos.ArchiveStatus archiveStatus)
    Update the status and status message of the job.
    com.netflix.genie.common.internal.dtos.JobStatus
    updateJobStatus(@NotBlank String id, @NotNull com.netflix.genie.common.internal.dtos.JobStatus currentStatus, @NotNull com.netflix.genie.common.internal.dtos.JobStatus newStatus, String newStatusMessage)
    Update the status of the job identified with id to be newStatus provided that the current status of the job matches newStatus.
    void
    updateLauncherExt(@NotBlank(message="No job id entered. Unable to update.") String id, @NotNull(message="Status cannot be null.") com.fasterxml.jackson.databind.JsonNode launcherExtension)
    Update the launcher extension field for this job.
    void
    updateRequestedLauncherExt(@NotBlank(message="No job id entered. Unable to update.") String id, @NotNull(message="Status cannot be null.") com.fasterxml.jackson.databind.JsonNode launcherExtension)
    Update the requested launcher extension field for this job.
    int
    updateStatusForUnusedCommands(com.netflix.genie.common.internal.dtos.CommandStatus desiredStatus, Instant commandCreatedThreshold, Set<com.netflix.genie.common.internal.dtos.CommandStatus> currentStatuses, int batchSize)
    Update the status of a command to the desiredStatus if its status is in currentStatuses, it was created before commandCreatedThreshold and it hasn't been used in any job.
    <R extends com.netflix.genie.common.internal.dtos.CommonResource>
    void
    updateTagsForResource(@NotBlank String id, Set<@Size(max=255) String> tags, Class<R> resourceClass)
    Update the set of tags associated with the resource with the given id.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • JpaPersistenceServiceImpl

      public JpaPersistenceServiceImpl(javax.persistence.EntityManager entityManager, JpaRepositories jpaRepositories, com.netflix.genie.common.internal.tracing.brave.BraveTracingComponents tracingComponents)
      Constructor.
      Parameters:
      entityManager - The EntityManager to use
      jpaRepositories - All the repositories in the Genie application
      tracingComponents - All the Brave related tracing components needed to add metadata to Spans

  • Method Details

    • saveApplication

      public String saveApplication(@Valid @Valid com.netflix.genie.common.internal.dtos.ApplicationRequest applicationRequest) throws IdAlreadyExistsException
      Save a new application.
      Specified by:
      saveApplication in interface PersistenceService
      Parameters:
      applicationRequest - The ApplicationRequest containing the metadata of the application to create
      Returns:
      The unique id of the application that was created
      Throws:
      IdAlreadyExistsException - If the ID is already used by another application

    • getApplication

      public com.netflix.genie.common.internal.dtos.Application getApplication(@NotBlank @NotBlank String id) throws NotFoundException
      Get the application metadata for given id.
      Specified by:
      getApplication in interface PersistenceService
      Parameters:
      id - unique id for application configuration to get. Not null/empty.
      Returns:
      The Application
      Throws:
      NotFoundException - if no application with id exists

    • findApplications

      @Transactional(readOnly=true) public org.springframework.data.domain.Page<com.netflix.genie.common.internal.dtos.Application> findApplications(@Nullable String name, @Nullable String user, @Nullable Set<com.netflix.genie.common.internal.dtos.ApplicationStatus> statuses, @Nullable Set<String> tags, @Nullable String type, org.springframework.data.domain.Pageable page)
      Find applications which match the given filter criteria.
      Specified by:
      findApplications in interface PersistenceService
      Parameters:
      name - Name of the application. Can be null or empty.
      user - The user who created the application. Can be null/empty
      statuses - The statuses of the applications to find. Can be null.
      tags - Tags allocated to this application
      type - The type of the application to find
      page - The page requested for the search results
      Returns:
      The page of found applications

    • updateApplication

      public void updateApplication(@NotBlank @NotBlank String id, @Valid @Valid com.netflix.genie.common.internal.dtos.Application updateApp) throws NotFoundException, PreconditionFailedException
      Update an Application.
      Specified by:
      updateApplication in interface PersistenceService
      Parameters:
      id - The id of the application to update
      updateApp - Information to update for the application configuration with
      Throws:
      NotFoundException - If no Application with id exists
      PreconditionFailedException - If id and updateApp id don't match

    • deleteAllApplications

      public void deleteAllApplications() throws PreconditionFailedException
      Delete all applications from the system.
      Specified by:
      deleteAllApplications in interface PersistenceService
      Throws:
      PreconditionFailedException - When any Application is still associated with a command

    • deleteApplication

      public void deleteApplication(@NotBlank @NotBlank String id) throws PreconditionFailedException
      Delete an Application from the system.
      Specified by:
      deleteApplication in interface PersistenceService
      Parameters:
      id - unique id of the application to delete
      Throws:
      PreconditionFailedException - When the Application is still used by any command

    • getCommandsForApplication

      public Set<com.netflix.genie.common.internal.dtos.Command> getCommandsForApplication(@NotBlank @NotBlank String id, @Nullable Set<com.netflix.genie.common.internal.dtos.CommandStatus> statuses) throws NotFoundException
      Get all the commands the application with given id is associated with.
      Specified by:
      getCommandsForApplication in interface PersistenceService
      Parameters:
      id - The id of the application to get the commands for.
      statuses - The desired status(es) to filter the associated commands for
      Returns:
      The commands the application is used by
      Throws:
      NotFoundException - If no Application with id exists

    • deleteUnusedApplications

      @Transactional(isolation=READ_COMMITTED) public long deleteUnusedApplications(Instant createdThreshold, int batchSize)
      Delete any unused applications that were created before the given time. Unused means they aren't linked to any other resources in the Genie system like jobs or commands and therefore referential integrity is maintained.
      Specified by:
      deleteUnusedApplications in interface PersistenceService
      Parameters:
      createdThreshold - The instant in time that any application had to be created before (exclusive) to be considered for deletion. Presents ability to filter out newly created applications if desired.
      batchSize - The maximum number of applications to delete in a single transaction
      Returns:
      The number of successfully deleted applications

    • saveCluster

      public String saveCluster(@Valid @Valid com.netflix.genie.common.internal.dtos.ClusterRequest clusterRequest) throws IdAlreadyExistsException
      Save a Cluster.
      Specified by:
      saveCluster in interface PersistenceService
      Parameters:
      clusterRequest - The cluster information to save
      Returns:
      The id of the saved cluster
      Throws:
      IdAlreadyExistsException - If a Cluster with the given id already exists

    • getCluster

      public com.netflix.genie.common.internal.dtos.Cluster getCluster(@NotBlank @NotBlank String id) throws NotFoundException
      Get the Cluster identified by the given id.
      Specified by:
      getCluster in interface PersistenceService
      Parameters:
      id - unique id of the Cluster to get
      Returns:
      The Cluster
      Throws:
      NotFoundException - If no Cluster with id exists

    • findClusters

      @Transactional(readOnly=true) public org.springframework.data.domain.Page<com.netflix.genie.common.internal.dtos.Cluster> findClusters(@Nullable String name, @Nullable Set<com.netflix.genie.common.internal.dtos.ClusterStatus> statuses, @Nullable Set<String> tags, @Nullable Instant minUpdateTime, @Nullable Instant maxUpdateTime, org.springframework.data.domain.Pageable page)
      Find and Clusters that match the given parameters. Null or empty parameters are ignored.
      Specified by:
      findClusters in interface PersistenceService
      Parameters:
      name - cluster name
      statuses - ClusterStatus that clusters must be in to be matched
      tags - tags attached to this cluster
      minUpdateTime - min time when cluster was updated
      maxUpdateTime - max time when cluster was updated
      page - The page to get
      Returns:
      All the clusters matching the criteria

    • updateCluster

      public void updateCluster(@NotBlank @NotBlank String id, @Valid @Valid com.netflix.genie.common.internal.dtos.Cluster updateCluster) throws NotFoundException, PreconditionFailedException
      Update a Cluster with the given information.
      Specified by:
      updateCluster in interface PersistenceService
      Parameters:
      id - The id of the cluster to update
      updateCluster - The information to update the cluster with
      Throws:
      NotFoundException - If no Cluster with id exists
      PreconditionFailedException - If the id doesn't match the updateCluster id

    • deleteAllClusters

      public void deleteAllClusters() throws PreconditionFailedException
      Delete all clusters from database.
      Specified by:
      deleteAllClusters in interface PersistenceService
      Throws:
      PreconditionFailedException - If the cluster is still associated with any job

    • deleteCluster

      public void deleteCluster(@NotBlank @NotBlank String id) throws PreconditionFailedException
      Delete a Cluster by id.
      Specified by:
      deleteCluster in interface PersistenceService
      Parameters:
      id - unique id of the cluster to delete
      Throws:
      PreconditionFailedException - If the cluster is still associated with any job

    • deleteUnusedClusters

      @Transactional(isolation=READ_COMMITTED) public long deleteUnusedClusters(Set<com.netflix.genie.common.internal.dtos.ClusterStatus> deleteStatuses, Instant clusterCreatedThreshold, int batchSize)
      Delete all clusters that are in one of the given states, aren't attached to any jobs and were created before the given time.
      Specified by:
      deleteUnusedClusters in interface PersistenceService
      Parameters:
      deleteStatuses - The set of ClusterStatus a cluster must be in to be considered for deletion.
      clusterCreatedThreshold - The instant in time before which a cluster must have been created to be considered for deletion. Exclusive.
      batchSize - The maximum number of clusters to delete in a single transaction
      Returns:
      The number of clusters deleted

    • findClustersMatchingCriterion

      public Set<com.netflix.genie.common.internal.dtos.Cluster> findClustersMatchingCriterion(@Valid @Valid com.netflix.genie.common.internal.dtos.Criterion criterion, boolean addDefaultStatus)
      Find all the Cluster's that match the given Criterion.
      Specified by:
      findClustersMatchingCriterion in interface PersistenceService
      Parameters:
      criterion - The Criterion supplied that each cluster needs to completely match to be returned
      addDefaultStatus - true if the a default status should be added to the supplied Criterion if the supplied criterion doesn't already have a status
      Returns:
      All the Cluster's which matched the Criterion

    • findClustersMatchingAnyCriterion

      public Set<com.netflix.genie.common.internal.dtos.Cluster> findClustersMatchingAnyCriterion(@NotEmpty @NotEmpty Set<@Valid com.netflix.genie.common.internal.dtos.Criterion> criteria, boolean addDefaultStatus)
      Find all the Cluster's that match any of the given Criterion.
      Specified by:
      findClustersMatchingAnyCriterion in interface PersistenceService
      Parameters:
      criteria - The set of Criterion supplied that a cluster needs to completely match at least one of to be returned
      addDefaultStatus - true if the a default status should be added to the supplied Criterion if the supplied criterion doesn't already have a status
      Returns:
      All the Cluster's which matched the Criterion

    • saveCommand

      public String saveCommand(@Valid @Valid com.netflix.genie.common.internal.dtos.CommandRequest commandRequest) throws IdAlreadyExistsException
      Save a Command in the system based on the given CommandRequest.
      Specified by:
      saveCommand in interface PersistenceService
      Parameters:
      commandRequest - encapsulates the command configuration information to create
      Returns:
      The id of the command that was saved
      Throws:
      IdAlreadyExistsException - If there was a conflict on the unique id for the command

    • getCommand

      public com.netflix.genie.common.internal.dtos.Command getCommand(@NotBlank @NotBlank String id) throws NotFoundException
      Get the metadata for the Command identified by the id.
      Specified by:
      getCommand in interface PersistenceService
      Parameters:
      id - unique id for command to get. Not null/empty.
      Returns:
      The command
      Throws:
      NotFoundException - If no Command exists with the given id

    • findCommands

      @Transactional(readOnly=true) public org.springframework.data.domain.Page<com.netflix.genie.common.internal.dtos.Command> findCommands(@Nullable String name, @Nullable String user, @Nullable Set<com.netflix.genie.common.internal.dtos.CommandStatus> statuses, @Nullable Set<String> tags, org.springframework.data.domain.Pageable page)
      Find commands matching the given filter criteria.
      Specified by:
      findCommands in interface PersistenceService
      Parameters:
      name - Name of command config
      user - The name of the user who created the configuration
      statuses - The status of the commands to get. Can be null.
      tags - tags allocated to this command
      page - The page of results to get
      Returns:
      All the commands matching the specified criteria

    • updateCommand

      public void updateCommand(@NotBlank @NotBlank String id, @Valid @Valid com.netflix.genie.common.internal.dtos.Command updateCommand) throws NotFoundException, PreconditionFailedException
      Update a Command.
      Specified by:
      updateCommand in interface PersistenceService
      Parameters:
      id - The id of the command configuration to update. Not null or empty.
      updateCommand - contains the information to update the command with
      Throws:
      NotFoundException - If no Command exists with the given id
      PreconditionFailedException - When the id doesn't match the id of updateCommand

    • deleteAllCommands

      public void deleteAllCommands() throws PreconditionFailedException
      Delete all commands from the system.
      Specified by:
      deleteAllCommands in interface PersistenceService
      Throws:
      PreconditionFailedException - If any constraint is violated trying to delete a command

    • deleteCommand

      public void deleteCommand(@NotBlank @NotBlank String id) throws NotFoundException
      Delete a Command from system.
      Specified by:
      deleteCommand in interface PersistenceService
      Parameters:
      id - unique if of the command to delete
      Throws:
      NotFoundException - If no Command exists with the given id

    • addApplicationsForCommand

      public void addApplicationsForCommand(@NotBlank @NotBlank String id, @NotEmpty @NotEmpty List<@NotBlank String> applicationIds) throws NotFoundException, PreconditionFailedException
      Add applications for the command.
      Specified by:
      addApplicationsForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to add the application file to. Not null/empty/blank.
      applicationIds - The ids of the applications to add. Not null.
      Throws:
      NotFoundException - If no Command exists with the given id or one of the applications doesn't exist
      PreconditionFailedException - If an application with one of the ids is already associated with the command

    • setApplicationsForCommand

      public void setApplicationsForCommand(@NotBlank @NotBlank String id, @NotNull @NotNull List<@NotBlank String> applicationIds) throws NotFoundException, PreconditionFailedException
      Set the applications for the command.
      Specified by:
      setApplicationsForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to add the application file to. Not null/empty/blank.
      applicationIds - The ids of the applications to set. Not null.
      Throws:
      NotFoundException - If no Command exists with the given id or one of the applications doesn't exist
      PreconditionFailedException - If there are duplicate application ids in the list

    • getApplicationsForCommand

      public List<com.netflix.genie.common.internal.dtos.Application> getApplicationsForCommand(String id) throws NotFoundException
      Get the applications for a given command.
      Specified by:
      getApplicationsForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to get the application for. Not null/empty/blank.
      Returns:
      The applications or exception if none exist.
      Throws:
      NotFoundException - If no Command exists with the given id

    • removeApplicationsForCommand

      public void removeApplicationsForCommand(@NotBlank @NotBlank String id) throws NotFoundException, PreconditionFailedException
      Remove all the applications from the command.
      Specified by:
      removeApplicationsForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to remove the application from. Not null/empty/blank.
      Throws:
      NotFoundException - If no Command exists with the given id
      PreconditionFailedException - If applications are unable to be removed

    • removeApplicationForCommand

      public void removeApplicationForCommand(@NotBlank @NotBlank String id, @NotBlank @NotBlank String appId) throws NotFoundException
      Remove the application from the command.
      Specified by:
      removeApplicationForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to remove the application from. Not null/empty/blank.
      appId - The id of the application to remove. Not null/empty/blank
      Throws:
      NotFoundException - If no Command exists with the given id

    • getClustersForCommand

      public Set<com.netflix.genie.common.internal.dtos.Cluster> getClustersForCommand(@NotBlank @NotBlank String id, @Nullable Set<com.netflix.genie.common.internal.dtos.ClusterStatus> statuses) throws NotFoundException
      Get all the clusters the command with given id is associated with.
      Specified by:
      getClustersForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to get the clusters for.
      statuses - The status of the clusters returned
      Returns:
      The clusters the command is available on.
      Throws:
      NotFoundException - If no Command exists with the given id

    • getClusterCriteriaForCommand

      public List<com.netflix.genie.common.internal.dtos.Criterion> getClusterCriteriaForCommand(String id) throws NotFoundException
      For the given command id return the Cluster Criterion in priority order that is currently associated with this command if any.
      Specified by:
      getClusterCriteriaForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to get the criteria for
      Returns:
      The cluster criteria in priority order
      Throws:
      NotFoundException - If no command with id exists

    • addClusterCriterionForCommand

      public void addClusterCriterionForCommand(String id, @Valid @Valid com.netflix.genie.common.internal.dtos.Criterion criterion) throws NotFoundException
      Add a new Criterion to the existing list of cluster criteria for the command identified by id. This new criterion will be the lowest priority criterion.
      Specified by:
      addClusterCriterionForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to add to
      criterion - The new Criterion to add
      Throws:
      NotFoundException - If no command with id exists

    • addClusterCriterionForCommand

      public void addClusterCriterionForCommand(String id, @Valid @Valid com.netflix.genie.common.internal.dtos.Criterion criterion, @Min(0L) @javax.validation.constraints.Min(0L) int priority) throws NotFoundException
      Add a new Criterion to the existing list of cluster criteria for the command identified by id. The priority is the place in the list this new criterion should be placed. A value of 0 indicates it should be placed at the front of the list with the highest possible priority. 1 would be second in the list etc. If priority is > the current size of the cluster criteria list this new criterion will be placed at the end as the lowest priority item.
      Specified by:
      addClusterCriterionForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to add to
      criterion - The new Criterion to add
      priority - The place in the existing cluster criteria list this new criterion should be placed. Min 0.
      Throws:
      NotFoundException - If no command with id exists

    • setClusterCriteriaForCommand

      public void setClusterCriteriaForCommand(String id, List<@Valid com.netflix.genie.common.internal.dtos.Criterion> clusterCriteria) throws NotFoundException
      For the command identified by id reset the entire list of cluster criteria to match the contents of clusterCriteria.
      Specified by:
      setClusterCriteriaForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to set the cluster criteria for
      clusterCriteria - The priority list of Criterion to set
      Throws:
      NotFoundException - If no command with id exists

    • removeClusterCriterionForCommand

      public void removeClusterCriterionForCommand(String id, @Min(0L) @javax.validation.constraints.Min(0L) int priority) throws NotFoundException
      Remove the Criterion with the given priority from the current list of cluster criteria associated with the command identified by id. A value of 0 for priority will result in the first element in the list being removed, 1 the second element and so on.
      Specified by:
      removeClusterCriterionForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to remove the criterion from
      priority - The priority of the criterion to remove
      Throws:
      NotFoundException - If no command with id exists

    • removeAllClusterCriteriaForCommand

      public void removeAllClusterCriteriaForCommand(String id) throws NotFoundException
      Remove all the Criterion currently associated with the command identified by id.
      Specified by:
      removeAllClusterCriteriaForCommand in interface PersistenceService
      Parameters:
      id - The id of the command to remove the criteria from
      Throws:
      NotFoundException - If no command with id exists

    • findCommandsMatchingCriterion

      public Set<com.netflix.genie.common.internal.dtos.Command> findCommandsMatchingCriterion(@Valid @Valid com.netflix.genie.common.internal.dtos.Criterion criterion, boolean addDefaultStatus)
      Find all the Command's that match the given Criterion.
      Specified by:
      findCommandsMatchingCriterion in interface PersistenceService
      Parameters:
      criterion - The Criterion supplied that each command needs to completely match to be returned
      addDefaultStatus - true if a default status should be added to the supplied criterion if a status isn't already present
      Returns:
      All the Command's which matched the Criterion

    • updateStatusForUnusedCommands

      @Transactional(isolation=READ_COMMITTED) public int updateStatusForUnusedCommands(com.netflix.genie.common.internal.dtos.CommandStatus desiredStatus, Instant commandCreatedThreshold, Set<com.netflix.genie.common.internal.dtos.CommandStatus> currentStatuses, int batchSize)
      Update the status of a command to the desiredStatus if its status is in currentStatuses, it was created before commandCreatedThreshold and it hasn't been used in any job.
      Specified by:
      updateStatusForUnusedCommands in interface PersistenceService
      Parameters:
      desiredStatus - The new status the matching commands should have
      commandCreatedThreshold - The instant in time which a command must have been created before to be considered for update. Exclusive
      currentStatuses - The set of current statuses a command must have to be considered for update
      batchSize - The maximum number of commands to update in a single transaction
      Returns:
      The number of commands whose statuses were updated to desiredStatus

    • deleteUnusedCommands

      @Transactional(isolation=READ_COMMITTED) public long deleteUnusedCommands(Set<com.netflix.genie.common.internal.dtos.CommandStatus> deleteStatuses, Instant commandCreatedThreshold, int batchSize)
      Bulk delete commands from the database where their status is in deleteStatuses they were created before commandCreatedThreshold and they aren't attached to any jobs still in the database.
      Specified by:
      deleteUnusedCommands in interface PersistenceService
      Parameters:
      deleteStatuses - The set of statuses a command must be in in order to be considered for deletion
      commandCreatedThreshold - The instant in time a command must have been created before to be considered for deletion. Exclusive.
      batchSize - The maximum number of commands to delete in a single transaction
      Returns:
      The number of commands that were deleted

    • getJob

      public com.netflix.genie.common.dto.Job getJob(@NotBlank @NotBlank String id) throws com.netflix.genie.common.exceptions.GenieException
      Get job information for given job id.
      Specified by:
      getJob in interface PersistenceService
      Parameters:
      id - id of job to look up
      Returns:
      the job
      Throws:
      com.netflix.genie.common.exceptions.GenieException - if there is an error

    • getJobExecution

      public com.netflix.genie.common.dto.JobExecution getJobExecution(@NotBlank @NotBlank String id) throws com.netflix.genie.common.exceptions.GenieException
      Get job execution for given job id.
      Specified by:
      getJobExecution in interface PersistenceService
      Parameters:
      id - id of job execution to look up
      Returns:
      the job
      Throws:
      com.netflix.genie.common.exceptions.GenieException - if there is an error

    • getJobMetadata

      public com.netflix.genie.common.dto.JobMetadata getJobMetadata(@NotBlank @NotBlank String id) throws com.netflix.genie.common.exceptions.GenieException
      Get the metadata about a job.
      Specified by:
      getJobMetadata in interface PersistenceService
      Parameters:
      id - The id of the job to get metadata for
      Returns:
      The metadata for a job
      Throws:
      com.netflix.genie.common.exceptions.GenieException - If any error occurs

    • findJobs

      @Transactional(readOnly=true) public org.springframework.data.domain.Page<com.netflix.genie.common.dto.search.JobSearchResult> findJobs(@Nullable String id, @Nullable String name, @Nullable String user, @Nullable Set<com.netflix.genie.common.dto.JobStatus> statuses, @Nullable Set<String> tags, @Nullable String clusterName, @Nullable String clusterId, @Nullable String commandName, @Nullable String commandId, @Nullable Instant minStarted, @Nullable Instant maxStarted, @Nullable Instant minFinished, @Nullable Instant maxFinished, @Nullable String grouping, @Nullable String groupingInstance, @NotNull @NotNull org.springframework.data.domain.Pageable page)
      Find jobs which match the given filter criteria.
      Specified by:
      findJobs in interface PersistenceService
      Parameters:
      id - id for job
      name - name of job
      user - user who submitted job
      statuses - statuses of job
      tags - tags for the job
      clusterName - name of cluster for job
      clusterId - id of cluster for job
      commandName - name of the command run in the job
      commandId - id of the command run in the job
      minStarted - The time which the job had to start after in order to be return (inclusive)
      maxStarted - The time which the job had to start before in order to be returned (exclusive)
      minFinished - The time which the job had to finish after in order to be return (inclusive)
      maxFinished - The time which the job had to finish before in order to be returned (exclusive)
      grouping - The job grouping to search for
      groupingInstance - The job grouping instance to search for
      page - Page information of job to get
      Returns:
      Metadata information on jobs which match the criteria

    • deleteJobsCreatedBefore

      @Transactional(isolation=READ_COMMITTED) public long deleteJobsCreatedBefore(@NotNull @NotNull Instant creationThreshold, @NotNull @NotNull Set<com.netflix.genie.common.internal.dtos.JobStatus> excludeStatuses, @Min(1L) @javax.validation.constraints.Min(1L) int batchSize)
      This method will delete a chunk of jobs whose creation time is earlier than the given date.
      Specified by:
      deleteJobsCreatedBefore in interface PersistenceService
      Parameters:
      creationThreshold - The instant in time before which all jobs should be deleted
      excludeStatuses - The set of statuses that should be excluded from deletion if a job is in one of these statuses
      batchSize - The maximum number of jobs that should be deleted per query
      Returns:
      the number of deleted jobs

    • saveJobSubmission

      @Nonnull public String saveJobSubmission(@Valid @Valid JobSubmission jobSubmission) throws IdAlreadyExistsException
      Save the given job submission information in the underlying data store.

      The job record will be created with initial state of JobStatus.RESERVED meaning that the unique id returned by this API has been reserved within Genie and no other job can use it. If JobSubmission contains some attachments these attachments will be persisted to a configured storage system (i.e. local disk, S3, etc) and added to the set of dependencies for the job. The underlying attachment storage system must be accessible by the agent process configured by the system. For example if the server is set up to write attachments to local disk but the agent is not running locally but instead on the remote system it will not be able to access those attachments (as dependencies) and fail. See AttachmentService for more information.

      Specified by:
      saveJobSubmission in interface PersistenceService
      Parameters:
      jobSubmission - All the information the system has gathered regarding the job submission from the user either via the API or via the agent CLI
      Returns:
      The unique id of the job within the Genie ecosystem
      Throws:
      IdAlreadyExistsException - If the id the user requested already exists in the system for another job

    • getJobRequest

      public com.netflix.genie.common.internal.dtos.JobRequest getJobRequest(@NotBlank @NotBlank String id) throws NotFoundException
      Get the original request for a job.
      Specified by:
      getJobRequest in interface PersistenceService
      Parameters:
      id - The unique id of the job to get
      Returns:
      The job request if one was found
      Throws:
      NotFoundException - If no job with id exists

    • saveResolvedJob

      public void saveResolvedJob(@NotBlank @NotBlank String id, @Valid @Valid ResolvedJob resolvedJob) throws NotFoundException
      Save the given resolved details for a job. Sets the job status to JobStatus.RESOLVED.
      Specified by:
      saveResolvedJob in interface PersistenceService
      Parameters:
      id - The id of the job
      resolvedJob - The resolved information for the job
      Throws:
      NotFoundException - When the job identified by id can't be found and the specification can't be saved or one of the resolved cluster, command or applications not longer exist in the system.

    • getJobSpecification

      public Optional<com.netflix.genie.common.internal.dtos.JobSpecification> getJobSpecification(@NotBlank @NotBlank String id) throws NotFoundException
      Get the saved job specification for the given job. If the job hasn't had a job specification resolved an empty Optional will be returned.
      Specified by:
      getJobSpecification in interface PersistenceService
      Parameters:
      id - The id of the job
      Returns:
      The JobSpecification if one is present else an empty Optional
      Throws:
      NotFoundException - If no job with id exists

    • claimJob

      public void claimJob(@NotBlank @NotBlank String id, @Valid @Valid com.netflix.genie.common.internal.dtos.AgentClientMetadata agentClientMetadata) throws NotFoundException, com.netflix.genie.common.internal.exceptions.unchecked.GenieJobAlreadyClaimedException, com.netflix.genie.common.internal.exceptions.unchecked.GenieInvalidStatusException
      Set a job identified by id to be owned by the agent identified by agentClientMetadata. The job status in the system will be set to JobStatus.CLAIMED
      Specified by:
      claimJob in interface PersistenceService
      Parameters:
      id - The id of the job to claim. Must exist in the system.
      agentClientMetadata - The metadata about the client claiming the job
      Throws:
      NotFoundException - if no job with the given id exists
      com.netflix.genie.common.internal.exceptions.unchecked.GenieJobAlreadyClaimedException - if the job with the given id already has been claimed
      com.netflix.genie.common.internal.exceptions.unchecked.GenieInvalidStatusException - if the current job status is not JobStatus.RESOLVED

    • updateJobStatus

      public com.netflix.genie.common.internal.dtos.JobStatus updateJobStatus(@NotBlank @NotBlank String id, @NotNull @NotNull com.netflix.genie.common.internal.dtos.JobStatus currentStatus, @NotNull @NotNull com.netflix.genie.common.internal.dtos.JobStatus newStatus, @Nullable String newStatusMessage) throws NotFoundException
      Update the status of the job identified with id to be newStatus provided that the current status of the job matches newStatus. Optionally a status message can be provided to provide more details to users. If the newStatus is JobStatus.RUNNING the start time will be set. If the newStatus is a member of JobStatus.getFinishedStatuses() and the job had a started time set the finished time of the job will be set. If the currentStatus is different from what the source of truth thinks this function will skip the update and just return the current source of truth value.
      Specified by:
      updateJobStatus in interface PersistenceService
      Parameters:
      id - The id of the job to update status for. Must exist in the system.
      currentStatus - The status the caller to this API thinks the job currently has
      newStatus - The new status the caller would like to update the status to
      newStatusMessage - An optional status message to associate with this change
      Returns:
      The job status in the source of truth
      Throws:
      NotFoundException - if no job with the given id exists

    • updateJobArchiveStatus

      public void updateJobArchiveStatus(@NotBlank(message="No job id entered. Unable to update.") @NotBlank(message="No job id entered. Unable to update.") String id, @NotNull(message="Status cannot be null.") @NotNull(message="Status cannot be null.") com.netflix.genie.common.internal.dtos.ArchiveStatus archiveStatus) throws NotFoundException
      Update the status and status message of the job.
      Specified by:
      updateJobArchiveStatus in interface PersistenceService
      Parameters:
      id - The id of the job to update the status for.
      archiveStatus - The updated archive status for the job.
      Throws:
      NotFoundException - If no job with the given id exists

    • getJobStatus

      public com.netflix.genie.common.internal.dtos.JobStatus getJobStatus(@NotBlank @NotBlank String id) throws NotFoundException
      Get the status for a job with the given id.
      Specified by:
      getJobStatus in interface PersistenceService
      Parameters:
      id - The id of the job to get status for
      Returns:
      The job status
      Throws:
      NotFoundException - If no job with the given id exists

    • getJobArchiveStatus

      public com.netflix.genie.common.internal.dtos.ArchiveStatus getJobArchiveStatus(@NotBlank @NotBlank String id) throws NotFoundException
      Get the archive status for a job with the given id.
      Specified by:
      getJobArchiveStatus in interface PersistenceService
      Parameters:
      id - The id of the job to get status for
      Returns:
      The job archive status
      Throws:
      NotFoundException - If no job with the given id exists

    • getJobArchiveLocation

      public Optional<String> getJobArchiveLocation(@NotBlank @NotBlank String id) throws NotFoundException
      Get the location a job directory was archived to if at all.
      Specified by:
      getJobArchiveLocation in interface PersistenceService
      Parameters:
      id - The id of the job to get the location for
      Returns:
      The archive location or Optional.empty()
      Throws:
      NotFoundException - When there is no job with id id

    • getFinishedJob

      public com.netflix.genie.common.internal.dtos.FinishedJob getFinishedJob(@NotBlank @NotBlank String id) throws NotFoundException, com.netflix.genie.common.internal.exceptions.unchecked.GenieInvalidStatusException
      Get a DTO representing a finished job.
      Specified by:
      getFinishedJob in interface PersistenceService
      Parameters:
      id - The id of the job to retrieve
      Returns:
      the finished job
      Throws:
      NotFoundException - if no job with the given id exists
      com.netflix.genie.common.internal.exceptions.unchecked.GenieInvalidStatusException - if the current status of the job is not final

    • isApiJob

      public boolean isApiJob(@NotBlank @NotBlank String id) throws NotFoundException
      Get whether the job with the given ID was submitted via the REST API or other mechanism.
      Specified by:
      isApiJob in interface PersistenceService
      Parameters:
      id - The id of the job. Not blank.
      Returns:
      true if the job was submitted via the API. false otherwise
      Throws:
      NotFoundException - If no job with id exists

    • getJobCluster

      public com.netflix.genie.common.internal.dtos.Cluster getJobCluster(@NotBlank @NotBlank String id) throws NotFoundException
      Get the cluster the job used or is using.
      Specified by:
      getJobCluster in interface PersistenceService
      Parameters:
      id - The id of the job to get the cluster for
      Returns:
      The Cluster
      Throws:
      NotFoundException - If either the job or the cluster is not found

    • getJobCommand

      public com.netflix.genie.common.internal.dtos.Command getJobCommand(@NotBlank @NotBlank String id) throws NotFoundException
      Get the command the job used or is using.
      Specified by:
      getJobCommand in interface PersistenceService
      Parameters:
      id - The id of the job to get the command for
      Returns:
      The Command
      Throws:
      NotFoundException - If either the job or the command is not found

    • getJobApplications

      public List<com.netflix.genie.common.internal.dtos.Application> getJobApplications(@NotBlank @NotBlank String id) throws NotFoundException
      Get the applications the job used or is currently using.
      Specified by:
      getJobApplications in interface PersistenceService
      Parameters:
      id - The id of the job to get the applications for
      Returns:
      The Applications
      Throws:
      NotFoundException - If either the job or the applications were not found

    • getActiveJobCountForUser

      @Transactional(readOnly=true) public long getActiveJobCountForUser(@NotBlank @NotBlank String user)
      Get the count of 'active' jobs for a given user across all instances.
      Specified by:
      getActiveJobCountForUser in interface PersistenceService
      Parameters:
      user - The user name
      Returns:
      the number of active jobs for a given user

    • getUserResourcesSummaries

      @Transactional(readOnly=true) public Map<String,com.netflix.genie.common.dto.UserResourcesSummary> getUserResourcesSummaries(Set<com.netflix.genie.common.internal.dtos.JobStatus> statuses, boolean api)
      Get a map of summaries of resources usage for each user with at least one active job.
      Specified by:
      getUserResourcesSummaries in interface PersistenceService
      Parameters:
      statuses - The set of JobStatus a job must be in to be considered in this request
      api - Whether the job was submitted via the api (true) or the agent cli (false)
      Returns:
      a map of user resources summaries, keyed on user name

    • getUsedMemoryOnHost

      public long getUsedMemoryOnHost(@NotBlank @NotBlank String hostname)
      Get the amount of memory currently used on the given host by Genie jobs in any of the following states.

      JobStatus.CLAIMED JobStatus.INIT JobStatus.RUNNING

      Specified by:
      getUsedMemoryOnHost in interface PersistenceService
      Parameters:
      hostname - The hostname to get the memory for
      Returns:
      The amount of memory being used in MB by all jobs

    • getActiveJobs

      public Set<String> getActiveJobs()
      Get the set of active jobs.
      Specified by:
      getActiveJobs in interface PersistenceService
      Returns:
      The set of job ids which currently have a status which is considered active

    • getUnclaimedJobs

      public Set<String> getUnclaimedJobs()
      Get the set of jobs in that have not reached CLAIMED state.
      Specified by:
      getUnclaimedJobs in interface PersistenceService
      Returns:
      The set of job ids for jobs which are active but haven't been claimed

    • getHostJobInformation

      public JobInfoAggregate getHostJobInformation(@NotBlank @NotBlank String hostname)
      Get all the aggregate metadata information about jobs running on a given hostname.
      Specified by:
      getHostJobInformation in interface PersistenceService
      Parameters:
      hostname - The hostname the agent is running the job on
      Returns:
      A JobInfoAggregate containing the metadata information

    • getJobsWithStatusAndArchiveStatusUpdatedBefore

      public Set<String> getJobsWithStatusAndArchiveStatusUpdatedBefore(@NotEmpty @NotEmpty Set<com.netflix.genie.common.internal.dtos.JobStatus> statuses, @NotEmpty @NotEmpty Set<com.netflix.genie.common.internal.dtos.ArchiveStatus> archiveStatuses, @NotNull @NotNull Instant updated)
      Get the set of jobs (agent only) whose state is in statuses and archive status is in archiveStatuses and last updated before updated.
      Specified by:
      getJobsWithStatusAndArchiveStatusUpdatedBefore in interface PersistenceService
      Parameters:
      statuses - the set of job statuses
      archiveStatuses - the set of job archive statuses
      updated - the threshold of last update
      Returns:
      a set of job ids

    • updateRequestedLauncherExt

      public void updateRequestedLauncherExt(@NotBlank(message="No job id entered. Unable to update.") @NotBlank(message="No job id entered. Unable to update.") String id, @NotNull(message="Status cannot be null.") @NotNull(message="Status cannot be null.") com.fasterxml.jackson.databind.JsonNode launcherExtension) throws NotFoundException
      Update the requested launcher extension field for this job.
      Specified by:
      updateRequestedLauncherExt in interface PersistenceService
      Parameters:
      id - The id of the job to update the laucher extension for.
      launcherExtension - The updated requested launcher extension JSON blob.
      Throws:
      NotFoundException - If no job with the given id exists

    • getRequestedLauncherExt

      public com.fasterxml.jackson.databind.JsonNode getRequestedLauncherExt(@NotBlank @NotBlank String id) throws NotFoundException
      Get the command the job used or is using.
      Specified by:
      getRequestedLauncherExt in interface PersistenceService
      Parameters:
      id - The id of the job to get the command for
      Returns:
      The JsonNode passed to the launcher at launch, or a NullNode if none was saved
      Throws:
      NotFoundException - If no job with the given id exists

    • updateLauncherExt

      public void updateLauncherExt(@NotBlank(message="No job id entered. Unable to update.") @NotBlank(message="No job id entered. Unable to update.") String id, @NotNull(message="Status cannot be null.") @NotNull(message="Status cannot be null.") com.fasterxml.jackson.databind.JsonNode launcherExtension) throws NotFoundException
      Update the launcher extension field for this job.
      Specified by:
      updateLauncherExt in interface PersistenceService
      Parameters:
      id - The id of the job to update the launcher extension for.
      launcherExtension - The updated launcher extension JSON blob.
      Throws:
      NotFoundException - If no job with the given id exists

    • getLauncherExt

      public com.fasterxml.jackson.databind.JsonNode getLauncherExt(@NotBlank @NotBlank String id) throws NotFoundException
      Get the job requested launcher extension.
      Specified by:
      getLauncherExt in interface PersistenceService
      Parameters:
      id - The id of the job to get the command for
      Returns:
      The JsonNode emitted by the launcher at launch
      Throws:
      NotFoundException - If no job with the given id exists

    • addConfigsToResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void addConfigsToResource(@NotBlank @NotBlank String id, Set<@Size(max=1024) String> configs, Class<R> resourceClass) throws NotFoundException
      Add configuration files to the existing set for a resource.
      Specified by:
      addConfigsToResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the configs should be associated with
      Parameters:
      id - The id of the resource to add the configuration file to. Not null/empty/blank.
      configs - The configuration files to add. Max file length is 1024 characters.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • getConfigsForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> Set<String> getConfigsForResource(@NotBlank @NotBlank String id, Class<R> resourceClass) throws NotFoundException
      Get the set of configuration files associated with the resource with the given id.
      Specified by:
      getConfigsForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the configs are associated with
      Parameters:
      id - The id of the resource to get the configuration files for. Not null/empty/blank.
      resourceClass - The class of the resource
      Returns:
      The set of configuration files as paths
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • updateConfigsForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void updateConfigsForResource(@NotBlank @NotBlank String id, Set<@Size(max=1024) String> configs, Class<R> resourceClass) throws NotFoundException
      Update the set of configuration files associated with the resource with the given id.
      Specified by:
      updateConfigsForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the configs should be associated with
      Parameters:
      id - The id of the resource to update the configuration files for. Not null/empty/blank.
      configs - The configuration files to replace existing configurations with. Not null/empty.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • removeAllConfigsForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void removeAllConfigsForResource(@NotBlank @NotBlank String id, Class<R> resourceClass) throws NotFoundException
      Remove all configuration files from the resource.
      Specified by:
      removeAllConfigsForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the configs should be associated with
      Parameters:
      id - The id of the resource to remove the configuration file from. Not null/empty/blank.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • removeConfigForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void removeConfigForResource(@NotBlank @NotBlank String id, @NotBlank @NotBlank String config, Class<R> resourceClass) throws NotFoundException
      Remove a configuration file from the given resource.
      Specified by:
      removeConfigForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the configs should be associated with
      Parameters:
      id - The id of the resource to remove the configuration file from. Not null/empty/blank.
      config - The configuration file to remove. Not null/empty/blank.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • addDependenciesToResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void addDependenciesToResource(@NotBlank @NotBlank String id, Set<@Size(max=1024) String> dependencies, Class<R> resourceClass) throws NotFoundException
      Add dependency files to the existing set for a resource.
      Specified by:
      addDependenciesToResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the dependencies should be associated with
      Parameters:
      id - The id of the resource to add the dependency file to. Not null/empty/blank.
      dependencies - The dependency files to add. Max file length is 1024 characters.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • getDependenciesForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> Set<String> getDependenciesForResource(@NotBlank @NotBlank String id, Class<R> resourceClass) throws NotFoundException
      Get the set of dependency files associated with the resource with the given id.
      Specified by:
      getDependenciesForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the dependencies are associated with
      Parameters:
      id - The id of the resource to get the dependency files for. Not null/empty/blank.
      resourceClass - The class of the resource
      Returns:
      The set of dependency files as paths
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • updateDependenciesForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void updateDependenciesForResource(@NotBlank @NotBlank String id, Set<@Size(max=1024) String> dependencies, Class<R> resourceClass) throws NotFoundException
      Update the set of dependency files associated with the resource with the given id.
      Specified by:
      updateDependenciesForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the dependencies should be associated with
      Parameters:
      id - The id of the resource to update the dependency files for. Not null/empty/blank.
      dependencies - The dependency files to replace existing dependencys with. Not null/empty.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • removeAllDependenciesForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void removeAllDependenciesForResource(@NotBlank @NotBlank String id, Class<R> resourceClass) throws NotFoundException
      Remove all dependency files from the resource.
      Specified by:
      removeAllDependenciesForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the dependencies should be associated with
      Parameters:
      id - The id of the resource to remove the dependency file from. Not null/empty/blank.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • removeDependencyForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void removeDependencyForResource(@NotBlank @NotBlank String id, @NotBlank @NotBlank String dependency, Class<R> resourceClass) throws NotFoundException
      Remove a dependency file from the given resource.
      Specified by:
      removeDependencyForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the dependencies should be associated with
      Parameters:
      id - The id of the resource to remove the dependency file from. Not null/empty/blank.
      dependency - The dependency file to remove. Not null/empty/blank.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • addTagsToResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void addTagsToResource(@NotBlank @NotBlank String id, Set<@Size(max=255) String> tags, Class<R> resourceClass) throws NotFoundException
      Add tags to the existing set for a resource.
      Specified by:
      addTagsToResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the tags should be associated with
      Parameters:
      id - The id of the resource to add the tags to. Not null/empty/blank.
      tags - The tags to add. Max file length is 255 characters.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • getTagsForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> Set<String> getTagsForResource(@NotBlank @NotBlank String id, Class<R> resourceClass) throws NotFoundException
      Get the set of tags associated with the resource with the given id.
      Specified by:
      getTagsForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the tags are associated with
      Parameters:
      id - The id of the resource to get the tags for. Not null/empty/blank.
      resourceClass - The class of the resource
      Returns:
      The set of tags
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • updateTagsForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void updateTagsForResource(@NotBlank @NotBlank String id, Set<@Size(max=255) String> tags, Class<R> resourceClass) throws NotFoundException
      Update the set of tags associated with the resource with the given id.
      Specified by:
      updateTagsForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the tags should be associated with
      Parameters:
      id - The id of the resource to update the tags for. Not null/empty/blank.
      tags - The tags to replace existing tags with. Not null/empty.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • removeAllTagsForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void removeAllTagsForResource(@NotBlank @NotBlank String id, Class<R> resourceClass) throws NotFoundException
      Remove all tags from the resource.
      Specified by:
      removeAllTagsForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the tags should be associated with
      Parameters:
      id - The id of the resource to remove the tags from. Not null/empty/blank.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • removeTagForResource

      public <R extends com.netflix.genie.common.internal.dtos.CommonResource> void removeTagForResource(@NotBlank @NotBlank String id, @NotBlank @NotBlank String tag, Class<R> resourceClass) throws NotFoundException
      Remove a tag from the given resource.
      Specified by:
      removeTagForResource in interface PersistenceService
      Type Parameters:
      R - The resource type that the tag should be associated with
      Parameters:
      id - The id of the resource to remove the tag from. Not null/empty/blank.
      tag - The tag to remove. Not null/empty/blank.
      resourceClass - The class of the resource
      Throws:
      NotFoundException - If no resource of type PersistenceService with id exists

    • deleteUnusedTags

      @Transactional(isolation=READ_COMMITTED) public long deleteUnusedTags(@NotNull @NotNull Instant createdThreshold, @Min(1L) @javax.validation.constraints.Min(1L) int batchSize)
      Delete all tags from the database that aren't referenced which were created before the supplied created threshold.
      Specified by:
      deleteUnusedTags in interface PersistenceService
      Parameters:
      createdThreshold - The instant in time where tags created before this time that aren't referenced will be deleted. Inclusive
      batchSize - The maximum number of tags to delete in a single transaction
      Returns:
      The number of tags deleted

    • deleteUnusedFiles

      @Transactional(isolation=READ_COMMITTED) public long deleteUnusedFiles(@NotNull @NotNull Instant createdThresholdLowerBound, @NotNull @NotNull Instant createdThresholdUpperBound, @Min(1L) @javax.validation.constraints.Min(1L) int batchSize)
      Delete all files from the database that aren't referenced which were created before the supplied created threshold.
      Specified by:
      deleteUnusedFiles in interface PersistenceService
      Parameters:
      createdThresholdLowerBound - The instant in time when files created after this time that aren't referenced will be selected. Inclusive.
      createdThresholdUpperBound - The instant in time when files created before this time that aren't referenced will be selected. Inclusive.
      batchSize - The maximum number of files to delete in a single transaction
      Returns:
      The number of files deleted