Package org.openmetadata.service.jdbi3

Class EntityRepository<T extends EntityInterface>

java.lang.Object
org.openmetadata.service.jdbi3.EntityRepository<T>
Direct Known Subclasses:
AppMarketPlaceRepository, AppRepository, BotRepository, ChartRepository, ClassificationRepository, ContainerRepository, DashboardDataModelRepository, DashboardRepository, DatabaseRepository, DatabaseSchemaRepository, DataInsightChartRepository, DataProductRepository, DocumentRepository, DomainRepository, EventSubscriptionRepository, GlossaryRepository, GlossaryTermRepository, IngestionPipelineRepository, KpiRepository, MetricsRepository, MlModelRepository, PersonaRepository, PipelineRepository, PolicyRepository, QueryRepository, ReportRepository, RoleRepository, SearchIndexRepository, ServiceEntityRepository, StoredProcedureRepository, TableRepository, TagRepository, TeamRepository, TestCaseRepository, TestConnectionDefinitionRepository, TestDefinitionRepository, TestSuiteRepository, TopicRepository, TypeRepository, UserRepository, WebAnalyticEventRepository, WorkflowRepository
public abstract class EntityRepository<T extends EntityInterface> extends Object
This is the base class used by Entity Resources to perform READ and WRITE operations to the backend database to Create, Retrieve, Update, and Delete entities.

An entity has two types of fields - `attributes` and `relationships`.

  • The `attributes` are the core properties of the entity, example - entity id, name, fullyQualifiedName, columns for a table, etc.
  • The `relationships` are an associated between two entities, example - table belongs to a database, table has a tag, user owns a table, etc. All relationships are captured using EntityReference.
Entities are stored as JSON documents in the database. Each entity is stored in a separate table and is accessed through a Data Access Object or DAO that corresponds to each of the entity. For example, table_entity is the database table used to store JSON docs corresponding to table entity and CollectionDAO.TableDAO is used as the DAO object to access the table_entity table. All DAO objects for an entity are available in daoCollection.

Relationships between entity is stored in a separate table that captures the edge - fromEntity, toEntity, and the relationship name entity_relationship table and are supported by CollectionDAO.EntityRelationshipDAO DAO object.

JSON document of an entity stores only required attributes of an entity. Some attributes such as href are not stored and are created on the fly.

Json document of an entity does not store relationships. As an example, JSON document for table entity does not store the relationship database which is of type EntityReference. This is always retrieved from the relationship table when required to ensure, the data stored is efficiently and consistently, and relationship information does not become stale.

  • Field Details

    • CACHE_WITH_NAME

      public static final com.google.common.cache.LoadingCache<org.apache.commons.lang3.tuple.Pair<String,String>,EntityInterface> CACHE_WITH_NAME

    • CACHE_WITH_ID

      public static final com.google.common.cache.LoadingCache<org.apache.commons.lang3.tuple.Pair<String,UUID>,EntityInterface> CACHE_WITH_ID

    • entityType

      protected final String entityType

    • dao

      protected final EntityDAO<T extends EntityInterface> dao

    • daoCollection

      protected final CollectionDAO daoCollection

    • searchRepository

      protected final SearchRepository searchRepository

    • allowedFields

      protected final Set<String> allowedFields

    • supportsSoftDelete

      public final boolean supportsSoftDelete

    • supportsTags

      protected final boolean supportsTags

    • supportsOwner

      protected final boolean supportsOwner

    • supportsStyle

      protected final boolean supportsStyle

    • supportsLifeCycle

      protected final boolean supportsLifeCycle

    • supportsFollower

      protected final boolean supportsFollower

    • supportsExtension

      protected final boolean supportsExtension

    • supportsVotes

      protected final boolean supportsVotes

    • supportsDomain

      protected final boolean supportsDomain

    • supportsDataProducts

      protected final boolean supportsDataProducts

    • supportsReviewers

      protected final boolean supportsReviewers

    • supportsExperts

      protected final boolean supportsExperts

    • quoteFqn

      protected boolean quoteFqn

    • renameAllowed

      protected boolean renameAllowed

    • putFields

      protected final EntityUtil.Fields putFields
      Fields that can be updated during PUT operation

    • supportsSearch

      protected boolean supportsSearch

  • Constructor Details

  • Method Details

    • setFields

      protected abstract void setFields(T entity, EntityUtil.Fields fields)
      Set the requested fields in an entity. This is used for requesting specific fields in the object during GET operations. It is also used during PUT and PATCH operations to set up fields that can be updated.

    • clearFields

      protected abstract void clearFields(T entity, EntityUtil.Fields fields)
      Set the requested fields in an entity. This is used for requesting specific fields in the object during GET operations. It is also used during PUT and PATCH operations to set up fields that can be updated.

    • prepare

      protected abstract void prepare(T entity, boolean update)
      This method is used for validating an entity to be created during POST, PUT, and PATCH operations and prepare the entity with all the required attributes and relationships.

      The implementation of this method must perform the following:

      1. Prepare the values for attributes that are not required in the request but can be derived on the server side. Example - >FullyQualifiedNames of an entity can be derived from the hierarchy that an entity belongs to .
      2. Validate all the attributes of an entity.
      3. Validate all the relationships of an entity. As an example - during table creation, relationships such as Tags, Owner, Databasea table belongs to are validated. During validation additional information that is not required in the create/update request are set up in the corresponding relationship fields.
      At the end of this operation, entity is expected to be valid and fully constructed with all the fields that will be sent as payload in the POST, PUT, and PATCH operations response.
      See Also:

    • storeEntity

      protected abstract void storeEntity(T entity, boolean update)
      An entity is stored in the backend database as JSON document. The JSON includes some attributes of the entity and does not include attributes such as href. The relationship fields of an entity is never stored in the JSON document. It is always reconstructed based on relationship edges from the backend database.

      As an example, when table entity is stored, the attributes such as href and the relationships such as owner, database, and tags are set to null. These attributes are restored back after the JSON document is stored to be sent as response.
      See Also:

    • storeRelationships

      protected abstract void storeRelationships(T entity)
      This method is called to store all the relationships of an entity. It is expected that all relationships are already validated and completely setup before this method is called and no validation of relationships is required.
      See Also:

    • setInheritedFields

      protected void setInheritedFields(T entity, EntityUtil.Fields fields)
      This method is called to set inherited fields that an entity inherits from its parent.
      See Also:

    • addServiceRelationship

      protected final void addServiceRelationship(T entity, EntityReference service)

    • restorePatchAttributes

      protected void restorePatchAttributes(T original, T updated)
      PATCH operations can't overwrite certain fields, such as entity ID, fullyQualifiedNames etc. Instead of throwing an error, we take lenient approach of ignoring the user error and restore those attributes based on what is already stored in the original entity.

    • setFullyQualifiedName

      public void setFullyQualifiedName(T entity)
      Set fullyQualifiedName of an entity

    • initSeedDataFromResources

      public final void initSeedDataFromResources() throws IOException
      Initialize data from json files if seed data does not exist in corresponding tables. Seed data is stored under openmetadata-service/src/main/resources/json/data/{entityType}

      This method needs to be explicitly called, typically from initialize method. See RoleResource.initialize(OpenMetadataApplicationConfig)

      Throws:
      IOException

    • getEntitiesFromSeedData

      public final List<T> getEntitiesFromSeedData() throws IOException
      Throws:
      IOException

    • getEntitiesFromSeedData

      public final List<T> getEntitiesFromSeedData(String path) throws IOException
      Throws:
      IOException

    • getEntitiesFromSeedData

      public static <U> List<U> getEntitiesFromSeedData(String entityType, String path, Class<U> clazz) throws IOException
      Throws:
      IOException

    • initializeEntity

      public final void initializeEntity(T entity)
      Initialize a given entity if it does not exist.

    • copy

      public final T copy(T entity, CreateEntity request, String updatedBy)

    • getUpdater

      protected EntityRepository<T>.EntityUpdater getUpdater(T original, T updated, EntityRepository.Operation operation)

    • get

      public final T get(javax.ws.rs.core.UriInfo uriInfo, UUID id, EntityUtil.Fields fields)

    • get

      public final T get(javax.ws.rs.core.UriInfo uriInfo, UUID id, EntityUtil.Fields fields, Include include, boolean fromCache)
      Used for getting an entity with a set of requested fields

    • getReference

      public final EntityReference getReference(UUID id, Include include) throws EntityNotFoundException
      getReference is used for getting the entity references from the entity in the cache.
      Throws:
      EntityNotFoundException

    • find

      public final T find(UUID id, Include include) throws EntityNotFoundException
      Find method is used for getting an entity only with core fields stored as JSON without any relational fields set
      Throws:
      EntityNotFoundException

    • getByName

      public T getByName(javax.ws.rs.core.UriInfo uriInfo, String fqn, EntityUtil.Fields fields)

    • getByName

      public final T getByName(javax.ws.rs.core.UriInfo uriInfo, String fqn, EntityUtil.Fields fields, Include include, boolean fromCache)

    • getReferenceByName

      public final EntityReference getReferenceByName(String fqn, Include include)

    • findByNameOrNull

      public final T findByNameOrNull(String fqn, Include include)

    • findByName

      public final T findByName(String fqn, Include include)
      Find method is used for getting an entity only with core fields stored as JSON without any relational fields set

    • listAll

      public final List<T> listAll(EntityUtil.Fields fields, ListFilter filter)

    • listAfter

      public ResultList<T> listAfter(javax.ws.rs.core.UriInfo uriInfo, EntityUtil.Fields fields, ListFilter filter, int limitParam, String after)

    • listAfterWithSkipFailure

      public final ResultList<T> listAfterWithSkipFailure(javax.ws.rs.core.UriInfo uriInfo, EntityUtil.Fields fields, ListFilter filter, int limitParam, String after)

    • listBefore

      public ResultList<T> listBefore(javax.ws.rs.core.UriInfo uriInfo, EntityUtil.Fields fields, ListFilter filter, int limitParam, String before)

    • getVersion

      public final T getVersion(UUID id, String version)

    • listVersions

      public final EntityHistory listVersions(UUID id)

    • create

      public final T create(javax.ws.rs.core.UriInfo uriInfo, T entity)

    • createInternal

      public final T createInternal(T entity)

    • prepareInternal

      public final void prepareInternal(T entity, boolean update)

    • storeRelationshipsInternal

      public final void storeRelationshipsInternal(T entity)

    • setFieldsInternal

      public final T setFieldsInternal(T entity, EntityUtil.Fields fields)

    • clearFieldsInternal

      public final void clearFieldsInternal(T entity, EntityUtil.Fields fields)

    • createOrUpdate

      public final RestUtil.PutResponse<T> createOrUpdate(javax.ws.rs.core.UriInfo uriInfo, T updated)

    • postCreate

      protected void postCreate(T entity)

    • postUpdate

      protected void postUpdate(T original, T updated)

    • update

      public final RestUtil.PutResponse<T> update(javax.ws.rs.core.UriInfo uriInfo, T original, T updated)

    • patch

      public final RestUtil.PatchResponse<T> patch(javax.ws.rs.core.UriInfo uriInfo, UUID id, String user, javax.json.JsonPatch patch)

    • addFollower

      public final RestUtil.PutResponse<T> addFollower(String updatedBy, UUID entityId, UUID userId)

    • updateVote

      public final RestUtil.PutResponse<T> updateVote(String updatedBy, UUID entityId, VoteRequest request)

    • delete

      public final RestUtil.DeleteResponse<T> delete(String updatedBy, UUID id, boolean recursive, boolean hardDelete)

    • deleteByName

      public final RestUtil.DeleteResponse<T> deleteByName(String updatedBy, String name, boolean recursive, boolean hardDelete)

    • preDelete

      protected void preDelete(T entity, String deletedBy)

    • postDelete

      protected void postDelete(T entity)

    • deleteFromSearch

      public final void deleteFromSearch(T entity, EventType changeType)

    • restoreFromSearch

      public final void restoreFromSearch(T entity)

    • deleteInternalByName

      public final RestUtil.DeleteResponse<T> deleteInternalByName(String updatedBy, String name, boolean recursive, boolean hardDelete)

    • deleteInternal

      public final RestUtil.DeleteResponse<T> deleteInternal(String updatedBy, UUID id, boolean recursive, boolean hardDelete)

    • cleanup

      protected void cleanup(T entityInterface)

    • deleteFollower

      public final RestUtil.PutResponse<T> deleteFollower(String updatedBy, UUID entityId, UUID userId)

    • getResultList

      public final ResultList<T> getResultList(List<T> entities, String beforeCursor, String afterCursor, int total)

    • getResultList

      public final ResultList<T> getResultList(List<T> entities, List<EntityError> errors, String beforeCursor, String afterCursor, int total)

    • store

      protected void store(T entity, boolean update)

    • storeTimeSeries

      protected void storeTimeSeries(String fqn, String extension, String jsonSchema, String entityJson)

    • getExtensionAtTimestamp

      public final String getExtensionAtTimestamp(String fqn, String extension, Long timestamp)

    • getLatestExtensionFromTimeSeries

      public final String getLatestExtensionFromTimeSeries(String fqn, String extension)

    • getResultsFromAndToTimestamps

      public final List<String> getResultsFromAndToTimestamps(String fullyQualifiedName, String extension, Long startTs, Long endTs)

    • getResultsFromAndToTimestamps

      public final List<String> getResultsFromAndToTimestamps(String fqn, String extension, Long startTs, Long endTs, EntityTimeSeriesDAO.OrderBy orderBy)

    • deleteExtensionAtTimestamp

      public final void deleteExtensionAtTimestamp(String fqn, String extension, Long timestamp)

    • deleteExtensionBeforeTimestamp

      public final void deleteExtensionBeforeTimestamp(String fqn, String extension, Long timestamp)

    • storeExtension

      public final void storeExtension(EntityInterface entity)

    • removeExtension

      public final void removeExtension(EntityInterface entity)

    • getExtension

      public final Object getExtension(T entity)

    • applyColumnTags

      protected void applyColumnTags(List<Column> columns)

    • applyTags

      protected void applyTags(T entity)

    • applyTags

      public final void applyTags(List<TagLabel> tagLabels, String targetFQN)
      Apply tags tagLabels to the entity or field identified by targetFQN

    • getTags

      protected List<TagLabel> getTags(T entity)

    • getTags

      protected List<TagLabel> getTags(String fqn)

    • getTagsByPrefix

      public final Map<String,List<TagLabel>> getTagsByPrefix(String prefix, String postfix)

    • getFollowers

      protected List<EntityReference> getFollowers(T entity)

    • getVotes

      protected Votes getVotes(T entity)

    • withHref

      public final T withHref(javax.ws.rs.core.UriInfo uriInfo, T entity)

    • getHref

      public final URI getHref(javax.ws.rs.core.UriInfo uriInfo, UUID id)

    • restoreEntity

      public final RestUtil.PutResponse<T> restoreEntity(String updatedBy, String entityType, UUID id)

    • addRelationship

      public final void addRelationship(UUID fromId, UUID toId, String fromEntity, String toEntity, Relationship relationship)

    • addRelationship

      public final void addRelationship(UUID fromId, UUID toId, String fromEntity, String toEntity, Relationship relationship, boolean bidirectional)

    • addRelationship

      public final void addRelationship(UUID fromId, UUID toId, String fromEntity, String toEntity, Relationship relationship, String json, boolean bidirectional)

    • bulkAddToRelationship

      public final void bulkAddToRelationship(UUID fromId, List<UUID> toId, String fromEntity, String toEntity, Relationship relationship)

    • findBoth

      public final List<EntityReference> findBoth(UUID entity1, String entityType1, Relationship relationship, String entity2)

    • findFrom

      public final List<EntityReference> findFrom(UUID toId, String toEntityType, Relationship relationship, String fromEntityType)

    • findFromRecords

      public final List<CollectionDAO.EntityRelationshipRecord> findFromRecords(UUID toId, String toEntityType, Relationship relationship, String fromEntityType)

    • getContainer

      public final EntityReference getContainer(UUID toId)

    • getContainer

      public final EntityReference getContainer(UUID toId, String fromEntityType)

    • getFromEntityRef

      public final EntityReference getFromEntityRef(UUID toId, Relationship relationship, String fromEntityType, boolean mustHaveRelationship)

    • getToEntityRef

      public final EntityReference getToEntityRef(UUID fromId, Relationship relationship, String toEntityType, boolean mustHaveRelationship)

    • ensureSingleRelationship

      public final void ensureSingleRelationship(String entityType, UUID id, List<CollectionDAO.EntityRelationshipRecord> relations, String relationshipName, String toEntityType, boolean mustHaveRelationship)

    • findTo

      public final List<EntityReference> findTo(UUID fromId, String fromEntityType, Relationship relationship, String toEntityType)

    • findToRecords

      public final List<CollectionDAO.EntityRelationshipRecord> findToRecords(UUID fromId, String fromEntityType, Relationship relationship, String toEntityType)

    • deleteRelationship

      public final void deleteRelationship(UUID fromId, String fromEntityType, UUID toId, String toEntityType, Relationship relationship)

    • deleteTo

      public final void deleteTo(UUID toId, String toEntityType, Relationship relationship, String fromEntityType)

    • deleteFrom

      public final void deleteFrom(UUID fromId, String fromEntityType, Relationship relationship, String toEntityType)

    • validateUsers

      public final void validateUsers(List<EntityReference> entityReferences)

    • validateRoles

      public final void validateRoles(List<EntityReference> roles)

    • getOwner

      public final EntityReference getOwner(T entity)

    • getDomain

      public final EntityReference getDomain(T entity)

    • getParentEntity

      public EntityInterface getParentEntity(T entity, String fields)

    • getParent

      public final EntityReference getParent(T entity)

    • getChildren

      protected List<EntityReference> getChildren(T entity)

    • getReviewers

      protected List<EntityReference> getReviewers(T entity)

    • getExperts

      protected List<EntityReference> getExperts(T entity)

    • getOwner

      public final EntityReference getOwner(EntityReference ref)

    • inheritDomain

      public final void inheritDomain(T entity, EntityUtil.Fields fields, EntityInterface parent)

    • inheritOwner

      public final void inheritOwner(T entity, EntityUtil.Fields fields, EntityInterface parent)

    • inheritExperts

      public final void inheritExperts(T entity, EntityUtil.Fields fields, EntityInterface parent)

    • inheritReviewers

      public final void inheritReviewers(T entity, EntityUtil.Fields fields, EntityInterface parent)

    • populateOwner

      protected void populateOwner(EntityReference owner)

    • storeOwner

      protected void storeOwner(T entity, EntityReference owner)

    • storeDomain

      protected void storeDomain(T entity, EntityReference domain)

    • storeDataProducts

      protected void storeDataProducts(T entity, List<EntityReference> dataProducts)

    • bulkAssetsOperation

      protected BulkOperationResult bulkAssetsOperation(UUID entityId, String fromEntity, Relationship relationship, BulkAssets request, boolean isAdd)

    • updateOwner

      public final void updateOwner(T ownedEntity, EntityReference originalOwner, EntityReference newOwner)

    • getFields

      public final EntityUtil.Fields getFields(String fields)

    • getFields

      protected final EntityUtil.Fields getFields(Set<String> fields)

    • getCommonFields

      public final Set<String> getCommonFields(Set<String> input)

    • getAllowedFieldsCopy

      public final Set<String> getAllowedFieldsCopy()

    • getCustomPropertyFQNPrefix

      protected String getCustomPropertyFQNPrefix(String entityType)

    • getCustomPropertyFQN

      protected String getCustomPropertyFQN(String entityType, String propertyName)

    • getIngestionPipelines

      protected List<EntityReference> getIngestionPipelines(T service)

    • checkSystemEntityDeletion

      protected void checkSystemEntityDeletion(T entity)

    • validateOwner

      public final EntityReference validateOwner(EntityReference owner)

    • validateTags

      protected void validateTags(T entity)

    • validateTags

      protected void validateTags(List<TagLabel> labels)

    • validateDomain

      public final EntityReference validateDomain(String domainFqn)

    • validateDomain

      public final void validateDomain(EntityReference domain)

    • validateDataProducts

      public final void validateDataProducts(List<EntityReference> dataProducts)

    • exportToCsv

      public String exportToCsv(String name, String user) throws IOException
      Override this method to support downloading CSV functionality
      Throws:
      IOException

    • importFromCsv

      public CsvImportResult importFromCsv(String name, String csv, boolean dryRun, String user) throws IOException
      Load CSV provided for bulk upload
      Throws:
      IOException

    • getAllTags

      public List<TagLabel> getAllTags(EntityInterface entity)

    • getTaskWorkflow

      public FeedRepository.TaskWorkflow getTaskWorkflow(FeedRepository.ThreadContext threadContext)

    • getSuggestionWorkflow

      public SuggestionRepository.SuggestionWorkflow getSuggestionWorkflow(EntityInterface entity)

    • applySuggestion

      public EntityInterface applySuggestion(EntityInterface entity, String childFQN, Suggestion suggestion)

    • getSuggestionFields

      public String getSuggestionFields(Suggestion suggestion)
      Bring in the necessary fields required to have all the information before applying a suggestion

    • validateTaskThread

      public final void validateTaskThread(FeedRepository.ThreadContext threadContext)

    • validateColumnTags

      protected void validateColumnTags(List<Column> columns)