Package io.openlineage.spark.shade.extension.v1

Interface LineageRelation

public interface LineageRelation
Interface to be implemented by extension classes that extend org.apache.spark.sql.sources.BaseRelation.

Implementing this interface allows for the extraction of lineage information from BaseRelation objects. The methods getNamespace and getName, provided by DatasetIdentifier, must be implemented by the classes that implement this interface. This identifier must follow the naming conventions outlined in the OpenLineage Naming Specification, ensuring consistency across datasets for lineage tracking and data cataloging.

  • Method Summary

    Modifier and Type
    Method
    Description
    io.openlineage.client.utils.DatasetIdentifier
    getLineageDatasetIdentifier(String sparkListenerEventName, io.openlineage.client.OpenLineage openLineage)
    Returns a DatasetIdentifier containing the namespace and name of the dataset for lineage tracking purposes.

  • Method Details

    • getLineageDatasetIdentifier

      io.openlineage.client.utils.DatasetIdentifier getLineageDatasetIdentifier(String sparkListenerEventName, io.openlineage.client.OpenLineage openLineage)
      Returns a DatasetIdentifier containing the namespace and name of the dataset for lineage tracking purposes.
      Parameters:
      sparkListenerEventName - the name of the Spark listener event triggering the lineage extraction
      openLineage - an instance of OpenLineage used for lineage-related operations
      Returns:
      a DatasetIdentifier representing the dataset associated with the event