Index

A C D E F G H I K L M N O P R S T V
All Classes and Interfaces|All Packages|Constant Field Values

A

apply(LogicalPlan) - Method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftRelationVisitor

apply(SaveIntoDataSourceCommand) - Method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.plan.RedshiftSaveIntoDataSourceCommandBuilder

C

COALESCE_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

CONFIG_LOG_MCPS - Static variable in class datahub.spark.conf.SparkConfigParser

Constants - Class in io.openlineage.spark.agent.vendor.redshift

Constants() - Constructor for class io.openlineage.spark.agent.vendor.redshift.Constants

convertOpenLineageRunEventToDatahubJob(OpenLineage.RunEvent) - Method in class datahub.spark.DatahubEventEmitter

createOutputDatasetBuilder(OpenLineageContext) - Method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.plan.RedshiftEventHandlerFactory

D

DATABRICKS_CLUSTER_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

DATAHUB_FLOW_NAME - Static variable in class datahub.spark.conf.SparkConfigParser

datahub.spark - package datahub.spark

datahub.spark.conf - package datahub.spark.conf

datahub.spark.converter - package datahub.spark.converter

DatahubEmitterConfig - Interface in datahub.spark.conf

DatahubEventEmitter - Class in datahub.spark

DatahubEventEmitter(SparkOpenLineageConfig, String) - Constructor for class datahub.spark.DatahubEventEmitter

DatahubSparkListener - Class in datahub.spark

DatahubSparkListener() - Constructor for class datahub.spark.DatahubSparkListener

DATASET_ENV_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

DATASET_HIVE_PLATFORM_ALIAS - Static variable in class datahub.spark.conf.SparkConfigParser

DATASET_INCLUDE_SCHEMA_METADATA - Static variable in class datahub.spark.conf.SparkConfigParser

DATASET_INCLUDE_SCHEMA_METADATA_DEPRECATED_ALIAS - Static variable in class datahub.spark.conf.SparkConfigParser

DATASET_LOWERCASE_URNS - Static variable in class datahub.spark.conf.SparkConfigParser

DATASET_MATERIALIZE_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

DATASET_PLATFORM_INSTANCE_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

datasourceFacet(OpenLineage, String) - Static method in class io.openlineage.spark.agent.util.PlanUtils

Construct a OpenLineage.DatasourceDatasetFacet given a namespace for the datasource.

DEFAULT_SCHEMA - Static variable in class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftDataset

DELTA_LAKE_PLATFORM - Static variable in class datahub.spark.converter.SparkStreamingEventToDatahub

DISABLE_SSL_VERIFICATION_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

DISABLE_SYMLINK_RESOLUTION - Static variable in class datahub.spark.conf.SparkConfigParser

DOMAINS_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

E

emit(OpenLineage.RunEvent) - Method in class datahub.spark.DatahubEventEmitter

emit(StreamingQueryProgress) - Method in class datahub.spark.DatahubEventEmitter

emitCoalesced() - Method in class datahub.spark.DatahubEventEmitter

emitMcps(List<MetadataChangeProposal>) - Method in class datahub.spark.DatahubEventEmitter

EMITTER_TYPE - Static variable in class datahub.spark.conf.SparkConfigParser

empty() - Static method in interface io.openlineage.spark.api.Vendors

F

FABRIC_TYPE_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

FILE_EMITTER_FILE_NAME - Static variable in class datahub.spark.conf.SparkConfigParser

FILE_PARTITION_REGEXP_PATTERN - Static variable in class datahub.spark.conf.SparkConfigParser

FILE_PLATFORM - Static variable in class datahub.spark.converter.SparkStreamingEventToDatahub

FileDatahubEmitterConfig - Class in datahub.spark.conf

FileDatahubEmitterConfig(FileEmitterConfig) - Constructor for class datahub.spark.conf.FileDatahubEmitterConfig

findRDDPaths(List<RDD<?>>) - Static method in class io.openlineage.spark.agent.util.PlanUtils

Given a list of RDDs, it collects list of data location directories.

findRDDPaths(RDD) - Static method in class io.openlineage.spark.agent.util.RddPathUtils

G

generateCoalescedMcps() - Method in class datahub.spark.DatahubEventEmitter

generateMcpFromStreamingProgressEvent(StreamingQueryProgress, SparkLineageConf, Map<String, MetadataChangeProposalWrapper>) - Static method in class datahub.spark.converter.SparkStreamingEventToDatahub

generateUrnFromStreamingDescription(String, SparkLineageConf) - Static method in class datahub.spark.converter.SparkStreamingEventToDatahub

getCommonFabricType(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getCommonPlatformInstance(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getDatabricksClusterTags(String) - Static method in class datahub.spark.conf.SparkConfigParser

getDatabricksTags(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getDatahubPlatform(String) - Static method in class datahub.spark.converter.SparkStreamingEventToDatahub

getDatasets(DatasetFactory<D>, String, Optional<String>, Optional<String>, StructType) - Static method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftDataset

getDirectoryPath(Path, Configuration) - Static method in class io.openlineage.spark.agent.util.PlanUtils

getDirectoryPathOl(Path, Configuration) - Static method in class io.openlineage.spark.agent.util.PlanUtils

getDomains(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getEventHandlerFactories() - Method in interface io.openlineage.spark.api.Vendors

getEventHandlerFactories() - Method in class io.openlineage.spark.api.VendorsImpl

getEventHandlerFactory() - Method in class io.openlineage.spark.agent.vendor.redshift.RedshiftVendor

getFilePartitionRegexpPattern(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getHivePlatformAlias(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getInputVisitors(OpenLineageContext) - Method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftVisitorFactory

getKafkaTopicFromPath(String) - Static method in class datahub.spark.converter.SparkStreamingEventToDatahub

getOutputVisitors(OpenLineageContext) - Method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftVisitorFactory

getParentJobKey(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getPathSpecListMap(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getPipelineName(Config, SparkAppContext) - Static method in class datahub.spark.conf.SparkConfigParser

getPlatformInstance(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getRemovePartitionPattern(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getSparkAppName(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getSparkMaster(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getStreamingHeartbeatSec(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getTags(Config) - Static method in class datahub.spark.conf.SparkConfigParser

getType() - Method in interface datahub.spark.conf.DatahubEmitterConfig

getVendors() - Static method in interface io.openlineage.spark.api.Vendors

getVendors(List<String>) - Static method in interface io.openlineage.spark.api.Vendors

getVisitorFactories() - Method in interface io.openlineage.spark.api.Vendors

getVisitorFactories() - Method in class io.openlineage.spark.api.VendorsImpl

getVisitorFactory() - Method in class io.openlineage.spark.agent.vendor.redshift.RedshiftVendor

GMS_AUTH_TOKEN - Static variable in class datahub.spark.conf.SparkConfigParser

GMS_URL_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

H

hasRedshiftClasses() - Static method in class io.openlineage.spark.agent.vendor.redshift.RedshiftVendor

I

initializeEmitter(Config) - Method in class datahub.spark.DatahubSparkListener

io.openlineage.spark.agent.util - package io.openlineage.spark.agent.util

io.openlineage.spark.agent.vendor.redshift - package io.openlineage.spark.agent.vendor.redshift

io.openlineage.spark.agent.vendor.redshift.lifecycle - package io.openlineage.spark.agent.vendor.redshift.lifecycle

io.openlineage.spark.agent.vendor.redshift.lifecycle.plan - package io.openlineage.spark.agent.vendor.redshift.lifecycle.plan

io.openlineage.spark.api - package io.openlineage.spark.api

isCoalesceEnabled(Config) - Static method in class datahub.spark.conf.SparkConfigParser

isDatasetMaterialize(Config) - Static method in class datahub.spark.conf.SparkConfigParser

isDefinedAt(LogicalPlan) - Method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftRelationVisitor

isDefinedAt(SparkListenerEvent) - Method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.plan.RedshiftSaveIntoDataSourceCommandBuilder

isDisableSymlinkResolution(Config) - Static method in class datahub.spark.conf.SparkConfigParser

isEmitCoalescePeriodically(Config) - Static method in class datahub.spark.conf.SparkConfigParser

isIncludeSchemaMetadata(Config) - Static method in class datahub.spark.conf.SparkConfigParser

isLegacyLineageCleanupEnabled(Config) - Static method in class datahub.spark.conf.SparkConfigParser

isLogMcps(Config) - Static method in class datahub.spark.conf.SparkConfigParser

isLowerCaseDatasetUrns(Config) - Static method in class datahub.spark.conf.SparkConfigParser

isPatchEnabled(Config) - Static method in class datahub.spark.conf.SparkConfigParser

isRedshiftClass(LogicalPlan) - Method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftRelationVisitor

isRedshiftSource(CreatableRelationProvider) - Static method in class io.openlineage.spark.agent.vendor.redshift.lifecycle.plan.RedshiftSaveIntoDataSourceCommandBuilder

isStreaming() - Method in class datahub.spark.DatahubEventEmitter

isVendorAvailable() - Method in class io.openlineage.spark.agent.vendor.redshift.RedshiftVendor

K

KAFKA_EMITTER_BOOTSTRAP - Static variable in class datahub.spark.conf.SparkConfigParser

KAFKA_EMITTER_PRODUCER_CONFIG - Static variable in class datahub.spark.conf.SparkConfigParser

KAFKA_EMITTER_SCHEMA_REGISTRY_CONFIG - Static variable in class datahub.spark.conf.SparkConfigParser

KAFKA_EMITTER_SCHEMA_REGISTRY_URL - Static variable in class datahub.spark.conf.SparkConfigParser

KAFKA_MCP_TOPIC - Static variable in class datahub.spark.conf.SparkConfigParser

KAFKA_PLATFORM - Static variable in class datahub.spark.converter.SparkStreamingEventToDatahub

KafkaDatahubEmitterConfig - Class in datahub.spark.conf

KafkaDatahubEmitterConfig(KafkaEmitterConfig) - Constructor for class datahub.spark.conf.KafkaDatahubEmitterConfig

KafkaDatahubEmitterConfig(KafkaEmitterConfig, String) - Constructor for class datahub.spark.conf.KafkaDatahubEmitterConfig

L

LEGACY_LINEAGE_CLEANUP - Static variable in class datahub.spark.conf.SparkConfigParser

M

MAX_RETRIES - Static variable in class datahub.spark.conf.SparkConfigParser

merge(Collection<? extends PartialFunction<T, ? extends Collection<D>>>) - Static method in class io.openlineage.spark.agent.util.PlanUtils

Given a list of PartialFunctions merge to produce a single function that will test the input against each function one by one until a match is found or PartialFunction$.empty() is returned.

moveKeysToRoot(Properties, String) - Static method in class datahub.spark.conf.SparkConfigParser

N

namespaceUri(URI) - Static method in class io.openlineage.spark.agent.util.PlanUtils

O

onApplicationEnd(SparkListenerApplicationEnd) - Method in class datahub.spark.DatahubSparkListener

onApplicationStart(SparkListenerApplicationStart) - Method in class datahub.spark.DatahubSparkListener

onJobEnd(SparkListenerJobEnd) - Method in class datahub.spark.DatahubSparkListener

onJobStart(SparkListenerJobStart) - Method in class datahub.spark.DatahubSparkListener

onOtherEvent(SparkListenerEvent) - Method in class datahub.spark.DatahubSparkListener

onTaskEnd(SparkListenerTaskEnd) - Method in class datahub.spark.DatahubSparkListener

P

PARENT_JOB_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

parentRunFacet(UUID, String, String) - Static method in class io.openlineage.spark.agent.util.PlanUtils

Construct a OpenLineage.ParentRunFacet given the parent job's parentRunId, job name, and namespace.

parsePropertiesToConfig(Properties) - Static method in class datahub.spark.conf.SparkConfigParser

parseSparkConfig() - Static method in class datahub.spark.conf.SparkConfigParser

PATCH_ENABLED - Static variable in class datahub.spark.conf.SparkConfigParser

PATH_SPEC_LIST_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

PIPELINE_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

PIPELINE_PLATFORM_INSTANCE_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

PlanUtils - Class in io.openlineage.spark.agent.util

Utility functions for traversing a LogicalPlan.

PlanUtils() - Constructor for class io.openlineage.spark.agent.util.PlanUtils

PLATFORM_INSTANCE_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

PLATFORM_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

R

RddPathUtils - Class in io.openlineage.spark.agent.util

Utility class to extract paths from RDD nodes.

RddPathUtils() - Constructor for class io.openlineage.spark.agent.util.RddPathUtils

REDSHIFT_CLASS_NAME - Static variable in class io.openlineage.spark.agent.vendor.redshift.Constants

REDSHIFT_PREFIX - Static variable in class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftDataset

REDSHIFT_PROVIDER_CLASS_NAME - Static variable in class io.openlineage.spark.agent.vendor.redshift.Constants

RedshiftDataset - Class in io.openlineage.spark.agent.vendor.redshift.lifecycle

RedshiftDataset() - Constructor for class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftDataset

RedshiftEventHandlerFactory - Class in io.openlineage.spark.agent.vendor.redshift.lifecycle.plan

RedshiftEventHandlerFactory() - Constructor for class io.openlineage.spark.agent.vendor.redshift.lifecycle.plan.RedshiftEventHandlerFactory

RedshiftRelationVisitor<D extends io.openlineage.client.OpenLineage.Dataset> - Class in io.openlineage.spark.agent.vendor.redshift.lifecycle

LogicalPlan visitor that matches SaveIntoDataSourceCommands that use a RedshiftRelation.

RedshiftRelationVisitor(OpenLineageContext, DatasetFactory<D>) - Constructor for class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftRelationVisitor

RedshiftSaveIntoDataSourceCommandBuilder - Class in io.openlineage.spark.agent.vendor.redshift.lifecycle.plan

RedshiftSaveIntoDataSourceCommandBuilder(OpenLineageContext) - Constructor for class io.openlineage.spark.agent.vendor.redshift.lifecycle.plan.RedshiftSaveIntoDataSourceCommandBuilder

RedshiftVendor - Class in io.openlineage.spark.agent.vendor.redshift

RedshiftVendor() - Constructor for class io.openlineage.spark.agent.vendor.redshift.RedshiftVendor

RedshiftVisitorFactory - Class in io.openlineage.spark.agent.vendor.redshift.lifecycle

RedshiftVisitorFactory() - Constructor for class io.openlineage.spark.agent.vendor.redshift.lifecycle.RedshiftVisitorFactory

REMOVE_PARTITION_PATTERN - Static variable in class datahub.spark.conf.SparkConfigParser

REMOVE_PATTERN_GROUP - Static variable in class io.openlineage.spark.agent.util.RemovePathPatternUtils

removeInputsPathPattern(OpenLineageContext, List<OpenLineage.InputDataset>) - Static method in class io.openlineage.spark.agent.util.RemovePathPatternUtils

removeOutputsPathPattern(OpenLineageContext, List<OpenLineage.OutputDataset>) - Static method in class io.openlineage.spark.agent.util.RemovePathPatternUtils

removeOutputsPathPattern_ol(OpenLineageContext, List<OpenLineage.OutputDataset>) - Static method in class io.openlineage.spark.agent.util.RemovePathPatternUtils

RemovePathPatternUtils - Class in io.openlineage.spark.agent.util

Utility class to handle removing path patterns in dataset names.

RemovePathPatternUtils() - Constructor for class io.openlineage.spark.agent.util.RemovePathPatternUtils

REST_DISABLE_CHUNKED_ENCODING - Static variable in class datahub.spark.conf.SparkConfigParser

RestDatahubEmitterConfig - Class in datahub.spark.conf

RestDatahubEmitterConfig(RestEmitterConfig) - Constructor for class datahub.spark.conf.RestDatahubEmitterConfig

RETRY_INTERVAL_IN_SEC - Static variable in class datahub.spark.conf.SparkConfigParser

S

S3_EMITTER_ACCESS_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

S3_EMITTER_BUCKET - Static variable in class datahub.spark.conf.SparkConfigParser

S3_EMITTER_ENDPOINT - Static variable in class datahub.spark.conf.SparkConfigParser

S3_EMITTER_FILE_NAME - Static variable in class datahub.spark.conf.SparkConfigParser

S3_EMITTER_PREFIX - Static variable in class datahub.spark.conf.SparkConfigParser

S3_EMITTER_PROFILE - Static variable in class datahub.spark.conf.SparkConfigParser

S3_EMITTER_REGION - Static variable in class datahub.spark.conf.SparkConfigParser

S3_EMITTER_SECRET_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

S3DatahubEmitterConfig - Class in datahub.spark.conf

S3DatahubEmitterConfig(S3EmitterConfig) - Constructor for class datahub.spark.conf.S3DatahubEmitterConfig

safeApply(PartialFunction<D, List<T>>, D) - Static method in class io.openlineage.spark.agent.util.PlanUtils

apply method implementation that should never throw an error or exception

safeIsDefinedAt(PartialFunction, Object) - Static method in class io.openlineage.spark.agent.util.PlanUtils

isDefinedAt method implementation that should never throw an error or exception

safeIsInstanceOf(Object, String) - Static method in class io.openlineage.spark.agent.util.PlanUtils

instanceOf alike implementation which does not fail in case of a missing class.

schemaFacet(OpenLineage, StructType) - Static method in class io.openlineage.spark.agent.util.PlanUtils

Given a schema, construct a valid OpenLineage.SchemaDatasetFacet.

setConfig(SparkLineageConf) - Method in class datahub.spark.DatahubEventEmitter

setStreaming(boolean) - Method in class datahub.spark.DatahubEventEmitter

SPARK_APP_NAME - Static variable in class datahub.spark.conf.SparkConfigParser

SPARK_DATABRICKS_CLUSTER_USAGE_TAGS_CLUSTER_ALL_TAGS - Static variable in class datahub.spark.conf.SparkConfigParser

SPARK_MASTER - Static variable in class datahub.spark.conf.SparkConfigParser

SPARK_OPENLINEAGE_DATASET_REMOVE_PATH_PATTERN - Static variable in class io.openlineage.spark.agent.util.RemovePathPatternUtils

SPARK_PLATFORM_INSTANCE_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

SparkAppContext - Class in datahub.spark.conf

SparkAppContext() - Constructor for class datahub.spark.conf.SparkAppContext

SparkConfigParser - Class in datahub.spark.conf

sparkConfigToDatahubOpenlineageConf(Config, SparkAppContext) - Static method in class datahub.spark.conf.SparkConfigParser

SparkLineageConf - Class in datahub.spark.conf

SparkLineageConf() - Constructor for class datahub.spark.conf.SparkLineageConf

SparkStreamingEventToDatahub - Class in datahub.spark.converter

STAGE_METADATA_COALESCING - Static variable in class datahub.spark.conf.SparkConfigParser

STREAMING_HEARTBEAT - Static variable in class datahub.spark.conf.SparkConfigParser

STREAMING_JOB - Static variable in class datahub.spark.conf.SparkConfigParser

T

TAGS_KEY - Static variable in class datahub.spark.conf.SparkConfigParser

toSparkLineageConf(Config, SparkAppContext, DatahubEmitterConfig) - Static method in class datahub.spark.conf.SparkLineageConf

toStructType(List<Attribute>) - Static method in class io.openlineage.spark.agent.util.PlanUtils

Given a list of attributes, constructs a valid OpenLineage.SchemaDatasetFacet.

V

Vendors - Interface in io.openlineage.spark.api

VENDORS - Static variable in interface io.openlineage.spark.api.Vendors

VendorsImpl - Class in io.openlineage.spark.api

VendorsImpl(List<Vendor>) - Constructor for class io.openlineage.spark.api.VendorsImpl

A C D E F G H I K L M N O P R S T V
All Classes and Interfaces|All Packages|Constant Field Values