Package datahub.spark.converter

Class SparkStreamingEventToDatahub

java.lang.Object

datahub.spark.converter.SparkStreamingEventToDatahub

public class SparkStreamingEventToDatahub
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DELTA_LAKE_PLATFORM
static final String	FILE_PLATFORM
static final String	KAFKA_PLATFORM

Method Summary

Modifier and Type	Method	Description
static List<datahub.event.MetadataChangeProposalWrapper>	generateMcpFromStreamingProgressEvent(org.apache.spark.sql.streaming.StreamingQueryProgress event, SparkLineageConf conf, Map<String,datahub.event.MetadataChangeProposalWrapper> schemaMap)
static Optional<com.linkedin.common.urn.DatasetUrn>	generateUrnFromStreamingDescription(String description, SparkLineageConf sparkLineageConf)
static String	getDatahubPlatform(String namespace)
static String	getKafkaTopicFromPath(String path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DELTA_LAKE_PLATFORM

public static final String DELTA_LAKE_PLATFORM

See Also:

• Constant Field Values

FILE_PLATFORM

public static final String FILE_PLATFORM

See Also:

• Constant Field Values

KAFKA_PLATFORM

public static final String KAFKA_PLATFORM

See Also:

• Constant Field Values

Method Details

generateMcpFromStreamingProgressEvent

public static List<datahub.event.MetadataChangeProposalWrapper> generateMcpFromStreamingProgressEvent(org.apache.spark.sql.streaming.StreamingQueryProgress event,
 SparkLineageConf conf,
 Map<String,datahub.event.MetadataChangeProposalWrapper> schemaMap)

generateUrnFromStreamingDescription

public static Optional<com.linkedin.common.urn.DatasetUrn> generateUrnFromStreamingDescription(String description,
 SparkLineageConf sparkLineageConf)

getDatahubPlatform

public static String getDatahubPlatform(String namespace)

getKafkaTopicFromPath

public static String getKafkaTopicFromPath(String path)