org.elasticsearch.spark.sql.streaming

Class EsSinkMetadataLog

java.lang.Object

org.apache.spark.sql.execution.streaming.HDFSMetadataLog<T[]>

org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

org.elasticsearch.spark.sql.streaming.EsSinkMetadataLog

All Implemented Interfaces:

org.apache.spark.internal.Logging, org.apache.spark.sql.execution.streaming.MetadataLog<EsSinkStatus[]>

public class EsSinkMetadataLog
extends org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

Provides logic for managing batch ID committing as well as cleaning up state between batch executions for the EsSparkSqlStreamingSink

Constructor Summary

Constructors

Constructor and Description
EsSinkMetadataLog(Settings settings, SparkSession sparkSession, String path)

Method Summary

Modifier and Type	Method and Description
scala.collection.Seq<EsSinkStatus>	compactLogs(scala.collection.Seq<EsSinkStatus> logs)
protected int	defaultCompactInterval()
protected long	fileCleanupDelayMs()
protected boolean	isDeletingExpiredLog()
static int	VERSION_NUMBER()

Methods inherited from class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog

add, allFiles, batchIdToPath, COMPACT_FILE_SUFFIX, COMPACT_LATENCY_WARN_THRESHOLD_MS, compactInterval, deriveCompactInterval, deserialize, getAllValidBatches, getBatchIdFromFileName, getValidBatchesBeforeCompactionBatch, isBatchFile, isCompactionBatch, minBatchesToRetain, nextCompactionBatchId, pathToBatchId, purge, serialize

Methods inherited from class org.apache.spark.sql.execution.streaming.HDFSMetadataLog

batchFilesFilter, fileManager, get, get, getLatest, getOrderedBatchFiles, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, metadataPath, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, purgeAfter, validateVersion, verifyBatchIds

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initLock, uninitialize

Constructor Detail

EsSinkMetadataLog

public EsSinkMetadataLog(Settings settings,
                         SparkSession sparkSession,
                         String path)

Method Detail

VERSION_NUMBER

public static int VERSION_NUMBER()

fileCleanupDelayMs

protected long fileCleanupDelayMs()

Specified by:

fileCleanupDelayMs in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

isDeletingExpiredLog

protected boolean isDeletingExpiredLog()

Specified by:

isDeletingExpiredLog in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

defaultCompactInterval

protected int defaultCompactInterval()

Specified by:

defaultCompactInterval in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

compactLogs

public scala.collection.Seq<EsSinkStatus> compactLogs(scala.collection.Seq<EsSinkStatus> logs)

Specified by:

compactLogs in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>