org.apache.hudi.utilities.deltastreamer

Class SchedulerConfGenerator

java.lang.Object

org.apache.hudi.utilities.deltastreamer.SchedulerConfGenerator

public class SchedulerConfGenerator
extends Object

Utility Class to generate Spark Scheduling allocation file. This kicks in only when user sets spark.scheduler.mode=FAIR at spark-submit time

Field Summary

Fields

Modifier and Type	Field and Description
static String	COMPACT_POOL_NAME
static String	DELTASYNC_POOL_NAME
protected static org.apache.log4j.Logger	log
static String	SPARK_SCHEDULER_ALLOCATION_FILE_KEY
static String	SPARK_SCHEDULER_MODE_KEY

Constructor Summary

Constructors

Constructor and Description
SchedulerConfGenerator()

Method Summary

Modifier and Type	Method and Description
static Map<String,String>	getSparkSchedulingConfigs(HoodieDeltaStreamer.Config cfg) Helper to set Spark Scheduling Configs dynamically

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static volatile org.apache.log4j.Logger log

DELTASYNC_POOL_NAME

public static final String DELTASYNC_POOL_NAME

See Also:

Constant Field Values

COMPACT_POOL_NAME

public static final String COMPACT_POOL_NAME

See Also:

Constant Field Values

SPARK_SCHEDULER_MODE_KEY

public static final String SPARK_SCHEDULER_MODE_KEY

See Also:

Constant Field Values

SPARK_SCHEDULER_ALLOCATION_FILE_KEY

public static final String SPARK_SCHEDULER_ALLOCATION_FILE_KEY

See Also:

Constant Field Values

Constructor Detail

SchedulerConfGenerator

public SchedulerConfGenerator()

Method Detail

getSparkSchedulingConfigs

public static Map<String,String> getSparkSchedulingConfigs(HoodieDeltaStreamer.Config cfg)
                                                    throws Exception

Helper to set Spark Scheduling Configs dynamically

Parameters:

cfg - Config

Throws:

Exception