org.apache.hudi.common.engine

Class HoodieLocalEngineContext

java.lang.Object

org.apache.hudi.common.engine.HoodieEngineContext

org.apache.hudi.common.engine.HoodieLocalEngineContext

public final class HoodieLocalEngineContext
extends HoodieEngineContext

A java based engine context, use this implementation on the query engine integrations if needed.

Field Summary

Fields inherited from class org.apache.hudi.common.engine.HoodieEngineContext

taskContextSupplier

Constructor Summary

Constructors

Constructor and Description
HoodieLocalEngineContext(org.apache.hadoop.conf.Configuration conf)
HoodieLocalEngineContext(org.apache.hadoop.conf.Configuration conf, TaskContextSupplier taskContextSupplier)

Method Summary

Modifier and Type	Method and Description
<T> HoodieData<T>	emptyHoodieData()
<I,O> List<O>	flatMap(List<I> data, SerializableFunction<I,Stream<O>> func, int parallelism)
<I> void	foreach(List<I> data, SerializableConsumer<I> consumer, int parallelism)
Option<String>	getProperty(EngineProperty key)
<I,O> List<O>	map(List<I> data, SerializableFunction<I,O> func, int parallelism)
<I,K,V> Stream<ImmutablePair<K,V>>	mapPartitionsToPairAndReduceByKey(Stream<I> data, SerializablePairFlatMapFunction<Iterator<I>,K,V> flatMapToPairFunc, SerializableBiFunction<V,V,V> reduceFunc, int parallelism)
<I,K,V> Map<K,V>	mapToPair(List<I> data, SerializablePairFunction<I,K,V> func, Integer parallelism)
<I,K,V> List<V>	mapToPairAndReduceByKey(List<I> data, SerializablePairFunction<I,K,V> mapToPairFunc, SerializableBiFunction<V,V,V> reduceFunc, int parallelism)
HoodieAccumulator	newAccumulator()
<T> HoodieData<T>	parallelize(List<T> data, int parallelism)
<I,K,V> List<V>	reduceByKey(List<Pair<K,V>> data, SerializableBiFunction<V,V,V> reduceFunc, int parallelism)
void	setJobStatus(String activeModule, String activityDescription)
void	setProperty(EngineProperty key, String value)

Methods inherited from class org.apache.hudi.common.engine.HoodieEngineContext

getHadoopConf, getTaskContextSupplier, parallelize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieLocalEngineContext

public HoodieLocalEngineContext(org.apache.hadoop.conf.Configuration conf)

HoodieLocalEngineContext

public HoodieLocalEngineContext(org.apache.hadoop.conf.Configuration conf,
                                TaskContextSupplier taskContextSupplier)

Method Detail

newAccumulator

public HoodieAccumulator newAccumulator()

Specified by:

newAccumulator in class HoodieEngineContext

emptyHoodieData

public <T> HoodieData<T> emptyHoodieData()

Specified by:

emptyHoodieData in class HoodieEngineContext

parallelize

public <T> HoodieData<T> parallelize(List<T> data,
                                     int parallelism)

Specified by:

parallelize in class HoodieEngineContext

map

public <I,O> List<O> map(List<I> data,
                         SerializableFunction<I,O> func,
                         int parallelism)

Specified by:

map in class HoodieEngineContext

mapToPairAndReduceByKey

public <I,K,V> List<V> mapToPairAndReduceByKey(List<I> data,
                                               SerializablePairFunction<I,K,V> mapToPairFunc,
                                               SerializableBiFunction<V,V,V> reduceFunc,
                                               int parallelism)

Specified by:

mapToPairAndReduceByKey in class HoodieEngineContext

mapPartitionsToPairAndReduceByKey

public <I,K,V> Stream<ImmutablePair<K,V>> mapPartitionsToPairAndReduceByKey(Stream<I> data,
                                                                            SerializablePairFlatMapFunction<Iterator<I>,K,V> flatMapToPairFunc,
                                                                            SerializableBiFunction<V,V,V> reduceFunc,
                                                                            int parallelism)

Specified by:

mapPartitionsToPairAndReduceByKey in class HoodieEngineContext

reduceByKey

public <I,K,V> List<V> reduceByKey(List<Pair<K,V>> data,
                                   SerializableBiFunction<V,V,V> reduceFunc,
                                   int parallelism)

Specified by:

reduceByKey in class HoodieEngineContext

flatMap

public <I,O> List<O> flatMap(List<I> data,
                             SerializableFunction<I,Stream<O>> func,
                             int parallelism)

Specified by:

flatMap in class HoodieEngineContext

foreach

public <I> void foreach(List<I> data,
                        SerializableConsumer<I> consumer,
                        int parallelism)

Specified by:

foreach in class HoodieEngineContext

mapToPair

public <I,K,V> Map<K,V> mapToPair(List<I> data,
                                  SerializablePairFunction<I,K,V> func,
                                  Integer parallelism)

Specified by:

mapToPair in class HoodieEngineContext

setProperty

public void setProperty(EngineProperty key,
                        String value)

Specified by:

setProperty in class HoodieEngineContext

getProperty

public Option<String> getProperty(EngineProperty key)

Specified by:

getProperty in class HoodieEngineContext

setJobStatus

public void setJobStatus(String activeModule,
                         String activityDescription)

Specified by:

setJobStatus in class HoodieEngineContext