org.apache.tez.runtime.library.cartesianproduct

Class CartesianProductVertexManager

java.lang.Object

org.apache.tez.dag.api.VertexManagerPlugin

org.apache.tez.runtime.library.cartesianproduct.CartesianProductVertexManager

public class CartesianProductVertexManager
extends org.apache.tez.dag.api.VertexManagerPlugin

This VM wrap a real vertex manager implementation object. It choose whether it's partitioned or fair implementation according to the config. All method invocations are actually redirected to real implementation. Predefined parallelism isn't allowed for cartesian product vertex. Parallellism has to be determined by vertex manager. If a vertex use this vertex, its input edges must be either cartesian product edge or broadcast edge. Sources can be either vertices or vertex groups (only in fair cartesian product). Slow start only works in partitioned case.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TEZ_CARTESIAN_PRODUCT_ENABLE_GROUPING Whether to disable grouping in fair cartesian product If this is set to true, it's best to set "tez.cartesian-product.num-partitions" to 1 to avoid unnecessary overhead caused by multiple partitions.
static boolean	TEZ_CARTESIAN_PRODUCT_ENABLE_GROUPING_DEFAULT
static String	TEZ_CARTESIAN_PRODUCT_GROUPING_FRACTION If every source vertex has this percents of tasks completed and generate some output, we can begin auto grouping.
static String	TEZ_CARTESIAN_PRODUCT_MAX_PARALLELISM Max parallelism, for fair cartesian product only.
static int	TEZ_CARTESIAN_PRODUCT_MAX_PARALLELISM_DEFAULT
static String	TEZ_CARTESIAN_PRODUCT_MIN_OPS_PER_WORKER Min cartesian product operations per worker, for fair cartesian product only.
static long	TEZ_CARTESIAN_PRODUCT_MIN_OPS_PER_WORKER_DEFAULT
static String	TEZ_CARTESIAN_PRODUCT_NUM_PARTITIONS Num partitions as int value, for fair cartesian product only.
static String	TEZ_CARTESIAN_PRODUCT_SLOW_START_MAX_FRACTION Schedule all tasks when the fraction of finished cartesian product source tasks reach this value
static float	TEZ_CARTESIAN_PRODUCT_SLOW_START_MAX_FRACTION_DEFAULT
static String	TEZ_CARTESIAN_PRODUCT_SLOW_START_MIN_FRACTION Begin scheduling task when the fraction of finished cartesian product source tasks reaches this value
static float	TEZ_CARTESIAN_PRODUCT_SLOW_START_MIN_FRACTION_DEFAULT

Constructor Summary

Constructors

Constructor and Description
CartesianProductVertexManager(org.apache.tez.dag.api.VertexManagerPluginContext context)

Method Summary

Modifier and Type	Method and Description
protected org.apache.tez.runtime.library.cartesianproduct.CartesianProductVertexManagerReal	getVertexManagerReal()
void	initialize()
void	onRootVertexInitialized(String inputName, org.apache.tez.dag.api.InputDescriptor inputDescriptor, List<org.apache.tez.runtime.api.Event> events) Currently direct input to cartesian product vertex is not supported
void	onSourceTaskCompleted(org.apache.tez.runtime.api.TaskAttemptIdentifier attempt)
void	onVertexManagerEventReceived(org.apache.tez.runtime.api.events.VertexManagerEvent vmEvent) no op currently, will be used for locality based optimization in future
void	onVertexStarted(List<org.apache.tez.runtime.api.TaskAttemptIdentifier> completions)
void	onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate)

Methods inherited from class org.apache.tez.dag.api.VertexManagerPlugin

getContext, onSourceTaskCompleted, onVertexStarted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_CARTESIAN_PRODUCT_SLOW_START_MIN_FRACTION

public static final String TEZ_CARTESIAN_PRODUCT_SLOW_START_MIN_FRACTION

Begin scheduling task when the fraction of finished cartesian product source tasks reaches this value

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_SLOW_START_MIN_FRACTION_DEFAULT

public static final float TEZ_CARTESIAN_PRODUCT_SLOW_START_MIN_FRACTION_DEFAULT

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_SLOW_START_MAX_FRACTION

public static final String TEZ_CARTESIAN_PRODUCT_SLOW_START_MAX_FRACTION

Schedule all tasks when the fraction of finished cartesian product source tasks reach this value

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_SLOW_START_MAX_FRACTION_DEFAULT

public static final float TEZ_CARTESIAN_PRODUCT_SLOW_START_MAX_FRACTION_DEFAULT

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_NUM_PARTITIONS

public static final String TEZ_CARTESIAN_PRODUCT_NUM_PARTITIONS

Num partitions as int value, for fair cartesian product only. Set this if auto determined num partition is not large enough

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_ENABLE_GROUPING

public static final String TEZ_CARTESIAN_PRODUCT_ENABLE_GROUPING

Whether to disable grouping in fair cartesian product If this is set to true, it's best to set "tez.cartesian-product.num-partitions" to 1 to avoid unnecessary overhead caused by multiple partitions.

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_ENABLE_GROUPING_DEFAULT

public static final boolean TEZ_CARTESIAN_PRODUCT_ENABLE_GROUPING_DEFAULT

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_GROUPING_FRACTION

public static final String TEZ_CARTESIAN_PRODUCT_GROUPING_FRACTION

If every source vertex has this percents of tasks completed and generate some output, we can begin auto grouping. Positive float value, max 1. If not set, auto grouping will begin once every source vertex generate enough output

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_MAX_PARALLELISM

public static final String TEZ_CARTESIAN_PRODUCT_MAX_PARALLELISM

Max parallelism, for fair cartesian product only. This is used to avoid get too many tasks. The value must be positive.

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_MAX_PARALLELISM_DEFAULT

public static final int TEZ_CARTESIAN_PRODUCT_MAX_PARALLELISM_DEFAULT

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_MIN_OPS_PER_WORKER

public static final String TEZ_CARTESIAN_PRODUCT_MIN_OPS_PER_WORKER

Min cartesian product operations per worker, for fair cartesian product only. This is used to avoid a task gets too small workload. The value must be positive.

See Also:

Constant Field Values

TEZ_CARTESIAN_PRODUCT_MIN_OPS_PER_WORKER_DEFAULT

public static final long TEZ_CARTESIAN_PRODUCT_MIN_OPS_PER_WORKER_DEFAULT

See Also:

Constant Field Values

Constructor Detail

CartesianProductVertexManager

public CartesianProductVertexManager(org.apache.tez.dag.api.VertexManagerPluginContext context)

Method Detail

initialize

public void initialize()
                throws Exception

Specified by:

initialize in class org.apache.tez.dag.api.VertexManagerPlugin

Throws:

Exception

getVertexManagerReal

protected org.apache.tez.runtime.library.cartesianproduct.CartesianProductVertexManagerReal getVertexManagerReal()

onVertexManagerEventReceived

public void onVertexManagerEventReceived(org.apache.tez.runtime.api.events.VertexManagerEvent vmEvent)
                                  throws Exception

no op currently, will be used for locality based optimization in future

Specified by:

onVertexManagerEventReceived in class org.apache.tez.dag.api.VertexManagerPlugin

Parameters:

vmEvent -

Throws:

Exception

onRootVertexInitialized

public void onRootVertexInitialized(String inputName,
                                    org.apache.tez.dag.api.InputDescriptor inputDescriptor,
                                    List<org.apache.tez.runtime.api.Event> events)
                             throws Exception

Currently direct input to cartesian product vertex is not supported

Specified by:

onRootVertexInitialized in class org.apache.tez.dag.api.VertexManagerPlugin

Parameters:

inputName -

inputDescriptor -

events -

Throws:

Exception

onVertexStarted

public void onVertexStarted(List<org.apache.tez.runtime.api.TaskAttemptIdentifier> completions)
                     throws Exception

Overrides:

onVertexStarted in class org.apache.tez.dag.api.VertexManagerPlugin

Throws:

Exception

onVertexStateUpdated

public void onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate)
                          throws Exception

Overrides:

onVertexStateUpdated in class org.apache.tez.dag.api.VertexManagerPlugin

Throws:

Exception

onSourceTaskCompleted

public void onSourceTaskCompleted(org.apache.tez.runtime.api.TaskAttemptIdentifier attempt)
                           throws Exception

Overrides:

onSourceTaskCompleted in class org.apache.tez.dag.api.VertexManagerPlugin

Throws:

Exception