com.linkedin.feathr.common.util

Class MvelContextUDFs

java.lang.Object

com.linkedin.feathr.common.util.MvelContextUDFs

public class MvelContextUDFs
extends java.lang.Object

MVEL is an open-source expression language and runtime that makes it easy to write concise statements that operate on structured data objects (such as Avro records), among other things. This class contains all the udfs used in Mvel for both online and offline

Method Summary

Modifier and Type	Method and Description
static boolean	and(boolean left, boolean right)
static java.lang.Double	cast_double(java.lang.Object input) Cast the input to double.
static java.lang.Float	cast_float(java.lang.Object input) Cast the input to float.
static java.lang.Integer	cast_int(java.lang.Object input) Cast the input to Integer.
static java.lang.String	concat(java.lang.String left, java.lang.String right) Concatenate two strings into one.
static java.lang.Float	cosineSimilarity(java.lang.Object obj1, java.lang.Object obj2)
static int	dayofmonth(java.lang.Object input)
static int	dayofweek(java.lang.Object input)
static java.util.Collection<java.lang.Object>	distinct(java.util.Collection<java.lang.Object> collection)
static java.lang.Double	dotProduct(java.lang.Object obj1, java.lang.Object obj2) Returns a standard dotProduct of two vector objects.
static java.util.Map<java.lang.String,java.lang.Float>	extract_term_value_from_array(java.util.ArrayList<org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema> array, java.lang.String termFieldName, java.lang.String valueFieldName)
static java.util.Map<java.lang.String,java.lang.Float>	extract_term_value_from_array(java.util.ArrayList<org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema> array, java.lang.String termFieldName, java.lang.String valueFieldName, java.lang.String filterExpr)
static java.util.Collection<java.lang.Object>	flatten(java.util.Collection<? extends java.util.Collection<java.lang.Object>> collection)
static java.lang.String	get_data_type(java.lang.Object input) Get the class type of the input object WARNING: This is only used for debug for users.
static java.util.Collection<java.lang.String>	getTerms(java.lang.Object item)
static java.util.List<java.lang.String>	getTopKTerms(java.lang.Object item, int k) return an ordered list of terms, based on descending order of corresponding values
static java.lang.String	getTopTerm(java.lang.Object item) get terms with the highest value (duplicate will resolved randomly)
static int	hourofday(java.lang.Object input)
static boolean	if_else(boolean input, boolean first, boolean second)
static java.lang.Double	if_else(boolean input, java.lang.Double first, java.lang.Double second)
static java.lang.Float	if_else(boolean input, java.lang.Float first, java.lang.Float second)
static java.lang.Integer	if_else(boolean input, java.lang.Integer first, java.lang.Integer second)
static java.lang.String	if_else(boolean input, java.lang.String first, java.lang.String second) Ternary operator.
static boolean	isNonZero(java.lang.Object vector) Return true if a feature has at least 1 term with value not zero.
static boolean	isnotnull(java.lang.Object input)
static boolean	isnull(java.lang.Object input)
static boolean	isPresent(java.lang.Object featureValue) return true if a feature variable is not null.
static boolean	not(boolean input)
static boolean	or(boolean left, boolean right)
static void	registerUDFs(org.mvel2.ParserConfiguration parserConfig)
static long	time_duration(java.lang.Object startTime, java.lang.Object endTime, java.lang.String outputGranularity)
static java.lang.Boolean	toBoolean(java.lang.Object item) Converts an object to a boolean
static java.lang.Object	toCategorical(java.lang.Object item) Convert input to categorical feature Example inputs that can be converted - Map("a" -> 2.0) return Map("a" -> 2.0) - Map("" -> 2.5) returns ("2.5", 1.0)
static java.lang.String	toLowerCase(java.lang.String input) convert input to lower case string
static java.lang.Object	toNumeric(java.lang.Object item) Convert input to numeric value Example inputs that can be converted - Map("" -> 2.0) returns 2.0 - Map("876" -> 1.0) returns 876
static java.lang.String	toUpperCase(java.lang.String input) convert input to upper case string

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

registerUDFs

public static void registerUDFs(org.mvel2.ParserConfiguration parserConfig)

get_data_type

public static java.lang.String get_data_type(java.lang.Object input)

Get the class type of the input object WARNING: This is only used for debug for users.

Returns:

Type in String form

cast_double

public static java.lang.Double cast_double(java.lang.Object input)

Cast the input to double. If it's null, null is returned. If it's string, it will try to parse it. For number types, it will do standard conversion. For other types, it will be coerced to double.

cast_float

public static java.lang.Float cast_float(java.lang.Object input)

Cast the input to float. If it's null, null is returned. If it's string, it will try to parse it. For number types, it will do standard conversion. For other types, it will be coerced to float.

cast_int

public static java.lang.Integer cast_int(java.lang.Object input)

Cast the input to Integer. If it's null, null is returned. If it's string, it will try to parse it. For number types, it will do standard conversion. For other types, it will be coerced to Integer.

and

public static boolean and(boolean left,
                          boolean right)

or

public static boolean or(boolean left,
                         boolean right)

not

public static boolean not(boolean input)

isnull

public static boolean isnull(java.lang.Object input)

isnotnull

public static boolean isnotnull(java.lang.Object input)

concat

public static java.lang.String concat(java.lang.String left,
                                      java.lang.String right)

Concatenate two strings into one.

if_else

public static java.lang.String if_else(boolean input,
                                       java.lang.String first,
                                       java.lang.String second)

Ternary operator. If input is evaluated to true, then first is returned, else second is returned.

if_else

public static java.lang.Double if_else(boolean input,
                                       java.lang.Double first,
                                       java.lang.Double second)

if_else

public static java.lang.Float if_else(boolean input,
                                      java.lang.Float first,
                                      java.lang.Float second)

if_else

public static java.lang.Integer if_else(boolean input,
                                        java.lang.Integer first,
                                        java.lang.Integer second)

if_else

public static boolean if_else(boolean input,
                              boolean first,
                              boolean second)

isNonZero

public static boolean isNonZero(java.lang.Object vector)

Return true if a feature has at least 1 term with value not zero. For checking if a feature is null, please use isPresent().

isPresent

public static boolean isPresent(java.lang.Object featureValue)

return true if a feature variable is not null.

toBoolean

public static java.lang.Boolean toBoolean(java.lang.Object item)

Converts an object to a boolean

toNumeric

public static java.lang.Object toNumeric(java.lang.Object item)

Convert input to numeric value Example inputs that can be converted - Map("" -> 2.0) returns 2.0 - Map("876" -> 1.0) returns 876

toCategorical

public static java.lang.Object toCategorical(java.lang.Object item)

Convert input to categorical feature Example inputs that can be converted - Map("a" -> 2.0) return Map("a" -> 2.0) - Map("" -> 2.5) returns ("2.5", 1.0)

getTerms

public static java.util.Collection<java.lang.String> getTerms(java.lang.Object item)

getTopKTerms

public static java.util.List<java.lang.String> getTopKTerms(java.lang.Object item,
                                                            int k)

return an ordered list of terms, based on descending order of corresponding values

Parameters:

item - Object that can be converted to Map of string to float

k - integer, if k<0, do reverse order selection, e.g. -1: select bottom first

Returns:

List of string, ordered

getTopTerm

public static java.lang.String getTopTerm(java.lang.Object item)

get terms with the highest value (duplicate will resolved randomly)

Parameters:

item - Object that can be converted to Map of string to float

Returns:

String

distinct

public static java.util.Collection<java.lang.Object> distinct(java.util.Collection<java.lang.Object> collection)

flatten

public static java.util.Collection<java.lang.Object> flatten(java.util.Collection<? extends java.util.Collection<java.lang.Object>> collection)

cosineSimilarity

public static java.lang.Float cosineSimilarity(java.lang.Object obj1,
                                               java.lang.Object obj2)

dotProduct

public static java.lang.Double dotProduct(java.lang.Object obj1,
                                          java.lang.Object obj2)

Returns a standard dotProduct of two vector objects. Use cosineSimilarity(Object, Object) for normalized dot-product.

toLowerCase

public static java.lang.String toLowerCase(java.lang.String input)

convert input to lower case string

Parameters:

input - input string

Returns:

lower case input

toUpperCase

public static java.lang.String toUpperCase(java.lang.String input)

convert input to upper case string

Parameters:

input - input string

Returns:

upper case input

time_duration

public static long time_duration(java.lang.Object startTime,
                                 java.lang.Object endTime,
                                 java.lang.String outputGranularity)

dayofweek

public static int dayofweek(java.lang.Object input)

dayofmonth

public static int dayofmonth(java.lang.Object input)

hourofday

public static int hourofday(java.lang.Object input)

extract_term_value_from_array

public static java.util.Map<java.lang.String,java.lang.Float> extract_term_value_from_array(java.util.ArrayList<org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema> array,
                                                                                            java.lang.String termFieldName,
                                                                                            java.lang.String valueFieldName)

extract_term_value_from_array

public static java.util.Map<java.lang.String,java.lang.Float> extract_term_value_from_array(java.util.ArrayList<org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema> array,
                                                                                            java.lang.String termFieldName,
                                                                                            java.lang.String valueFieldName,
                                                                                            java.lang.String filterExpr)