Packages

object Metrics extends StrictLogging

Linear Supertypes
StrictLogging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Metrics
  2. StrictLogging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class ContinuousMetric(name: String, function: (Column) ⇒ Column) extends Product with Serializable

    Case class ContinuousMetric with all corresponding Metrics

    Case class ContinuousMetric with all corresponding Metrics

    name

    : the name of the variable

    function

    : the metric function

  2. case class DiscreteMetric(name: String, function: ((Column, DataFrame)) ⇒ Column) extends Product with Serializable
  3. case class MetricsDatasets(continuousDF: Option[DataFrame], discreteDF: Option[DataFrame], frequenciesDF: Option[DataFrame]) extends Product with Serializable

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def categoryCountFreqDataframe(e: Column, dataInit: DataFrame): (Column, DataFrame)

    Function to compute the Dataframe with Category, Count and Frequencies obtain from the initial Dataframe

    Function to compute the Dataframe with Category, Count and Frequencies obtain from the initial Dataframe

    e

    : column of the variable.

    dataInit

    : initial DataFrame.

    returns

    (Column, DataFrame) : tuple2 of the column of the variable and the initial Dataframe

  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. def computeContinuousMetric(dataset: DataFrame, continuousAttributes: List[String], operations: List[ContinuousMetric]): Option[DataFrame]

    Function to compute the DataFrame metrics by row

    Function to compute the DataFrame metrics by row

    dataset

    : initial DataFrame.

    continuousAttributes

    : name list of all variables.

    operations

    : list of metrics you want to calculate.

    returns

    DataFrame : DataFrame metric of all variables by row.

  8. def computeDiscretMetric(dataInit: DataFrame, discreteAttrs: List[String], operations: List[DiscreteMetric]): Option[DataFrame]

    Function to compute and to combine all the partial DataFrame metric by variable (to get one DataFrame by row).

    Function to compute and to combine all the partial DataFrame metric by variable (to get one DataFrame by row).

    dataInit

    : initial DataFrame.

    discreteAttrs

    : name of the variable.

    operations

    : list of metrics you want to calculate.

    returns

    DataFrame : DataFrame with alle the metric by variable by row

  9. val continuousMetrics: List[ContinuousMetric]

    List of all available metrics

  10. def customCatCountFreq(colNameDataCatCount: (Column, DataFrame)): Column

    Customize catCountFreq for discrete variable

  11. def customCategory(colNameDataCatCount: (Column, DataFrame)): Column

    Customize Category for discrete variable

    Customize Category for discrete variable

    colNameDataCatCount

    : couple of name of the variable and the dataframe obtain from categoryCountFreqDataframe()

    returns

    Column : the computed value of the function metricCategory

  12. def customCountDiscrete(colNameDataCatCount: (Column, DataFrame)): Column

    Customize Count Discrete for discrete variable

    Customize Count Discrete for discrete variable

    colNameDataCatCount

    : couple of name of the variable and the dataframe obtain from categoryCountFreqDataframe()

    returns

    Column : the computed value of the function metricCountDiscret

  13. def customCountDistinct(colNameDataCatCount: (Column, DataFrame)): Column

    Customize CountDistinct for discrete variable

    Customize CountDistinct for discrete variable

    colNameDataCatCount

    : couple of name of the variable and the dataframe obtain from categoryCountFreqDataframe()

    returns

    Column : the computed value of the function metricCountDistinct

  14. def customCountMissValues(e: Column): Column

    Customize missing values

    Customize missing values

    e

    : the column

    returns

    Integer : the number of missing values, NaN values and null values

  15. def customCountMissValuesDiscrete(colNameDataCatCount: (Column, DataFrame)): Column

    Customize number of Missing Values for discrete variable

    Customize number of Missing Values for discrete variable

    colNameDataCatCount

    : couple of name of the variable and the dataframe obtain from categoryCountFreqDataframe()

    returns

    Column : the computed value of the function metricMissingValues

  16. def customFrequencies(colNameDataCatCount: (Column, DataFrame)): Column

    Customize Count Distinct for discrete variable

    Customize Count Distinct for discrete variable

    colNameDataCatCount

    : couple of name of the variable and the dataframe obtain from categoryCountFreqDataframe()

    returns

    Column : the computed value of the function metricCountDistinct

  17. def customMean(e: Column): Column

    Customize mean of the column e

    Customize mean of the column e

    e

    : the column

    returns

    Integer : the computed value of the mean

  18. def customMedian(e: Column): Column

    Customize Median of the column e

    Customize Median of the column e

    e

    : the column

    returns

    Integer : the computed value of the Median

  19. def customMetric(e: Column, metricName: String, metricFunction: (Column) ⇒ Column): Column

    Customize function metric in the case continuous variabes used for : mean, variance and stddev

    Customize function metric in the case continuous variabes used for : mean, variance and stddev

    e

    : the column

    metricName

    : the name of the metric

    metricFunction

    : the metric function

    returns

    : the computed value of the function

  20. def customMetricDiscret(e: Column, dataCategoryCount: DataFrame, metricName: String, metricFunction: (DataFrame) ⇒ Column): Column

    Customize Metric Discret for discrete variable

    Customize Metric Discret for discrete variable

    e

    : name of the column

    dataCategoryCount

    : the dataframe obtain from categoryCountFreqDataframe()

    metricName

    : te metric name

    metricFunction

    : the metric function

    returns

    Column : the computed value of the function

  21. def customMetricUDF(e: Column, metricName: String, metricFunction: (String, Column*) ⇒ Column, approxMethod: String, approxValue: Double): Column

    Customize function metric in the case continuous variabes used for : percentile 25, median and percentile75

    Customize function metric in the case continuous variabes used for : percentile 25, median and percentile75

    e

    : the column

    metricName

    : the name of the metric

    metricFunction

    : the metric function

    approxMethod

    : the approximation method

    approxValue

    : the value to pass to stat_method

  22. def customStddev(e: Column): Column

    Customize Stddev of the column e

    Customize Stddev of the column e

    e

    : the name of the column

    returns

    Integer : the computed value of the Stddev

  23. def customVariance(e: Column): Column

    Customize variance of the column e

    Customize variance of the column e

    e

    : the name of the column

    returns

    Integer : the computed value of the variance

  24. def dataToMetricData(colNamDataCatCountFreq: (Column, DataFrame), operations: List[DiscreteMetric]): DataFrame

    Function to compute the Dataframe metric by variable

    Function to compute the Dataframe metric by variable

    colNamDataCatCountFreq

    : tuple of column variable and the Dataframe with Category, Count and Frequencies obtain from categoryCountFreqDataframe()

    operations

    : list of metrics you want to calculate.

    returns

    Dataframe : with all the values of discrete metrics

  25. val discreteMetrics: List[DiscreteMetric]

    List of all available metrics.

  26. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  27. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  28. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  29. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  30. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  31. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  32. val logger: Logger
    Attributes
    protected
    Definition Classes
    StrictLogging
  33. def metricCatCountFreq(dataCategoryCount: DataFrame): Column

    Function to extract the column that contains the list of struct cat_count_freq

  34. def metricCategory(dataCategoryCount: DataFrame): Column

    Function to extract the column that contains the list of category

    Function to extract the column that contains the list of category

    dataCategoryCount

    : the data frame obtain from categoryCountFreqDataframe()

    returns

    Column : of that contain the list of category values

  35. def metricCountDiscret(dataCategoryCount: DataFrame): Column

    Function to extract the column that contains the list of CountDiscret

    Function to extract the column that contains the list of CountDiscret

    dataCategoryCount

    : the data frame obtain from categoryCountFreqDataframe()

    returns

    Column : of that contain the list of CountDiscrete values

  36. def metricCountDistinct(dataCategoryCount: DataFrame): Column

    Function to extract the column that contains the list of CountDistinct

    Function to extract the column that contains the list of CountDistinct

    dataCategoryCount

    : the data frame obtain from categoryCountFreqDataframe()

    returns

    Column : of that contain the list of CountDistinct values

  37. def metricFrequency(dataCategoryCount: DataFrame): Column

    Function to extract the column that contains the list of frequencies

    Function to extract the column that contains the list of frequencies

    dataCategoryCount

    : the data frame obtain from categoryCountFreqDataframe()

    returns

    Column : of that contain the list of frequencies values

  38. def metricMissingValues(dataCategoryCount: DataFrame): Column

    Function to extract the column that contains the list of number of Missing values

    Function to extract the column that contains the list of number of Missing values

    dataCategoryCount

    : the data frame obtain from categoryCountFreqDataframe()

    returns

    Column : of that contain the list of Missing Values values

  39. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  40. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  41. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  42. def percentile25(e: Column): Column

    Customize percentile of order 0.25 of the column e

    Customize percentile of order 0.25 of the column e

    e

    : the column

    returns

    Integer : the computed value of the percentile of order 0.25

  43. def percentile75(e: Column): Column

    Customize percentile of order 0.75 of the column e

    Customize percentile of order 0.75 of the column e

    e

    : the column

    returns

    Integer : the computed value of the percentile of order 0.75

  44. def regroupContinuousMetricsByVariable(nameCol: String, metricFrame: DataFrame): DataFrame

    Function to regroup and reformat all metrics for a given variable

    Function to regroup and reformat all metrics for a given variable

    nameCol

    : the name of the column.

    metricFrame

    : the DataFrame of all the computed metrics for each variable by columns.

    returns

    : the DataFrame metric associated to the variable (namecol).

  45. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  46. def toString(): String
    Definition Classes
    AnyRef → Any
  47. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  48. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  49. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  50. object CatCountFreq extends DiscreteMetric
  51. object CountDiscrete extends DiscreteMetric
  52. object CountDistinct extends DiscreteMetric
  53. object CountMissValues extends ContinuousMetric
  54. object CountMissValuesDiscrete extends DiscreteMetric
  55. object Kurtosis extends ContinuousMetric
  56. object Max extends ContinuousMetric
  57. object Mean extends ContinuousMetric
  58. object Median extends ContinuousMetric
  59. object Min extends ContinuousMetric
  60. object Percentile25 extends ContinuousMetric
  61. object Percentile75 extends ContinuousMetric
  62. object Skewness extends ContinuousMetric
  63. object Stddev extends ContinuousMetric
  64. object Sum extends ContinuousMetric
  65. object Variance extends ContinuousMetric

Inherited from StrictLogging

Inherited from AnyRef

Inherited from Any

Ungrouped