Class/Object

ai.catboost.spark

Pool

Related Docs: object Pool | package spark

Permalink

class Pool extends Params with HasLabelCol with HasFeaturesCol with HasWeightCol

CatBoost's abstraction of a dataset.

Features data can be stored in raw (features column has org.apache.spark.ml.linalg.Vector type) or quantized (float feature values are quantized into integer bin values, features column has Array[Byte] type) form.

Raw Pool can be transformed to quantized form using quantize method. This is useful if this dataset is used for training multiple times and quantization parameters do not change. Pre-quantized Pool allows to cache quantized features data and so do not re-run feature quantization step at the start of an each training.

Linear Supertypes
HasWeightCol, HasFeaturesCol, HasLabelCol, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Pool
  2. HasWeightCol
  3. HasFeaturesCol
  4. HasLabelCol
  5. Params
  6. Serializable
  7. Serializable
  8. Identifiable
  9. AnyRef
  10. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Pool(data: DataFrame)

    Permalink

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns.

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns. Only features and label columns with "features" and "label" names are assumed by default.

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[4]")
  .appName("PoolTest")
  .getOrCreate();
val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12", 0x0L, 0.12f),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22", 0x0L, 0.18f),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34", 0x1L, 1.0f)
)
val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType),
  StructField("groupId", LongType),
  StructField("weight", FloatType)
)
val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))
val pool = new Pool(df)
  .setGroupIdCol("groupId")
  .setWeightCol("weight")
pool.data.show()

  2. new Pool(uid: String, data: DataFrame = null, featuresLayout: TFeaturesLayout = null, quantizedFeaturesInfo: QuantizedFeaturesInfoPtr = null)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. final val baselineCol: Param[String]

    Permalink

  7. final def clear(param: Param[_]): Pool.this.type

    Permalink
    Definition Classes
    Params

  8. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. def copy(extra: ParamMap): Pool

    Permalink
    Definition Classes
    Pool → Params

  10. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  11. def count: Long

    Permalink

    returns

    Number of objects in the dataset, similar to the same method of org.apache.spark.sql.Dataset

  12. def createQuantizationSchema(quantizationParams: QuantizationParamsTrait): QuantizedFeaturesInfoPtr

    Permalink
    Attributes
    protected

  13. def createQuantized(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool

    Permalink
    Attributes
    protected

  14. val data: DataFrame

    Permalink

  15. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  16. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  17. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  18. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params

  19. def explainParams(): String

    Permalink
    Definition Classes
    Params

  20. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params

  21. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params

  22. final val featuresCol: Param[String]

    Permalink
    Definition Classes
    HasFeaturesCol

  23. var featuresLayout: TFeaturesLayout

    Permalink
    Attributes
    protected

  24. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  25. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  26. final def getBaselineCol: String

    Permalink

  27. def getBaselineCount: Int

    Permalink

    returns

    dimension of formula baseline, 0 if no baseline specified

  28. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  29. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  30. def getFeatureCount: Int

    Permalink

  31. def getFeatureNames: Array[String]

    Permalink

  32. final def getFeaturesCol: String

    Permalink
    Definition Classes
    HasFeaturesCol

  33. def getFeaturesLayout: TFeaturesLayout

    Permalink

  34. final def getGroupIdCol: String

    Permalink

  35. final def getGroupWeightCol: String

    Permalink

  36. final def getLabelCol: String

    Permalink
    Definition Classes
    HasLabelCol

  37. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params

  38. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params

  39. final def getSampleIdCol: String

    Permalink

  40. final def getSubgroupIdCol: String

    Permalink

  41. def getTargetType: ERawTargetType

    Permalink

  42. final def getTimestampCol: String

    Permalink

  43. final def getWeightCol: String

    Permalink
    Definition Classes
    HasWeightCol

  44. final val groupIdCol: Param[String]

    Permalink

  45. final val groupWeightCol: Param[String]

    Permalink

  46. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params

  47. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params

  48. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  49. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  50. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  51. def isQuantized: Boolean

    Permalink

  52. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  53. final val labelCol: Param[String]

    Permalink
    Definition Classes
    HasLabelCol

  54. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  55. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  56. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  57. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params

  58. def quantize(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool

    Permalink

    Create Pool with quantized features from Pool with raw features.

    Create Pool with quantized features from Pool with raw features. This variant of the method is useful if QuantizedFeaturesInfo with data for quantization (borders and nan modes) has already been computed. Used, for example, to quantize evaluation datasets after the training dataset has been quantized.

  59. def quantize(quantizationParams: QuantizationParamsTrait): Pool

    Permalink

    Create Pool with quantized features from Pool with raw features

    Create Pool with quantized features from Pool with raw features

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[*]")
  .appName("QuantizationTest")
  .getOrCreate();
val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12"),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22"),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34")
)
val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType)
)
val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))
val pool = new Pool(df)
val quantizedPool = pool.quantize(new QuantizationParams)
val quantizedPoolWithTwoBinsPerFeature = pool.quantize(new QuantizationParams().setBorderCount(1))
quantizedPool.data.show()
quantizedPoolWithTwoBinsPerFeature.data.show()

  60. val quantizedFeaturesInfo: QuantizedFeaturesInfoPtr

    Permalink

  61. def repartition(partitionCount: Int, byGroupColumnsIfPresent: Boolean = true): Pool

    Permalink

    Repartion data to the specified number of partitions.

    Repartion data to the specified number of partitions. Useful to repartition data to create one partition per executor for training (where each executor gets its' own CatBoost worker with a part of the training data).

  62. final val sampleIdCol: Param[String]

    Permalink

  63. final def set(paramPair: ParamPair[_]): Pool.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  64. final def set(param: String, value: Any): Pool.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  65. final def set[T](param: Param[T], value: T): Pool.this.type

    Permalink
    Definition Classes
    Params

  66. final def setBaselineCol(value: String): Pool.this.type

    Permalink

  67. final def setDefault(paramPairs: ParamPair[_]*): Pool.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  68. final def setDefault[T](param: Param[T], value: T): Pool.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  69. def setFeaturesCol(value: String): Pool

    Permalink

  70. final def setGroupIdCol(value: String): Pool.this.type

    Permalink

  71. final def setGroupWeightCol(value: String): Pool.this.type

    Permalink

  72. def setLabelCol(value: String): Pool

    Permalink

  73. final def setSampleIdCol(value: String): Pool.this.type

    Permalink

  74. final def setSubgroupIdCol(value: String): Pool.this.type

    Permalink

  75. final def setTimestampCol(value: String): Pool.this.type

    Permalink

  76. def setWeightCol(value: String): Pool

    Permalink

  77. final val subgroupIdCol: Param[String]

    Permalink

  78. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  79. final val timestampCol: Param[String]

    Permalink

  80. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any

  81. val uid: String

    Permalink
    Definition Classes
    Pool → Identifiable

  82. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  83. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  84. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  85. final val weightCol: Param[String]

    Permalink
    Definition Classes
    HasWeightCol

Inherited from HasWeightCol

Inherited from HasFeaturesCol

Inherited from HasLabelCol

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

setParam

Ungrouped