Packages

class Pool extends Params with HasLabelCol with HasFeaturesCol with HasWeightCol with Logging

CatBoost's abstraction of a dataset.

Features data can be stored in raw (features column has org.apache.spark.ml.linalg.Vector type) or quantized (float feature values are quantized into integer bin values, features column has Array[Byte] type) form.

Raw Pool can be transformed to quantized form using quantize method. This is useful if this dataset is used for training multiple times and quantization parameters do not change. Pre-quantized Pool allows to cache quantized features data and so do not re-run feature quantization step at the start of an each training.

Linear Supertypes
Logging, HasWeightCol, HasFeaturesCol, HasLabelCol, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Pool
  2. Logging
  3. HasWeightCol
  4. HasFeaturesCol
  5. HasLabelCol
  6. Params
  7. Serializable
  8. Serializable
  9. Identifiable
  10. AnyRef
  11. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Pool(data: DataFrame, pairsData: DataFrame)

    Construct Pool from DataFrame also specifying pairs data in an additional DataFrame

    Construct Pool from DataFrame also specifying pairs data in an additional DataFrame

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[4]")
  .appName("PoolWithPairsTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12", 0x0L, 0.12f, 0),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22", 0x0L, 0.18f, 1),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34", 0x1L, 1.0f, 2),
  Row(Vectors.dense(0.23, 0.01, 0.0), "0.0", 0x1L, 1.2f, 3)
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType),
  StructField("groupId", LongType),
  StructField("weight", FloatType)
  StructField("sampleId", LongType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val srcPairsData = Seq(
  Row(0x0L, 0, 1),
  Row(0x1L, 3, 2)
)

val srcPairsDataSchema = Seq(
  StructField("groupId", LongType),
  StructField("winnerId", IntegerType),
  StructField("loserId", IntegerType)
)

val pairsDf = spark.createDataFrame(
  spark.sparkContext.parallelize(srcPairsData),
  StructType(srcPairsDataSchema)
)

val pool = new Pool(df, pairsDf)
  .setGroupIdCol("groupId")
  .setWeightCol("weight")
  .setSampleIdCol("sampleId")

pool.data.show()
pool.pairsData.show()
  2. new Pool(data: DataFrame)

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns.

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns. Only features and label columns with "features" and "label" names are assumed by default.

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[4]")
  .appName("PoolTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12", 0x0L, 0.12f),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22", 0x0L, 0.18f),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34", 0x1L, 1.0f)
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType),
  StructField("groupId", LongType),
  StructField("weight", FloatType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val pool = new Pool(df)
  .setGroupIdCol("groupId")
  .setWeightCol("weight")

pool.data.show()
  3. new Pool(uid: String, data: DataFrame = null, featuresLayout: TFeaturesLayoutPtr = null, quantizedFeaturesInfo: QuantizedFeaturesInfoPtr = null, pairsData: DataFrame = null, partitionedByGroups: Boolean = false)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. final val baselineCol: Param[String]
  7. def cache(): Pool

    Persist Datasets of this Pool with the default storage level (MEMORY_AND_DISK).

  8. def calcNanModesAndBorders(nanModeAndBordersBuilder: TNanModeAndBordersBuilder, quantizationParams: QuantizationParamsTrait): Unit
    Attributes
    protected
  9. def checkpoint(): Pool

    Returns Pool with eagerly checkpointed Datasets.

  10. def checkpoint(eager: Boolean): Pool

    Returns Pool with checkpointed Datasets.

    Returns Pool with checkpointed Datasets.

    eager

    Whether to checkpoint Datasets immediately

  11. final def clear(param: Param[_]): Pool.this.type
    Definition Classes
    Params
  12. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  13. def copy(extra: ParamMap): Pool
    Definition Classes
    Pool → Params
  14. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  15. def count: Long

    returns

    Number of objects in the dataset, similar to the same method of org.apache.spark.sql.Dataset

  16. def createQuantizationSchema(quantizationParams: QuantizationParamsTrait): QuantizedFeaturesInfoPtr
    Attributes
    protected
  17. def createQuantized(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool
    Attributes
    protected
  18. val data: DataFrame
  19. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  20. def ensurePartitionByGroupsIfPresent(): Pool

    ensure that if groups are present data in partitions contains whole groups in consecutive order

  21. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  22. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  23. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  24. def explainParams(): String
    Definition Classes
    Params
  25. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  26. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  27. final val featuresCol: Param[String]
    Definition Classes
    HasFeaturesCol
  28. var featuresLayout: TFeaturesLayoutPtr
    Attributes
    protected
  29. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  30. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  31. final def getBaselineCol: String
  32. def getBaselineCount: Int

    returns

    dimension of formula baseline, 0 if no baseline specified

  33. def getCatFeaturesUniqValueCounts: Array[Int]
  34. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  35. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  36. def getEstimatedFeatureCount: Int
  37. def getFeatureCount: Int
  38. def getFeatureNames: Array[String]
  39. final def getFeaturesCol: String
    Definition Classes
    HasFeaturesCol
  40. def getFeaturesLayout: TFeaturesLayoutPtr
  41. final def getGroupIdCol: String
  42. final def getGroupWeightCol: String
  43. final def getLabelCol: String
    Definition Classes
    HasLabelCol
  44. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  45. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  46. final def getSampleIdCol: String
  47. final def getSubgroupIdCol: String
  48. def getTargetType: ERawTargetType
  49. final def getTimestampCol: String
  50. final def getWeightCol: String
    Definition Classes
    HasWeightCol
  51. final val groupIdCol: Param[String]
  52. final val groupWeightCol: Param[String]
  53. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  54. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  55. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  56. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  57. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  58. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  59. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  60. def isQuantized: Boolean
  61. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  62. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  63. final val labelCol: Param[String]
    Definition Classes
    HasLabelCol
  64. def localCheckpoint(): Pool

    Returns Pool with eagerly locally checkpointed Datasets.

  65. def localCheckpoint(eager: Boolean): Pool

    Returns Pool with locally checkpointed Datasets.

    Returns Pool with locally checkpointed Datasets.

    eager

    Whether to checkpoint Datasets immediately

  66. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  67. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  68. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  69. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  70. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  71. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  72. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  73. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  74. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  75. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  76. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  77. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  78. def mapQuantizedPartitions[R](selectedColumns: Seq[String], includeEstimatedFeatures: Boolean, includePairsIfPresent: Boolean, dstColumnNames: Array[String], dstRowLength: Int, f: (TDataProviderPtr, TDataProviderPtr, ArrayBuffer[Array[Any]], TLocalExecutor) ⇒ Iterator[R])(implicit arg0: Encoder[R], arg1: ClassTag[R]): Dataset[R]

    Map over partitions for quantized Pool

  79. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  80. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  81. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  82. def pairsCount: Long

    returns

    Number of pairs in the dataset

  83. val pairsData: DataFrame
  84. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  85. val partitionedByGroups: Boolean
  86. def persist(): Pool

    Persist Datasets of this Pool with the default storage level (MEMORY_AND_DISK).

  87. def persist(storageLevel: StorageLevel): Pool

    Returns Pool with Datasets persisted with the given storage level.

  88. def quantize(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool

    Create Pool with quantized features from Pool with raw features.

    Create Pool with quantized features from Pool with raw features. This variant of the method is useful if QuantizedFeaturesInfo with data for quantization (borders and nan modes) has already been computed. Used, for example, to quantize evaluation datasets after the training dataset has been quantized.

  89. def quantize(quantizationParams: QuantizationParamsTrait = new QuantizationParams()): Pool

    Create Pool with quantized features from Pool with raw features

    Create Pool with quantized features from Pool with raw features

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[*]")
  .appName("QuantizationTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12"),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22"),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34")
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val pool = new Pool(df)

val quantizedPool = pool.quantize(new QuantizationParams)
val quantizedPoolWithTwoBinsPerFeature = pool.quantize(new QuantizationParams().setBorderCount(1))
quantizedPool.data.show()
quantizedPoolWithTwoBinsPerFeature.data.show()
  90. def quantizeForModelApplication[Model <: PredictionModel[Vector, Model]](model: CatBoostModelTrait[Model]): Pool

    Create Pool with quantized features from Pool with raw features.

    Create Pool with quantized features from Pool with raw features. This variant of the method is used when we want to apply CatBoostModel on Pool

  91. val quantizedFeaturesInfo: QuantizedFeaturesInfoPtr
  92. def repartition(partitionCount: Int, byGroupColumnsIfPresent: Boolean = true): Pool

    Repartition data to the specified number of partitions.

    Repartition data to the specified number of partitions. Useful to repartition data to create one partition per executor for training (where each executor gets its' own CatBoost worker with a part of the training data).

  93. def sample(fraction: Double): Pool

    Create subset of this pool with the fraction of the samples (or groups of samples if present)

  94. final val sampleIdCol: Param[String]
  95. final def set(paramPair: ParamPair[_]): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  96. final def set(param: String, value: Any): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  97. final def set[T](param: Param[T], value: T): Pool.this.type
    Definition Classes
    Params
  98. final def setBaselineCol(value: String): Pool.this.type
  99. final def setDefault(paramPairs: ParamPair[_]*): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  100. final def setDefault[T](param: Param[T], value: T): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  101. def setFeaturesCol(value: String): Pool

  102. final def setGroupIdCol(value: String): Pool.this.type
  103. final def setGroupWeightCol(value: String): Pool.this.type
  104. def setLabelCol(value: String): Pool

  105. final def setSampleIdCol(value: String): Pool.this.type
  106. final def setSubgroupIdCol(value: String): Pool.this.type
  107. final def setTimestampCol(value: String): Pool.this.type
  108. def setWeightCol(value: String): Pool

  109. final val subgroupIdCol: Param[String]
  110. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  111. final val timestampCol: Param[String]
  112. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  113. val uid: String
    Definition Classes
    Pool → Identifiable
  114. def unpersist(blocking: Boolean): Pool

    Mark Datasets of this Pool as non-persistent, and remove all blocks for them from memory and disk.

    Mark Datasets of this Pool as non-persistent, and remove all blocks for them from memory and disk.

    blocking

    Whether to block until all blocks are deleted.

  115. def unpersist(): Pool

    Mark Datasets of this Pool as non-persistent, and remove all blocks for them from memory and disk.

  116. def updateCatFeaturesInfo(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Unit
    Attributes
    protected
  117. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  118. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  119. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  120. final val weightCol: Param[String]
    Definition Classes
    HasWeightCol

Inherited from Logging

Inherited from HasWeightCol

Inherited from HasFeaturesCol

Inherited from HasLabelCol

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Caching and Persistence

setParam

Ungrouped