AbstractVwCrossValidationObjective

Instance Constructors

new AbstractVwCrossValidationObjective(numFolds: Int, vwDataset: Iterator[String], vwTrainParamsString: Option[String], vwTestParamsString: Option[String])

numFolds
vwDataset
vwTrainParamsString
vwTestParamsString

Abstract Value Members

abstract def get(name: String): File

Definition Classes
FileFunctions
abstract def save(file: File): String

Definition Classes
FileFunctions

Concrete Value Members

final def !=(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def !=(arg0: Any): Boolean

Definition Classes
Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def ==(arg0: Any): Boolean

Definition Classes
Any
def apply(point: Point): Double

This method can run on the driver and/or the executor.
This method can run on the driver and/or the executor. It performs a k-fold cross validation over the vw input dataset passed through the class constructor. The dataset has been split in such a way that every fold has its own training and test set in the form of VW cache files.
point
a point object representing the hyper parameters to evaluate upon
returns
Double the cross validated average loss

Definition Classes
AbstractVwCrossValidationObjective → Objective
final def asInstanceOf[T0]: T0

Definition Classes
Any
val cacheBitSize: Int
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def debug(msg: ⇒ Any, t: ⇒ Throwable): Unit

Attributes
protected
Definition Classes
Logging
def debug(msg: ⇒ Any): Unit

Attributes
protected
Definition Classes
Logging
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def error(msg: ⇒ Any, t: ⇒ Throwable): Unit

Attributes
protected
Definition Classes
Logging
def error(msg: ⇒ Any): Unit

Attributes
protected
Definition Classes
Logging
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
val foldToVwCacheFiles: Map[Int, (String, String)]
def getCache(name: String): File

Definition Classes
VwDatasetFunctions
def getCacheBitSize(vwParamMap: Map[String, _]): Int

Definition Classes
VwFunctions
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getTestVwParams(vwParamMap: Map[String, _], point: Point): String

Definition Classes
VwFunctions
def getTrainVwParams(vwParamMap: Map[String, _], point: Point): String

Definition Classes
VwFunctions
def hashCode(): Int

Definition Classes
AnyRef → Any
def info(msg: ⇒ Any, t: ⇒ Throwable): Unit

Attributes
protected
Definition Classes
Logging
def info(msg: ⇒ Any): Unit

Attributes
protected
Definition Classes
Logging
def isDebugEnabled: Boolean

Attributes
protected
Definition Classes
Logging
def isErrorEnabled: Boolean

Attributes
protected
Definition Classes
Logging
def isInfoEnabled: Boolean

Attributes
protected
Definition Classes
Logging
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isTraceEnabled: Boolean

Attributes
protected
Definition Classes
Logging
def isWarnEnabled: Boolean

Attributes
protected
Definition Classes
Logging
def kFold(vwDataset: Iterator[String], folds: Int, cacheBitSize: Int): Map[Int, (String, String)]

This method takes the VW input file specified in the class constructor and partitions the file into a training set and test set for every fold.
This method takes the VW input file specified in the class constructor and partitions the file into a training set and test set for every fold. The train and test set for every fold are then input into VW to generate cache files. These cache files are added to the SparkContext so that they'll be accessible on the executors. To keep track of the train and test set caches for every fold, a Map is used where the key is the fold number and the value is (trainingSetCachePath, testSetCachePath). These file names do NEED to be unique so that they do not collide with other file names. The entirety of this method runs on the driver. All VW input training / test set files as well as cache files are deleted upon JVM exit.
This strategy has the downside of duplicating the dataset across every node K times. An alternative approach is to train K cache files and train the regressor K - 1 times and test on the last test cache file.
vwDataset
folds
returns
a map representation where key is the fold number and value is (trainingSetFilename, testSetFilename)

Definition Classes
VwCrossValidation
def kFold(vwDataset: Iterable[String], folds: Int, cacheBitSize: Int): Map[Int, (String, String)]

Definition Classes
VwCrossValidation
def kFold(inputPath: String, folds: Int, cacheBitSize: Int): Map[Int, (String, String)]

Definition Classes
VwCrossValidation
def logger: Logger

Attributes
protected
Definition Classes
Logging
def loggerName: String

Attributes
protected
Definition Classes
Logging
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val numFolds: Int
def parseVwArgs(args: Option[String]): Map[String, Any]

Definition Classes
VwFunctions
def save(inputIterator: Iterator[String]): String

Definition Classes
FileFunctions
def save(inputIterable: Iterable[String]): String

Definition Classes
FileFunctions
def save(inputPath: String): String

Definition Classes
FileFunctions
def saveAsCache(vwDatasetPath: String, vwCacheFilename: String, bitSize: Int): String

Definition Classes
VwDatasetFunctions
def saveAsCache(vwDatasetIterator: Iterator[String], vwCacheFilename: String, bitSize: Int): String

Definition Classes
VwDatasetFunctions
def saveAsCache(vwDatasetInputStream: InputStream, vwCacheFilename: String, bitSize: Int): String

Definition Classes
VwDatasetFunctions
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
def trace(msg: ⇒ Any, t: ⇒ Throwable): Unit

Attributes
protected
Definition Classes
Logging
def trace(msg: ⇒ Any): Unit

Attributes
protected
Definition Classes
Logging
val vwDataset: Iterator[String]
val vwTestParamsMap: Map[String, Any]
val vwTrainParamsMap: Map[String, Any]
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def warn(msg: ⇒ Any, t: ⇒ Throwable): Unit

Attributes
protected
Definition Classes
Logging
def warn(msg: ⇒ Any): Unit

Attributes
protected
Definition Classes
Logging

abstract class AbstractVwCrossValidationObjective extends Objective[Point, Double] with VwFunctions with VwCrossValidation with Logging

Instance Constructors

new AbstractVwCrossValidationObjective(numFolds: Int, vwDataset: Iterator[String], vwTrainParamsString: Option[String], vwTestParamsString: Option[String])

Abstract Value Members

abstract def get(name: String): File

abstract def save(file: File): String

Concrete Value Members

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

def apply(point: Point): Double

final def asInstanceOf[T0]: T0

val cacheBitSize: Int

def clone(): AnyRef

def debug(msg: ⇒ Any, t: ⇒ Throwable): Unit

def debug(msg: ⇒ Any): Unit

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def error(msg: ⇒ Any, t: ⇒ Throwable): Unit

def error(msg: ⇒ Any): Unit

def finalize(): Unit

val foldToVwCacheFiles: Map[Int, (String, String)]

def getCache(name: String): File

def getCacheBitSize(vwParamMap: Map[String, _]): Int

final def getClass(): Class[_]

def getTestVwParams(vwParamMap: Map[String, _], point: Point): String

def getTrainVwParams(vwParamMap: Map[String, _], point: Point): String

def hashCode(): Int

def info(msg: ⇒ Any, t: ⇒ Throwable): Unit

def info(msg: ⇒ Any): Unit

def isDebugEnabled: Boolean

def isErrorEnabled: Boolean

def isInfoEnabled: Boolean

final def isInstanceOf[T0]: Boolean

def isTraceEnabled: Boolean

def isWarnEnabled: Boolean

def kFold(vwDataset: Iterator[String], folds: Int, cacheBitSize: Int): Map[Int, (String, String)]

def kFold(vwDataset: Iterable[String], folds: Int, cacheBitSize: Int): Map[Int, (String, String)]

def kFold(inputPath: String, folds: Int, cacheBitSize: Int): Map[Int, (String, String)]

def logger: Logger

def loggerName: String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val numFolds: Int

def parseVwArgs(args: Option[String]): Map[String, Any]

def save(inputIterator: Iterator[String]): String

def save(inputIterable: Iterable[String]): String

def save(inputPath: String): String

def saveAsCache(vwDatasetPath: String, vwCacheFilename: String, bitSize: Int): String

def saveAsCache(vwDatasetIterator: Iterator[String], vwCacheFilename: String, bitSize: Int): String

def saveAsCache(vwDatasetInputStream: InputStream, vwCacheFilename: String, bitSize: Int): String

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def trace(msg: ⇒ Any, t: ⇒ Throwable): Unit

def trace(msg: ⇒ Any): Unit

val vwDataset: Iterator[String]

val vwTestParamsMap: Map[String, Any]

val vwTrainParamsMap: Map[String, Any]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def warn(msg: ⇒ Any, t: ⇒ Throwable): Unit

def warn(msg: ⇒ Any): Unit

Inherited from Logging

Inherited from VwCrossValidation

Inherited from VwDatasetFunctions

Inherited from FileFunctions

Inherited from VwFunctions

Inherited from Objective[Point, Double]

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped