ai.h2o.automl

Class Leaderboard

java.lang.Object

water.Iced<T>

water.Keyed<Leaderboard>

ai.h2o.automl.Leaderboard

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, water.Freezable<Leaderboard>

public class Leaderboard
extends water.Keyed<Leaderboard>

Utility to track all the models built for a given dataset type.

Note that if a new Leaderboard is made for the same project_name it'll keep using the old model list, which allows us to run AutoML multiple times and keep adding to the leaderboard.

The models are returned sorted by either an appropriate default metric for the model category (auc, mean per class error, or mean residual deviance), or by a metric that's set via #setMetricAndDirection.

TODO: make this robust against removal of models from the DKV.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Leaderboard.LeaderboardKeyV3

Field Summary

Fields

Modifier and Type	Field and Description
double[]	auc
double[]	logloss
double[]	mae
double[]	mean_per_class_error
double[]	mean_residual_deviance Additional metrics for the models in this leaderboard, in the same order as the models Regression metrics: rmse, mse, mae, and rmsle Binomial metrics: logloss, mean_per_class_error, rmse, & mse Multinomial metrics: logloss, mean_per_class_error, rmse, & mse
double[]	mse
double[]	rmse
double[]	rmsle
double[]	sort_metrics Sort metrics for the models in this leaderboard, in the same order as the models.

Fields inherited from class water.Keyed

_key

Constructor Summary

Constructors

Constructor and Description
Leaderboard(java.lang.String project_name, UserFeedback userFeedback, water.fvec.Frame leaderboardFrame, java.lang.String sort_metric)

Method Summary

Modifier and Type	Method and Description
java.lang.String	getProject()
static java.lang.String	idForProject(java.lang.String project_name)
java.lang.String	toString()
water.util.TwoDimTable	toTwoDimTable()

Methods inherited from class water.Keyed

checksum_impl, checksum, makeSchema, readAll_impl, readAll, remove_impl, remove, remove, remove, remove, writeAll_impl, writeAll

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

sort_metrics

public double[] sort_metrics

Sort metrics for the models in this leaderboard, in the same order as the models.

Updated inside addModels().

mean_residual_deviance

public double[] mean_residual_deviance

Additional metrics for the models in this leaderboard, in the same order as the models Regression metrics: rmse, mse, mae, and rmsle Binomial metrics: logloss, mean_per_class_error, rmse, & mse Multinomial metrics: logloss, mean_per_class_error, rmse, & mse

Updated inside addModels().

rmse

public double[] rmse

mse

public double[] mse

mae

public double[] mae

rmsle

public double[] rmsle

logloss

public double[] logloss

auc

public double[] auc

mean_per_class_error

public double[] mean_per_class_error

Constructor Detail

Leaderboard

public Leaderboard(java.lang.String project_name,
                   UserFeedback userFeedback,
                   water.fvec.Frame leaderboardFrame,
                   java.lang.String sort_metric)

Method Detail

idForProject

public static java.lang.String idForProject(java.lang.String project_name)

getProject

public java.lang.String getProject()

toTwoDimTable

public water.util.TwoDimTable toTwoDimTable()

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object