ai.h2o.automl

Class Leaderboard

java.lang.Object

water.Iced<T>

water.Keyed<Leaderboard>

ai.h2o.automl.Leaderboard

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, water.Freezable<Leaderboard>

public class Leaderboard
extends water.Keyed<Leaderboard>

Utility to track all the models built for a given dataset type.

Note that if a new Leaderboard is made for the same project_name it'll keep using the old model list, which allows us to run AutoML multiple times and keep adding to the leaderboard.

The models are returned sorted by either an appropriate default metric for the model category (auc, mean per class error, or mean residual deviance), or by a metric that's set via #setMetricAndDirection.

TODO: make this robust against removal of models from the DKV.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Leaderboard.LeaderboardKeyV3

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.lang.String[]	colFormatsBinomial
protected static java.lang.String[]	colFormatsMultinomial
protected static java.lang.String[]	colFormatsRegression
protected static java.lang.String[]	colTypesBinomial
protected static java.lang.String[]	colTypesMultinomial
protected static java.lang.String[]	colTypesRegression
double[]	logloss
double[]	mae
double[]	rmse Additional metrics for the models in this leaderboard, in the same order as the models rmse, mae, and rmsle for regression & logloss for binomial classification
double[]	rmsle
double[]	sort_metrics Sort metrics for the models in this leaderboard, in the same order as the models.

Fields inherited from class water.Keyed

_key

Constructor Summary

Constructors

Constructor and Description
Leaderboard(java.lang.String project_name, UserFeedback userFeedback, water.fvec.Frame leaderboardFrame)

Method Summary

Modifier and Type	Method and Description
void	addModel(water.Key<hex.Model> key)
void	addModel(hex.Model model)
void	addModels(water.Key<hex.Model>[] newModels) Add the given models to the leaderboard.
void	addTwoDimTableRowBinomial(water.util.TwoDimTable table, int row, java.lang.String[] modelIDs, double[] errors, double[] otherErrors)
void	addTwoDimTableRowMultinomial(water.util.TwoDimTable table, int row, java.lang.String[] modelIDs, double[] errors)
void	addTwoDimTableRowRegression(water.util.TwoDimTable table, int row, java.lang.String[] modelIDs, double[] errors, double[] rmse, double[] mae, double[] rmsle)
protected static java.lang.String[]	colHeaders(java.lang.String metric, java.lang.String[] other_metric)
protected static java.lang.String[]	colHeadersMult(java.lang.String metric)
static double[]	defaultMetricForModel(hex.Model m)
static double[]	defaultMetricForModel(hex.Model m, hex.ModelMetrics mm)
static java.lang.String[]	defaultMetricNameForModel(hex.Model m)
void	delete() Delete everything in the DKV that this points to.
void	deleteWithChildren()
hex.Model	getLeader()
int	getModelCount() Return the number of models in this Leaderboard.
water.Key<hex.Model>[]	getModelKeys()
hex.Model[]	getModels()
hex.Model[]	getModels(java.lang.String metric, boolean sortDecreasing)
static Leaderboard	getOrMakeLeaderboard(java.lang.String project_name, UserFeedback userFeedback, water.fvec.Frame leaderboardFrame)
static double[]	getOtherMetrics(java.lang.String other_metric, water.util.IcedHashMap<water.Key<hex.ModelMetrics>,hex.ModelMetrics> leaderboard_set_metrics, water.fvec.Frame leaderboardFrame, hex.Model[] models)
java.lang.String	getProject()
double[]	getSortMetrics()
static double[]	getSortMetrics(java.lang.String sort_metric, water.util.IcedHashMap<water.Key<hex.ModelMetrics>,hex.ModelMetrics> leaderboard_set_metrics, water.fvec.Frame leaderboardFrame, hex.Model[] models)
static java.lang.String	idForProject(java.lang.String project_name)
static water.util.TwoDimTable	makeTwoDimTable(java.lang.String tableHeader, java.lang.String sort_metric, java.lang.String[] other_metric, int length)
water.Key<hex.Model>[]	modelKeys(java.lang.String metric, boolean sortDecreasing)
java.lang.String	rankTsv()
void	setDefaultMetricAndDirection(hex.Model m)
void	setMetricAndDirection(java.lang.String metric, boolean sortDecreasing)
void	setMetricAndDirection(java.lang.String metric, java.lang.String[] otherMetrics, boolean sortDecreasing)
java.lang.String	timeTsv()
java.lang.String	toString()
static java.lang.String	toString(java.lang.String project_name, hex.Model[] models, java.lang.String fieldSeparator, java.lang.String lineSeparator, boolean includeTitle, boolean includeHeader)
java.lang.String	toString(java.lang.String fieldSeparator, java.lang.String lineSeparator)
water.util.TwoDimTable	toTwoDimTable()
water.util.TwoDimTable	toTwoDimTable(java.lang.String tableHeader, boolean leftJustifyModelIds)

Methods inherited from class water.Keyed

checksum_impl, checksum, makeSchema, readAll_impl, readAll, remove_impl, remove, remove, remove, remove, writeAll_impl, writeAll

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

sort_metrics

public double[] sort_metrics

Sort metrics for the models in this leaderboard, in the same order as the models.

Updated inside addModels().

rmse

public double[] rmse

Additional metrics for the models in this leaderboard, in the same order as the models rmse, mae, and rmsle for regression & logloss for binomial classification

Updated inside addModels().

mae

public double[] mae

rmsle

public double[] rmsle

logloss

public double[] logloss

colTypesMultinomial

protected static final java.lang.String[] colTypesMultinomial

colFormatsMultinomial

protected static final java.lang.String[] colFormatsMultinomial

colTypesBinomial

protected static final java.lang.String[] colTypesBinomial

colFormatsBinomial

protected static final java.lang.String[] colFormatsBinomial

colTypesRegression

protected static final java.lang.String[] colTypesRegression

colFormatsRegression

protected static final java.lang.String[] colFormatsRegression

Constructor Detail

Leaderboard

public Leaderboard(java.lang.String project_name,
                   UserFeedback userFeedback,
                   water.fvec.Frame leaderboardFrame)

Method Detail

getOrMakeLeaderboard

public static Leaderboard getOrMakeLeaderboard(java.lang.String project_name,
                                               UserFeedback userFeedback,
                                               water.fvec.Frame leaderboardFrame)

idForProject

public static java.lang.String idForProject(java.lang.String project_name)

getProject

public java.lang.String getProject()

setMetricAndDirection

public void setMetricAndDirection(java.lang.String metric,
                                  java.lang.String[] otherMetrics,
                                  boolean sortDecreasing)

setMetricAndDirection

public void setMetricAndDirection(java.lang.String metric,
                                  boolean sortDecreasing)

setDefaultMetricAndDirection

public void setDefaultMetricAndDirection(hex.Model m)

addModels

public final void addModels(water.Key<hex.Model>[] newModels)

Add the given models to the leaderboard. Note that to make this easier to use from Grid, which returns its models in random order, we allow the caller to add the same model multiple times and we eliminate the duplicates here.

Parameters:

newModels -

addModel

public void addModel(water.Key<hex.Model> key)

addModel

public void addModel(hex.Model model)

getModelKeys

public water.Key<hex.Model>[] getModelKeys()

Returns:

list of keys of models sorted by the default metric for the model category, fetched from the DKV

modelKeys

public water.Key<hex.Model>[] modelKeys(java.lang.String metric,
                                        boolean sortDecreasing)

Returns:

list of keys of models sorted by the given metric, fetched from the DKV

getModels

public hex.Model[] getModels()

Returns:

list of models sorted by the default metric for the model category

getModels

public hex.Model[] getModels(java.lang.String metric,
                             boolean sortDecreasing)

Returns:

list of models sorted by the given metric

getLeader

public hex.Model getLeader()

getModelCount

public int getModelCount()

Return the number of models in this Leaderboard.

getSortMetrics

public double[] getSortMetrics()

getOtherMetrics

public static double[] getOtherMetrics(java.lang.String other_metric,
                                       water.util.IcedHashMap<water.Key<hex.ModelMetrics>,hex.ModelMetrics> leaderboard_set_metrics,
                                       water.fvec.Frame leaderboardFrame,
                                       hex.Model[] models)

getSortMetrics

public static double[] getSortMetrics(java.lang.String sort_metric,
                                      water.util.IcedHashMap<water.Key<hex.ModelMetrics>,hex.ModelMetrics> leaderboard_set_metrics,
                                      water.fvec.Frame leaderboardFrame,
                                      hex.Model[] models)

delete

public void delete()

Delete everything in the DKV that this points to. We currently need to be able to call this after deleteWithChildren().

deleteWithChildren

public void deleteWithChildren()

defaultMetricForModel

public static double[] defaultMetricForModel(hex.Model m)

defaultMetricForModel

public static double[] defaultMetricForModel(hex.Model m,
                                             hex.ModelMetrics mm)

defaultMetricNameForModel

public static java.lang.String[] defaultMetricNameForModel(hex.Model m)

rankTsv

public java.lang.String rankTsv()

timeTsv

public java.lang.String timeTsv()

colHeaders

protected static final java.lang.String[] colHeaders(java.lang.String metric,
                                                     java.lang.String[] other_metric)

colHeadersMult

protected static final java.lang.String[] colHeadersMult(java.lang.String metric)

makeTwoDimTable

public static final water.util.TwoDimTable makeTwoDimTable(java.lang.String tableHeader,
                                                           java.lang.String sort_metric,
                                                           java.lang.String[] other_metric,
                                                           int length)

addTwoDimTableRowMultinomial

public void addTwoDimTableRowMultinomial(water.util.TwoDimTable table,
                                         int row,
                                         java.lang.String[] modelIDs,
                                         double[] errors)

addTwoDimTableRowBinomial

public void addTwoDimTableRowBinomial(water.util.TwoDimTable table,
                                      int row,
                                      java.lang.String[] modelIDs,
                                      double[] errors,
                                      double[] otherErrors)

addTwoDimTableRowRegression

public void addTwoDimTableRowRegression(water.util.TwoDimTable table,
                                        int row,
                                        java.lang.String[] modelIDs,
                                        double[] errors,
                                        double[] rmse,
                                        double[] mae,
                                        double[] rmsle)

toTwoDimTable

public water.util.TwoDimTable toTwoDimTable()

toTwoDimTable

public water.util.TwoDimTable toTwoDimTable(java.lang.String tableHeader,
                                            boolean leftJustifyModelIds)

toString

public static java.lang.String toString(java.lang.String project_name,
                                        hex.Model[] models,
                                        java.lang.String fieldSeparator,
                                        java.lang.String lineSeparator,
                                        boolean includeTitle,
                                        boolean includeHeader)

toString

public java.lang.String toString(java.lang.String fieldSeparator,
                                 java.lang.String lineSeparator)

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object