ai.h2o.automl.leaderboard

Class Leaderboard

java.lang.Object

water.Iced<T>

water.Keyed<T>

water.Lockable<Leaderboard>

ai.h2o.automl.leaderboard.Leaderboard

All Implemented Interfaces:

hex.ModelContainer<hex.Model>, java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, water.Freezable<Leaderboard>

public class Leaderboard
extends water.Lockable<Leaderboard>
implements hex.ModelContainer<hex.Model>

Utility to track all the models built for a given dataset type.

Note that if a new Leaderboard is made for the same project_name it'll keep using the old model list, which allows us to run AutoML multiple times and keep adding to the leaderboard.

The models are returned sorted by either an appropriate default metric for the model category (auc, mean per class error, or mean residual deviance), or by a metric that's set via #setMetricAndDirection.

TODO: make this robust against removal of models from the DKV.

See Also:

Serialized Form

Field Summary

Fields inherited from class water.Lockable

_lockers

Fields inherited from class water.Keyed

_key

Constructor Summary

Constructors

Constructor and Description
Leaderboard(java.lang.String project_name, EventLog eventLog, water.fvec.Frame leaderboardFrame, java.lang.String sort_metric) Constructs a new leaderboard (doesn't put it in DKV).

Method Summary

Modifier and Type	Method and Description
<M extends hex.Model> void	addModel(water.Key<M> key)
void	addModels(water.Key<hex.Model>[] modelKeys) Add the given models to the leaderboard.
hex.Model	getLeader()
java.lang.String[]	getMetrics() The sort metric is always the first element in the list of metrics.
int	getModelCount() Return the number of models in this Leaderboard.
water.Key<hex.Model>[]	getModelKeys()
int	getModelRank(water.Key<hex.Model> modelKey)
hex.Model[]	getModels()
hex.Model[]	getModelsSortedByMetric(java.lang.String metric)
static Leaderboard	getOrMake(java.lang.String project_name, EventLog eventLog, water.fvec.Frame leaderboardFrame, java.lang.String sort_metric) Retrieves a leaderboard from DKV or creates a fresh one and add it to DKV.
java.lang.String	getProject()
java.lang.String	getSortMetric() If no sort metric is provided when creating the leaderboard, then a default sort metric will be automatically chosen based on the problem type: binomial classification: auc multinomial classification: logloss regression: mean_residual_deviance
double[]	getSortMetricValues()
static java.lang.String	idForProject(java.lang.String project_name)
static boolean	isLossFunction(java.lang.String metric)
water.fvec.Frame	leaderboardFrame() Note: If no leaderboard was provided, then the models are sorted according to metrics obtained during training in the following priority order depending on availability: cross-validation metrics validation metrics training metrics
protected water.Futures	remove_impl(water.Futures fs, boolean cascade) Delete object and its dependencies from DKV, including models.
<M extends hex.Model> void	removeModel(water.Key<M> key, boolean cascade)
void	removeModels(water.Key<hex.Model>[] modelKeys, boolean cascade)
void	setExtensionsProvider(LeaderboardExtensionsProvider provider) Assign a LeaderboardExtensionsProvider to this leaderboard instance.
java.lang.String	toLogString()
java.lang.String	toString()
water.util.TwoDimTable	toTwoDimTable(java.lang.String... extensions) Creates a TwoDimTable representation of the leaderboard.

Methods inherited from class water.Lockable

delete_and_lock, delete_and_lock, delete_and_lock, delete, delete, delete, delete, read_lock, read_lock, read_lock, unlock_all, unlock, unlock, unlock, unlock, update, update, update, write_lock_to_read_lock, write_lock, write_lock, write_lock

Methods inherited from class water.Keyed

checksum_impl, checksum_impl, checksum, checksum, getKey, makeSchema, readAll_impl, readAll, remove_impl, remove_self_key_impl, remove, remove, remove, remove, remove, remove, removeQuietly, writeAll_impl, writeAll

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Leaderboard

public Leaderboard(java.lang.String project_name,
                   EventLog eventLog,
                   water.fvec.Frame leaderboardFrame,
                   java.lang.String sort_metric)

Constructs a new leaderboard (doesn't put it in DKV).

Parameters:

project_name -

eventLog -

leaderboardFrame -

sort_metric -

Method Detail

idForProject

public static java.lang.String idForProject(java.lang.String project_name)

Parameters:

project_name -

Returns:

a Leaderboard id for the project name

isLossFunction

public static boolean isLossFunction(java.lang.String metric)

Parameters:

metric -

Returns:

true iff the metric is a loss function

getOrMake

public static Leaderboard getOrMake(java.lang.String project_name,
                                    EventLog eventLog,
                                    water.fvec.Frame leaderboardFrame,
                                    java.lang.String sort_metric)

Retrieves a leaderboard from DKV or creates a fresh one and add it to DKV. Note that if the leaderboard is reused to add new models, we have to use the same leaderboard frame. IMPORTANT! if the leaderboard is created without leaderboardFrame, the models will be sorted according to their default metrics (in order of availability: cross-validation metrics, validation metrics, training metrics). Therefore, if some models were trained with/without cross-validation, or with different training or validation frames, then we can't guarantee the fairness of the leaderboard ranking.

Parameters:

project_name -

eventLog -

leaderboardFrame -

sort_metric -

Returns:

an existing leaderboard if there's already one in DKV for this project, or a new leaderboard added to DKV.

setExtensionsProvider

public void setExtensionsProvider(LeaderboardExtensionsProvider provider)

Assign a LeaderboardExtensionsProvider to this leaderboard instance.

Parameters:

provider - the provider used to generate the optional extension columns from the leaderboard.

See Also:

LeaderboardExtensionsProvider

getProject

public java.lang.String getProject()

getSortMetric

public java.lang.String getSortMetric()

If no sort metric is provided when creating the leaderboard, then a default sort metric will be automatically chosen based on the problem type:

binomial classification: auc

multinomial classification: logloss

regression: mean_residual_deviance

Returns:

the metric used to sort the models in the leaderboard.

getMetrics

public java.lang.String[] getMetrics()

The sort metric is always the first element in the list of metrics.

Returns:

the full list of metrics available in the leaderboard.

leaderboardFrame

public water.fvec.Frame leaderboardFrame()

Note: If no leaderboard was provided, then the models are sorted according to metrics obtained during training in the following priority order depending on availability:

cross-validation metrics

validation metrics

training metrics