weka.clusterers

Class CheckClusterer

java.lang.Object

weka.core.Check

weka.core.CheckScheme

weka.clusterers.CheckClusterer

All Implemented Interfaces:

OptionHandler, RevisionHandler

public class CheckClusterer
extends CheckScheme

Class for examining the capabilities and finding problems with clusterers. If you implement a clusterer using the WEKA.libraries, you should run the checks on it to ensure robustness and correct operation. Passing all the tests of this object does not mean bugs in the clusterer don't exist, but this will help find some common ones.

Typical usage:

java weka.clusterers.CheckClusterer -W clusterer_name -- clusterer_options

CheckClusterer reports on the following:

• Clusterer abilities
  • Possible command line options to the clusterer
  • Whether the clusterer can predict nominal, numeric, string, date or relational class attributes.
  • Whether the clusterer can handle numeric predictor attributes
  • Whether the clusterer can handle nominal predictor attributes
  • Whether the clusterer can handle string predictor attributes
  • Whether the clusterer can handle date predictor attributes
  • Whether the clusterer can handle relational predictor attributes
  • Whether the clusterer can handle multi-instance data
  • Whether the clusterer can handle missing predictor values
  • Whether the clusterer can handle instance weights
• Correct functioning
  • Correct initialisation during buildClusterer (i.e. no result changes when buildClusterer called repeatedly)
  • Whether the clusterer alters the data pased to it (number of instances, instance order, instance weights, etc)
• Degenerate cases
  • building clusterer with zero training instances
  • all but one predictor attribute values missing
  • all predictor attribute values missing
  • all but one class values missing
  • all class values missing

Running CheckClusterer with the debug option set will output the training dataset for any failed tests.

The weka.clusterers.AbstractClustererTest uses this class to test all the clusterers. Any changes here, have to be checked in that abstract test class, too.

Valid options are:

 -D
  Turn on debugging output.
 

 -S
  Silent mode - prints nothing to stdout.
 

 -N <num>
  The number of instances in the datasets (default 20).
 

 -nominal <num>
  The number of nominal attributes (default 2).
 

 -nominal-values <num>
  The number of values for nominal attributes (default 1).
 

 -numeric <num>
  The number of numeric attributes (default 1).
 

 -string <num>
  The number of string attributes (default 1).
 

 -date <num>
  The number of date attributes (default 1).
 

 -relational <num>
  The number of relational attributes (default 1).
 

 -num-instances-relational <num>
  The number of instances in relational/bag attributes (default 10).
 

 -words <comma-separated-list>
  The words to use in string attributes.
 

 -word-separators <chars>
  The word separators to use in string attributes.
 

 -W
  Full name of the clusterer analyzed.
  eg: weka.clusterers.SimpleKMeans
  (default weka.clusterers.SimpleKMeans)
 

 Options specific to clusterer weka.clusterers.SimpleKMeans:
 

 -N <num>
  number of clusters.
  (default 2).
 

 -V
  Display std. deviations for centroids.
 

 -M
  Replace missing values with mean/mode.
 

 -S <num>
  Random number seed.
  (default 10)
 

Options after -- are passed to the designated clusterer.

See Also:

TestInstances

Nested Class Summary

Nested classes/interfaces inherited from class weka.core.CheckScheme

CheckScheme.PostProcessor

Field Summary

Fields

Modifier and Type	Field and Description
protected Clusterer	m_Clusterer The clusterer to be examined

Fields inherited from class weka.core.CheckScheme

m_ClasspathProblems, m_NumDate, m_NumInstances, m_NumInstancesRelational, m_NumNominal, m_NumNumeric, m_NumRelational, m_NumString, m_PostProcessor, m_Words, m_WordSeparators

Fields inherited from class weka.core.Check

m_Debug, m_Silent

Constructor Summary

Constructors

Constructor and Description
CheckClusterer() default constructor

Method Summary

Modifier and Type	Method and Description
protected void	addMissing(Instances data, int level, boolean predictorMissing) Add missing values to a dataset.
protected boolean[]	canHandleMissing(boolean nominalPredictor, boolean numericPredictor, boolean stringPredictor, boolean datePredictor, boolean relationalPredictor, boolean multiInstance, boolean predictorMissing, int missingLevel) Checks basic missing value handling of the scheme.
protected boolean[]	canHandleZeroTraining(boolean nominalPredictor, boolean numericPredictor, boolean stringPredictor, boolean datePredictor, boolean relationalPredictor, boolean multiInstance) Checks whether the scheme can handle zero training instances.
protected boolean[]	canPredict(boolean nominalPredictor, boolean numericPredictor, boolean stringPredictor, boolean datePredictor, boolean relationalPredictor, boolean multiInstance) Checks basic prediction of the scheme, for simple non-troublesome datasets.
protected boolean[]	canTakeOptions() Checks whether the scheme can take command line options.
protected boolean[]	correctBuildInitialisation(boolean nominalPredictor, boolean numericPredictor, boolean stringPredictor, boolean datePredictor, boolean relationalPredictor, boolean multiInstance) Checks whether the scheme correctly initialises models when buildClusterer is called.
protected boolean[]	datasetIntegrity(boolean nominalPredictor, boolean numericPredictor, boolean stringPredictor, boolean datePredictor, boolean relationalPredictor, boolean multiInstance, boolean predictorMissing) Checks whether the scheme alters the training dataset during training.
protected boolean[]	declaresSerialVersionUID() tests for a serialVersionUID.
void	doTests() Begin the tests, reporting results to System.out
Clusterer	getClusterer() Get the clusterer used as the clusterer
java.lang.String[]	getOptions() Gets the current settings of the CheckClusterer.
java.lang.String	getRevision() Returns the revision string.
protected boolean[]	instanceWeights(boolean nominalPredictor, boolean numericPredictor, boolean stringPredictor, boolean datePredictor, boolean relationalPredictor, boolean multiInstance) Checks whether the clusterer can handle instance weights.
java.util.Enumeration<Option>	listOptions() Returns an enumeration describing the available options.
static void	main(java.lang.String[] args) Test method for this class
protected Instances	makeTestDataset(int seed, int numInstances, int numNominal, int numNumeric, int numString, int numDate, int numRelational, boolean multiInstance) Make a simple set of instances with variable position of the class attribute, which can later be modified for use in specific tests.
protected boolean[]	multiInstanceHandler() Checks whether the scheme handles multi-instance data.
protected void	printAttributeSummary(boolean nominalPredictor, boolean numericPredictor, boolean stringPredictor, boolean datePredictor, boolean relationalPredictor, boolean multiInstance) Print out a short summary string for the dataset characteristics
protected boolean[]	runBasicTest(boolean nominalPredictor, boolean numericPredictor, boolean stringPredictor, boolean datePredictor, boolean relationalPredictor, boolean multiInstance, int missingLevel, boolean predictorMissing, int numTrain, java.util.Vector<java.lang.String> accepts) Runs a text on the datasets with the given characteristics.
protected void	runTests(boolean weighted, boolean multiInstance, boolean updateable) Run a battery of tests
void	setClusterer(Clusterer newClusterer) Set the clusterer for testing.
void	setOptions(java.lang.String[] options) Parses a given list of options.
protected boolean[]	updateableClusterer() Checks whether the scheme can build models incrementally.
protected boolean[]	updatingEquality(boolean nominalPredictor, boolean numericPredictor, boolean stringPredictor, boolean datePredictor, boolean relationalPredictor, boolean multiInstance) Checks whether an updateable scheme produces the same model when trained incrementally as when batch trained.
protected boolean[]	weightedInstancesHandler() Checks whether the scheme says it can handle instance weights.

Methods inherited from class weka.core.CheckScheme

addMissing, arrayToList, attributeTypeToString, compareDatasets, getNumDate, getNumInstances, getNumInstancesRelational, getNumNominal, getNumNumeric, getNumRelational, getNumString, getPostProcessor, getWords, getWordSeparators, hasClasspathProblems, listToArray, process, setNumDate, setNumInstances, setNumInstancesRelational, setNumNominal, setNumNumeric, setNumRelational, setNumString, setPostProcessor, setWords, setWordSeparators

Methods inherited from class weka.core.Check

forName, getDebug, getSilent, print, println, println, runCheck, setDebug, setSilent

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

m_Clusterer

protected Clusterer m_Clusterer

The clusterer to be examined

Constructor Detail

CheckClusterer

public CheckClusterer()

default constructor

Method Detail

listOptions

public java.util.Enumeration<Option> listOptions()

Returns an enumeration describing the available options.

Specified by:

listOptions in interface OptionHandler

Overrides:

listOptions in class CheckScheme

Returns:

an enumeration of all the available options.

setOptions

public void setOptions(java.lang.String[] options)
                throws java.lang.Exception

Parses a given list of options.

Valid options are:

 -D
  Turn on debugging output.
 

 -S
  Silent mode - prints nothing to stdout.
 

 -N <num>
  The number of instances in the datasets (default 20).
 

 -nominal <num>
  The number of nominal attributes (default 2).
 

 -nominal-values <num>
  The number of values for nominal attributes (default 1).
 

 -numeric <num>
  The number of numeric attributes (default 1).
 

 -string <num>
  The number of string attributes (default 1).
 

 -date <num>
  The number of date attributes (default 1).
 

 -relational <num>
  The number of relational attributes (default 1).
 

 -num-instances-relational <num>
  The number of instances in relational/bag attributes (default 10).
 

 -words <comma-separated-list>
  The words to use in string attributes.
 

 -word-separators <chars>
  The word separators to use in string attributes.
 

 -W
  Full name of the clusterer analyzed.
  eg: weka.clusterers.SimpleKMeans
  (default weka.clusterers.SimpleKMeans)
 

 Options specific to clusterer weka.clusterers.SimpleKMeans:
 

 -N <num>
  number of clusters.
  (default 2).
 

 -V
  Display std. deviations for centroids.
 

 -M
  Replace missing values with mean/mode.
 

 -S <num>
  Random number seed.
  (default 10)
 

Specified by:

setOptions in interface OptionHandler

Overrides:

setOptions in class CheckScheme

Parameters:

options - the list of options as an array of strings

Throws:

java.lang.Exception - if an option is not supported

getOptions

public java.lang.String[] getOptions()

Gets the current settings of the CheckClusterer.

Specified by:

getOptions in interface OptionHandler

Overrides:

getOptions in class CheckScheme

Returns:

an array of strings suitable for passing to setOptions

doTests

public void doTests()

Begin the tests, reporting results to System.out

Specified by:

doTests in class CheckScheme

setClusterer

public void setClusterer(Clusterer newClusterer)

Set the clusterer for testing.

Parameters:

newClusterer - the Clusterer to use.

getClusterer

public Clusterer getClusterer()

Get the clusterer used as the clusterer

Returns:

the clusterer used as the clusterer

runTests

protected void runTests(boolean weighted,
                        boolean multiInstance,
                        boolean updateable)

Run a battery of tests

Parameters:

weighted - true if the clusterer says it handles weights

multiInstance - true if the clusterer is a multi-instance clusterer

updateable - true if the classifier is updateable

canTakeOptions

protected boolean[] canTakeOptions()

Checks whether the scheme can take command line options.

Returns:

index 0 is true if the clusterer can take options

updateableClusterer

protected boolean[] updateableClusterer()

Checks whether the scheme can build models incrementally.

Returns:

index 0 is true if the clusterer can train incrementally

weightedInstancesHandler

protected boolean[] weightedInstancesHandler()

Checks whether the scheme says it can handle instance weights.

Returns:

true if the clusterer handles instance weights

multiInstanceHandler

protected boolean[] multiInstanceHandler()

Checks whether the scheme handles multi-instance data.

Returns:

true if the clusterer handles multi-instance data

declaresSerialVersionUID

protected boolean[] declaresSerialVersionUID()

tests for a serialVersionUID. Fails in case the scheme doesn't declare a UID.

Returns:

index 0 is true if the scheme declares a UID

canPredict

protected boolean[] canPredict(boolean nominalPredictor,
                               boolean numericPredictor,
                               boolean stringPredictor,
                               boolean datePredictor,
                               boolean relationalPredictor,
                               boolean multiInstance)

Checks basic prediction of the scheme, for simple non-troublesome datasets.

Parameters:

nominalPredictor - if true use nominal predictor attributes

numericPredictor - if true use numeric predictor attributes

stringPredictor - if true use string predictor attributes

datePredictor - if true use date predictor attributes

relationalPredictor - if true use relational predictor attributes

multiInstance - whether multi-instance is needed

Returns:

index 0 is true if the test was passed, index 1 is true if test was acceptable

canHandleZeroTraining

protected boolean[] canHandleZeroTraining(boolean nominalPredictor,
                                          boolean numericPredictor,
                                          boolean stringPredictor,
                                          boolean datePredictor,
                                          boolean relationalPredictor,
                                          boolean multiInstance)

Checks whether the scheme can handle zero training instances.

Parameters:

nominalPredictor - if true use nominal predictor attributes

numericPredictor - if true use numeric predictor attributes

stringPredictor - if true use string predictor attributes

datePredictor - if true use date predictor attributes

relationalPredictor - if true use relational predictor attributes

multiInstance - whether multi-instance is needed

Returns:

index 0 is true if the test was passed, index 1 is true if test was acceptable

correctBuildInitialisation

protected boolean[] correctBuildInitialisation(boolean nominalPredictor,
                                               boolean numericPredictor,
                                               boolean stringPredictor,
                                               boolean datePredictor,
                                               boolean relationalPredictor,
                                               boolean multiInstance)

Checks whether the scheme correctly initialises models when buildClusterer is called. This test calls buildClusterer with one training dataset. buildClusterer is then called on a training set with different structure, and then again with the original training set. If the equals method of the ClusterEvaluation class returns false, this is noted as incorrect build initialisation.

Parameters:

nominalPredictor - if true use nominal predictor attributes

numericPredictor - if true use numeric predictor attributes

stringPredictor - if true use string predictor attributes

datePredictor - if true use date predictor attributes

relationalPredictor - if true use relational predictor attributes

multiInstance - whether multi-instance is needed

Returns:

index 0 is true if the test was passed

canHandleMissing

protected boolean[] canHandleMissing(boolean nominalPredictor,
                                     boolean numericPredictor,
                                     boolean stringPredictor,
                                     boolean datePredictor,
                                     boolean relationalPredictor,
                                     boolean multiInstance,
                                     boolean predictorMissing,
                                     int missingLevel)

Checks basic missing value handling of the scheme. If the missing values cause an exception to be thrown by the scheme, this will be recorded.

Parameters:

nominalPredictor - if true use nominal predictor attributes

numericPredictor - if true use numeric predictor attributes

stringPredictor - if true use string predictor attributes

datePredictor - if true use date predictor attributes

relationalPredictor - if true use relational predictor attributes

multiInstance - whether multi-instance is needed

predictorMissing - true if the missing values may be in the predictors

missingLevel - the percentage of missing values

Returns:

index 0 is true if the test was passed, index 1 is true if test was acceptable

instanceWeights

protected boolean[] instanceWeights(boolean nominalPredictor,
                                    boolean numericPredictor,
                                    boolean stringPredictor,
                                    boolean datePredictor,
                                    boolean relationalPredictor,
                                    boolean multiInstance)

Checks whether the clusterer can handle instance weights. This test compares the clusterer performance on two datasets that are identical except for the training weights. If the results change, then the clusterer must be using the weights. It may be possible to get a false positive from this test if the weight changes aren't significant enough to induce a change in clusterer performance (but the weights are chosen to minimize the likelihood of this).

Parameters:

nominalPredictor - if true use nominal predictor attributes

numericPredictor - if true use numeric predictor attributes

stringPredictor - if true use string predictor attributes

datePredictor - if true use date predictor attributes

relationalPredictor - if true use relational predictor attributes

multiInstance - whether multi-instance is needed

Returns:

index 0 true if the test was passed

datasetIntegrity

protected boolean[] datasetIntegrity(boolean nominalPredictor,
                                     boolean numericPredictor,
                                     boolean stringPredictor,
                                     boolean datePredictor,
                                     boolean relationalPredictor,
                                     boolean multiInstance,
                                     boolean predictorMissing)

Checks whether the scheme alters the training dataset during training. If the scheme needs to modify the training data it should take a copy of the training data. Currently checks for changes to header structure, number of instances, order of instances, instance weights.

Parameters:

nominalPredictor - if true use nominal predictor attributes

numericPredictor - if true use numeric predictor attributes

stringPredictor - if true use string predictor attributes

datePredictor - if true use date predictor attributes

relationalPredictor - if true use relational predictor attributes

multiInstance - whether multi-instance is needed

predictorMissing - true if we know the clusterer can handle (at least) moderate missing predictor values

Returns:

index 0 is true if the test was passed

updatingEquality

protected boolean[] updatingEquality(boolean nominalPredictor,
                                     boolean numericPredictor,
                                     boolean stringPredictor,
                                     boolean datePredictor,
                                     boolean relationalPredictor,
                                     boolean multiInstance)

Checks whether an updateable scheme produces the same model when trained incrementally as when batch trained. The model itself cannot be compared, so we compare the evaluation on test data for both models. It is possible to get a false positive on this test (likelihood depends on the classifier).

Parameters:

nominalPredictor - if true use nominal predictor attributes

numericPredictor - if true use numeric predictor attributes

stringPredictor - if true use string predictor attributes

datePredictor - if true use date predictor attributes

relationalPredictor - if true use relational predictor attributes

multiInstance - whether multi-instance is needed

Returns:

index 0 is true if the test was passed

runBasicTest

protected boolean[] runBasicTest(boolean nominalPredictor,
                                 boolean numericPredictor,
                                 boolean stringPredictor,
                                 boolean datePredictor,
                                 boolean relationalPredictor,
                                 boolean multiInstance,
                                 int missingLevel,
                                 boolean predictorMissing,
                                 int numTrain,
                                 java.util.Vector<java.lang.String> accepts)

Runs a text on the datasets with the given characteristics.

Parameters:

nominalPredictor - if true use nominal predictor attributes

numericPredictor - if true use numeric predictor attributes

stringPredictor - if true use string predictor attributes

datePredictor - if true use date predictor attributes

relationalPredictor - if true use relational predictor attributes

multiInstance - whether multi-instance is needed

missingLevel - the percentage of missing values

predictorMissing - true if the missing values may be in the predictors

numTrain - the number of instances in the training set

accepts - the acceptable string in an exception

Returns:

index 0 is true if the test was passed, index 1 is true if test was acceptable

addMissing

protected void addMissing(Instances data,
                          int level,
                          boolean predictorMissing)

Add missing values to a dataset.

Parameters:

data - the instances to add missing values to

level - the level of missing values to add (if positive, this is the probability that a value will be set to missing, if negative all but one value will be set to missing (not yet implemented))

predictorMissing - if true, predictor attributes will be modified

makeTestDataset

protected Instances makeTestDataset(int seed,
                                    int numInstances,
                                    int numNominal,
                                    int numNumeric,
                                    int numString,
                                    int numDate,
                                    int numRelational,
                                    boolean multiInstance)
                             throws java.lang.Exception

Make a simple set of instances with variable position of the class attribute, which can later be modified for use in specific tests.

Parameters:

seed - the random number seed

numInstances - the number of instances to generate

numNominal - the number of nominal attributes

numNumeric - the number of numeric attributes

numString - the number of string attributes

numDate - the number of date attributes

numRelational - the number of relational attributes

multiInstance - whether the dataset should a multi-instance dataset

Returns:

the test dataset

Throws:

java.lang.Exception - if the dataset couldn't be generated

See Also:

TestInstances.CLASS_IS_LAST

printAttributeSummary

protected void printAttributeSummary(boolean nominalPredictor,
                                     boolean numericPredictor,
                                     boolean stringPredictor,
                                     boolean datePredictor,
                                     boolean relationalPredictor,
                                     boolean multiInstance)

Print out a short summary string for the dataset characteristics

Parameters:

nominalPredictor - true if nominal predictor attributes are present

numericPredictor - true if numeric predictor attributes are present

stringPredictor - true if string predictor attributes are present

datePredictor - true if date predictor attributes are present

relationalPredictor - true if relational predictor attributes are present

multiInstance - whether multi-instance is needed

getRevision

public java.lang.String getRevision()

Returns the revision string.

Returns:

the revision

main

public static void main(java.lang.String[] args)

Test method for this class

Parameters:

args - the commandline options