weka.knowledgeflow.steps

Class ClassifierPerformanceEvaluator

java.lang.Object

weka.knowledgeflow.steps.BaseStep

weka.knowledgeflow.steps.ClassifierPerformanceEvaluator

All Implemented Interfaces:

java.io.Serializable, BaseStepExtender, Step

@KFStep(name="ClassifierPerformanceEvaluator",
        category="Evaluation",
        toolTipText="Evaluates batch classifiers",
        iconPath="weka/gui/knowledgeflow/icons/ClassifierPerformanceEvaluator.gif")
public class ClassifierPerformanceEvaluator
extends BaseStep

Step that implements batch classifier evaluation

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	ClassifierPerformanceEvaluator.AggregateableClassifierErrorsPlotInstances Subclass of ClassifierErrorsPlotInstances to allow plot point sizes to be scaled according to global min/max values.
protected class	ClassifierPerformanceEvaluator.EvaluationCallback Callback that gets notified when an evaluation task completes.
protected static class	ClassifierPerformanceEvaluator.EvaluationTask Class that performs the actual evaluation of a set/fold

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	m_costSensitiveEval True to perform cost sensitive evaluation
protected java.lang.String	m_costString The cost matrix (string form)
protected boolean	m_errorPlotPointSizeProportionalToMargin True if plot point sizes are to be rendered proportional to the size of the prediction margin
protected boolean	m_isReset True if the step has been reset
protected CostMatrix	m_matrix The cost matrix
protected int	m_maxSetNum The maximum set number in the batch of sets being processed
protected java.util.List<java.lang.String>	m_metricsList Holds a list of metric names
protected java.lang.String	m_selectedEvalMetrics Evaluation metrics to output
protected java.util.concurrent.atomic.AtomicInteger	m_setsToGo For counting down the sets left to process
protected java.util.concurrent.atomic.AtomicInteger	m_taskCount

Fields inherited from class weka.knowledgeflow.steps.BaseStep

m_stepIsResourceIntensive, m_stepManager, m_stepName

Constructor Summary

Constructors

Constructor and Description
ClassifierPerformanceEvaluator() Constructor

Method Summary

Modifier and Type	Method and Description
protected static Evaluation	adjustForInputMappedClassifier(Evaluation eval, Classifier classifier, Instances inst, ClassifierErrorsPlotInstances plotInstances, CostMatrix matrix) Adjust evaluation configuration if an InputMappedClassifier is being used
protected void	aggregateEvalTask(Evaluation eval, Classifier classifier, Instances testData, ClassifierErrorsPlotInstances plotInstances, int setNum, java.lang.String evalLabel) Aggregates a single evaluation task into the overall evaluation
java.lang.String	getCostMatrixString() Get the cost matrix to use as a string
java.lang.String	getCustomEditorForStep() Return the fully qualified name of a custom editor component (JComponent) to use for editing the properties of the step.
boolean	getErrorPlotPointSizeProportionalToMargin() Get whether the size of plot data points will be proportional to the prediction margin
boolean	getEvaluateWithRespectToCosts() Get whether to evaluate with respoect to costs
java.lang.String	getEvaluationMetricsToOutput() Get the evaluation metrics to output (as a comma-separated list).
java.util.List<java.lang.String>	getIncomingConnectionTypes() Get a list of incoming connection types that this step can accept.
java.util.List<java.lang.String>	getOutgoingConnectionTypes() Get a list of outgoing connection types that this step can produce.
void	processIncoming(Data data) Process an incoming data payload (if the step accepts incoming connections)
void	setCostMatrixString(java.lang.String cms) Set the cost matrix to use as a string
void	setErrorPlotPointSizeProportionalToMargin(boolean e) Set whether the size of plot data points will be proportional to the prediction margin
void	setEvaluateWithRespectToCosts(boolean useCosts) Set whether to evaluate with respoect to costs
void	setEvaluationMetricsToOutput(java.lang.String m) Set the evaluation metrics to output (as a comma-separated list).
void	stepInit() Initialize the step.
void	stop() Request that processing be stopped.
protected void	stringToList(java.lang.String l)

Methods inherited from class weka.knowledgeflow.steps.BaseStep

environmentSubstitute, getDefaultSettings, getInteractiveViewers, getInteractiveViewersImpls, getName, getStepManager, globalInfo, isResourceIntensive, isStopRequested, outputStructureForConnectionType, setName, setStepIsResourceIntensive, setStepManager, setStepMustRunSingleThreaded, start, stepMustRunSingleThreaded

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

m_errorPlotPointSizeProportionalToMargin

protected boolean m_errorPlotPointSizeProportionalToMargin

True if plot point sizes are to be rendered proportional to the size of the prediction margin

m_costSensitiveEval

protected boolean m_costSensitiveEval

True to perform cost sensitive evaluation

m_costString

protected java.lang.String m_costString

The cost matrix (string form)

m_matrix

protected CostMatrix m_matrix

The cost matrix

m_selectedEvalMetrics

protected java.lang.String m_selectedEvalMetrics

Evaluation metrics to output

m_metricsList

protected java.util.List<java.lang.String> m_metricsList

Holds a list of metric names

m_isReset

protected boolean m_isReset

True if the step has been reset

m_setsToGo

protected java.util.concurrent.atomic.AtomicInteger m_setsToGo

For counting down the sets left to process

m_maxSetNum

protected int m_maxSetNum

The maximum set number in the batch of sets being processed

m_taskCount

protected java.util.concurrent.atomic.AtomicInteger m_taskCount

Constructor Detail

ClassifierPerformanceEvaluator

public ClassifierPerformanceEvaluator()

Constructor

Method Detail

stringToList

protected void stringToList(java.lang.String l)

getErrorPlotPointSizeProportionalToMargin

@OptionMetadata(displayName="Error plot point size proportional to margin",
                description="Set the point size proportional to the prediction margin for classification error plots")
public boolean getErrorPlotPointSizeProportionalToMargin()

Get whether the size of plot data points will be proportional to the prediction margin

Returns:

true if plot data points will be rendered proportional to the size of the prediction margin

setErrorPlotPointSizeProportionalToMargin

public void setErrorPlotPointSizeProportionalToMargin(boolean e)

Set whether the size of plot data points will be proportional to the prediction margin

Parameters:

e - true if plot data points will be rendered proportional to the size of the prediction margin

getEvaluationMetricsToOutput

@ProgrammaticProperty
public java.lang.String getEvaluationMetricsToOutput()

Get the evaluation metrics to output (as a comma-separated list).

Returns:

the evaluation metrics to output

setEvaluationMetricsToOutput

public void setEvaluationMetricsToOutput(java.lang.String m)

Set the evaluation metrics to output (as a comma-separated list).

Parameters:

m - the evaluation metrics to output

setEvaluateWithRespectToCosts

@ProgrammaticProperty
public void setEvaluateWithRespectToCosts(boolean useCosts)

Set whether to evaluate with respoect to costs

Parameters:

useCosts - true to use cost-sensitive evaluation

getEvaluateWithRespectToCosts

public boolean getEvaluateWithRespectToCosts()

Get whether to evaluate with respoect to costs

Returns:

true to use cost-sensitive evaluation

setCostMatrixString

@ProgrammaticProperty
public void setCostMatrixString(java.lang.String cms)

Set the cost matrix to use as a string

Parameters:

cms - the cost matrix to use

getCostMatrixString

public java.lang.String getCostMatrixString()

Get the cost matrix to use as a string

Returns:

the cost matrix

getIncomingConnectionTypes

public java.util.List<java.lang.String> getIncomingConnectionTypes()

Get a list of incoming connection types that this step can accept. Ideally (and if appropriate), this should take into account the state of the step and any existing incoming connections. E.g. a step might be able to accept one (and only one) incoming batch data connection.

Returns:

a list of incoming connections that this step can accept given its current state

getOutgoingConnectionTypes

public java.util.List<java.lang.String> getOutgoingConnectionTypes()

Get a list of outgoing connection types that this step can produce. Ideally (and if appropriate), this should take into account the state of the step and the incoming connections. E.g. depending on what incoming connection is present, a step might be able to produce a trainingSet output, a testSet output or neither, but not both.

Returns:

a list of outgoing connections that this step can produce

stepInit

public void stepInit()
              throws WekaException

Description copied from interface: Step

Initialize the step.

Throws:

WekaException - if a problem occurs during initialization

stop

public void stop()

Description copied from class: BaseStep

Request that processing be stopped. Subclasses should call isStopRequested() periodically to see if they should stop processing.

Specified by:

stop in interface Step

Overrides:

stop in class BaseStep

processIncoming

public void processIncoming(Data data)
                     throws WekaException

Process an incoming data payload (if the step accepts incoming connections)

Specified by:

processIncoming in interface BaseStepExtender

Specified by:

processIncoming in interface Step

Overrides:

processIncoming in class BaseStep

Parameters:

data - the payload to process

Throws:

WekaException - if a problem occurs

aggregateEvalTask

protected void aggregateEvalTask(Evaluation eval,
                                 Classifier classifier,
                                 Instances testData,
                                 ClassifierErrorsPlotInstances plotInstances,
                                 int setNum,
                                 java.lang.String evalLabel)
                          throws java.lang.Exception

Aggregates a single evaluation task into the overall evaluation

Parameters:

eval - the partial evaluation to aggregate

classifier - the classifier used for evaluation

testData - the test data evaluated on

plotInstances - plotting instances

setNum - the set number processed

evalLabel - evaluation type

Throws:

java.lang.Exception - if a problem occurs

getCustomEditorForStep

public java.lang.String getCustomEditorForStep()

Return the fully qualified name of a custom editor component (JComponent) to use for editing the properties of the step. This method can return null, in which case the system will dynamically generate an editor using the GenericObjectEditor

Specified by:

getCustomEditorForStep in interface Step

Overrides:

getCustomEditorForStep in class BaseStep

Returns:

the fully qualified name of a step editor component

adjustForInputMappedClassifier

protected static Evaluation adjustForInputMappedClassifier(Evaluation eval,
                                                           Classifier classifier,
                                                           Instances inst,
                                                           ClassifierErrorsPlotInstances plotInstances,
                                                           CostMatrix matrix)
                                                    throws java.lang.Exception

Adjust evaluation configuration if an InputMappedClassifier is being used

Parameters:

eval - the evaluation object ot adjust

classifier - the classifier being used

inst - the instances being evaluated on

plotInstances - plotting instances

matrix - the CostMatrix to use, or null for no cost-sensitive evaluation

Returns:

the adjusted Evaluation object

Throws:

java.lang.Exception - if a problem occurs