InterquartileRange (interruptible-weka 0.1.4 API)

java.lang.Object
- weka.filters.Filter
- - weka.filters.SimpleFilter
  - - weka.filters.SimpleBatchFilter
    - - weka.filters.unsupervised.attribute.InterquartileRange

All Implemented Interfaces:: java.io.Serializable, CapabilitiesHandler, CapabilitiesIgnorer, CommandlineRunnable, OptionHandler, RevisionHandler, WeightedAttributesHandler

public class InterquartileRange
extends SimpleBatchFilter
implements WeightedAttributesHandler

A filter for detecting outliers and extreme values based on interquartile ranges. The filter skips the class attribute.

Outliers:
Q3 + OF*IQR < x <= Q3 + EVF*IQR
or
Q1 - EVF*IQR <= x < Q1 - OF*IQR

Extreme values:
x > Q3 + EVF*IQR
or
x < Q1 - EVF*IQR

Key:
Q1 = 25% quartile
Q3 = 75% quartile
IQR = Interquartile Range, difference between Q1 and Q3
OF = Outlier Factor
EVF = Extreme Value Factor

Valid options are:

 -D
  Turns on output of debugging information.

 -R <col1,col2-col4,...>
  Specifies list of columns to base outlier/extreme value detection
  on. If an instance is considered in at least one of those
  attributes an outlier/extreme value, it is tagged accordingly.
  'first' and 'last' are valid indexes.
  (default none)

 -O <num>
  The factor for outlier detection.
  (default: 3)

 -E <num>
  The factor for extreme values detection.
  (default: 2*Outlier Factor)

 -E-as-O
  Tags extreme values also as outliers.
  (default: off)

 -P
  Generates Outlier/ExtremeValue pair for each numeric attribute in
  the range, not just a single indicator pair for all the attributes.
  (default: off)

 -M
  Generates an additional attribute 'Offset' per Outlier/ExtremeValue
  pair that contains the multiplier that the value is off the median.
     value = median + 'multiplier' * IQR
 Note: implicitely sets '-P'. (default: off)

Thanks to Dale for a few brainstorming sessions.

See Also:: Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class InterquartileRange.ValueType
enum for obtaining the various determined IQR values.

Nested Classes
Modifier and Type	Class and Description
`static class`	`InterquartileRange.ValueType` enum for obtaining the various determined IQR values.

Field Summary

Fields
Modifier and Type	Field and Description
`protected int[]`	`m_AttributeIndices` the generated indices (only for performance reasons)
`protected Range`	`m_Attributes` the attribute range to work on
`protected boolean`	`m_DetectionPerAttribute` whether to generate Outlier/ExtremeValue attributes for each attribute instead of a general one
`protected boolean`	`m_ExtremeValuesAsOutliers` whether extreme values are also tagged as outliers
`protected double`	`m_ExtremeValuesFactor` the factor for detecting extreme values, by default 2*m_OutlierFactor
`protected double[]`	`m_IQR` the interquartile range
`protected double[]`	`m_LowerExtremeValue` the lower extreme value threshold (= Q1 - EVF*IQR)
`protected double[]`	`m_LowerOutlier` the lower outlier threshold (= Q1 - OF*IQR)
`protected double[]`	`m_Median` the median
`protected int[]`	`m_OutlierAttributePosition` the position of the outlier attribute
`protected double`	`m_OutlierFactor` the factor for detecting outliers
`protected boolean`	`m_OutputOffsetMultiplier` whether to add another attribute called "Offset", that lists the 'multiplier' by which the outlier/extreme value is away from the median, i.e., value = median + 'multiplier' * IQR automatically enables m_DetectionPerAttribute!
`protected double[]`	`m_UpperExtremeValue` the upper extreme value threshold (= Q3 + EVF*IQR)
`protected double[]`	`m_UpperOutlier` the upper outlier threshold (= Q3 + OF*IQR)
`static int`	`NON_NUMERIC` indicator for non-numeric attributes

Fields inherited from class weka.filters.Filter
m_Debug, m_DoNotCheckCapabilities, m_FirstBatchDone, m_InputRelAtts, m_InputStringAtts, m_NewBatch, m_OutputRelAtts, m_OutputStringAtts

Constructor Summary

Constructors
Constructor and Description

InterquartileRange()

Constructors
Constructor and Description
`InterquartileRange()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`java.lang.String`	`attributeIndicesTipText()` Returns the tip text for this property
`protected double`	`calculateMultiplier(Instance inst, int index)` returns the mulitplier of the IQR the instance is off the median for this particular attribute.
`protected void`	`computeThresholds(Instances instances)` computes the thresholds for outliers and extreme values
`java.lang.String`	`detectionPerAttributeTipText()` Returns the tip text for this property
`protected Instances`	`determineOutputFormat(Instances inputFormat)` Determines the output format based on the input format and returns this.
`java.lang.String`	`extremeValuesAsOutliersTipText()` Returns the tip text for this property
`java.lang.String`	`extremeValuesFactorTipText()` Returns the tip text for this property
`java.lang.String`	`getAttributeIndices()` Gets the current range selection
`Capabilities`	`getCapabilities()` Returns the Capabilities of this filter.
`boolean`	`getDetectionPerAttribute()` Gets whether an Outlier/ExtremeValue attribute pair is generated for each numeric attribute ("true") or just one pair for all numeric attributes together ("false").
`boolean`	`getExtremeValuesAsOutliers()` Get whether extreme values are also tagged as outliers.
`double`	`getExtremeValuesFactor()` Gets the factor for determining the thresholds for extreme values.
`java.lang.String[]`	`getOptions()` Gets the current settings of the filter.
`double`	`getOutlierFactor()` Gets the factor for determining the thresholds for outliers.
`boolean`	`getOutputOffsetMultiplier()` Gets whether an additional attribute "Offset" is generated per Outlier/ExtremeValue attribute pair that lists the multiplier the value is off the median: value = median + 'multiplier' * IQR.
`java.lang.String`	`getRevision()` Returns the revision string.
`double[]`	`getValues(InterquartileRange.ValueType type)` Returns the values for the specified type.
`java.lang.String`	`globalInfo()` Returns a string describing this filter
`protected boolean`	`isExtremeValue(Instance inst)` returns whether the instance is an extreme value or not
`protected boolean`	`isExtremeValue(Instance inst, int index)` returns whether the instance has an extreme value in the specified attribute or not
`protected boolean`	`isOutlier(Instance inst)` returns whether the instance is an outlier or not
`protected boolean`	`isOutlier(Instance inst, int index)` returns whether the instance has an outlier in the specified attribute or not
`java.util.Enumeration<Option>`	`listOptions()` Returns an enumeration describing the available options.
`static void`	`main(java.lang.String[] args)` Main method for testing this class.
`java.lang.String`	`outlierFactorTipText()` Returns the tip text for this property
`java.lang.String`	`outputOffsetMultiplierTipText()` Returns the tip text for this property
`protected Instances`	`process(Instances instances)` Processes the given data (may change the provided dataset) and returns the modified version.
`void`	`setAttributeIndices(java.lang.String value)` Sets which attributes are to be used for interquartile calculations and outlier/extreme value detection (only numeric attributes among the selection will be used).
`void`	`setAttributeIndicesArray(int[] value)` Sets which attributes are to be used for interquartile calculations and outlier/extreme value detection (only numeric attributes among the selection will be used).
`void`	`setDetectionPerAttribute(boolean value)` Set whether an Outlier/ExtremeValue attribute pair is generated for each numeric attribute ("true") or just one pair for all numeric attributes together ("false").
`void`	`setExtremeValuesAsOutliers(boolean value)` Set whether extreme values are also tagged as outliers.
`void`	`setExtremeValuesFactor(double value)` Sets the factor for determining the thresholds for extreme values.
`void`	`setOptions(java.lang.String[] options)` Parses a list of options for this object.
`void`	`setOutlierFactor(double value)` Sets the factor for determining the thresholds for outliers.
`void`	`setOutputOffsetMultiplier(boolean value)` Set whether an additional attribute "Offset" is generated per Outlier/ExtremeValue attribute pair that lists the multiplier the value is off the median: value = median + 'multiplier' * IQR.

Methods inherited from class weka.filters.SimpleBatchFilter
allowAccessToFullInputFormat, batchFinished, hasImmediateOutputFormat, input

Methods inherited from class weka.filters.SimpleFilter
reset, setInputFormat

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - NON_NUMERIC
```
public static final int NON_NUMERIC
```
    indicator for non-numeric attributes
    
    See Also:
    
    Constant Field Values
  - m_Attributes
```
protected Range m_Attributes
```
    the attribute range to work on
  - m_AttributeIndices
```
protected int[] m_AttributeIndices
```
    the generated indices (only for performance reasons)
  - m_OutlierFactor
```
protected double m_OutlierFactor
```
    the factor for detecting outliers
  - m_ExtremeValuesFactor
```
protected double m_ExtremeValuesFactor
```
    the factor for detecting extreme values, by default 2*m_OutlierFactor
  - m_ExtremeValuesAsOutliers
```
protected boolean m_ExtremeValuesAsOutliers
```
    whether extreme values are also tagged as outliers
  - m_UpperExtremeValue
```
protected double[] m_UpperExtremeValue
```
    the upper extreme value threshold (= Q3 + EVF*IQR)
  - m_UpperOutlier
```
protected double[] m_UpperOutlier
```
    the upper outlier threshold (= Q3 + OF*IQR)
  - m_LowerOutlier
```
protected double[] m_LowerOutlier
```
    the lower outlier threshold (= Q1 - OF*IQR)
  - m_IQR
```
protected double[] m_IQR
```
    the interquartile range
  - m_Median
```
protected double[] m_Median
```
    the median
  - m_LowerExtremeValue
```
protected double[] m_LowerExtremeValue
```
    the lower extreme value threshold (= Q1 - EVF*IQR)
  - m_DetectionPerAttribute
```
protected boolean m_DetectionPerAttribute
```
    whether to generate Outlier/ExtremeValue attributes for each attribute instead of a general one
  - m_OutlierAttributePosition
```
protected int[] m_OutlierAttributePosition
```
    the position of the outlier attribute
  - m_OutputOffsetMultiplier
```
protected boolean m_OutputOffsetMultiplier
```
    whether to add another attribute called "Offset", that lists the 'multiplier' by which the outlier/extreme value is away from the median, i.e., value = median + 'multiplier' * IQR
    automatically enables m_DetectionPerAttribute!
- Constructor Detail
  - InterquartileRange
```
public InterquartileRange()
```
- Method Detail
  - globalInfo
```
public java.lang.String globalInfo()
```
    Returns a string describing this filter
    
    Specified by:
    
    globalInfo in class SimpleFilter
    
    Returns:
    
    a description of the filter suitable for displaying in the explorer/experimenter gui
  - listOptions
```
public java.util.Enumeration<Option> listOptions()
```
    Returns an enumeration describing the available options.
    
    Specified by:
    
    listOptions in interface OptionHandler
    
    Overrides:
    
    listOptions in class Filter
    
    Returns:
    
    an enumeration of all the available options.
  - setOptions
```
public void setOptions(java.lang.String[] options)
                throws java.lang.Exception
```
    Parses a list of options for this object.
    Valid options are:
```
 -D
  Turns on output of debugging information.
 
```
```
 -R <col1,col2-col4,...>
  Specifies list of columns to base outlier/extreme value detection
  on. If an instance is considered in at least one of those
  attributes an outlier/extreme value, it is tagged accordingly.
  'first' and 'last' are valid indexes.
  (default none)
 
```
```
 -O <num>
  The factor for outlier detection.
  (default: 3)
 
```
```
 -E <num>
  The factor for extreme values detection.
  (default: 2*Outlier Factor)
 
```
```
 -E-as-O
  Tags extreme values also as outliers.
  (default: off)
 
```
```
 -P
  Generates Outlier/ExtremeValue pair for each numeric attribute in
  the range, not just a single indicator pair for all the attributes.
  (default: off)
 
```
```
 -M
  Generates an additional attribute 'Offset' per Outlier/ExtremeValue
  pair that contains the multiplier that the value is off the median.
     value = median + 'multiplier' * IQR
 Note: implicitely sets '-P'. (default: off)
 
```
    Specified by:
    
    setOptions in interface OptionHandler
    
    Overrides:
    
    setOptions in class Filter
    
    Parameters:
    
    options - the list of options as an array of strings
    
    Throws:
    
    java.lang.Exception - if an option is not supported
  - getOptions
```
public java.lang.String[] getOptions()
```
    Gets the current settings of the filter.
    
    Specified by:
    
    getOptions in interface OptionHandler
    
    Overrides:
    
    getOptions in class Filter
    
    Returns:
    
    an array of strings suitable for passing to setOptions
  - attributeIndicesTipText
```
public java.lang.String attributeIndicesTipText()
```
    Returns the tip text for this property
    
    Returns:
    
    tip text for this property suitable for displaying in the explorer/experimenter gui
  - getAttributeIndices
```
public java.lang.String getAttributeIndices()
```
    Gets the current range selection
    
    Returns:
    
    a string containing a comma separated list of ranges
  - setAttributeIndices
```
public void setAttributeIndices(java.lang.String value)
```
    Sets which attributes are to be used for interquartile calculations and outlier/extreme value detection (only numeric attributes among the selection will be used).
    
    Parameters:
    
    value - a string representing the list of attributes. Since the string will typically come from a user, attributes are indexed from 1.
    eg: first-3,5,6-last
    
    Throws:
    
    java.lang.IllegalArgumentException - if an invalid range list is supplied
  - setAttributeIndicesArray
```
public void setAttributeIndicesArray(int[] value)
```
    Sets which attributes are to be used for interquartile calculations and outlier/extreme value detection (only numeric attributes among the selection will be used).
    
    Parameters:
    
    value - an array containing indexes of attributes to work on. Since the array will typically come from a program, attributes are indexed from 0.
    
    Throws:
    
    java.lang.IllegalArgumentException - if an invalid set of ranges is supplied
  - outlierFactorTipText
```
public java.lang.String outlierFactorTipText()
```
    Returns the tip text for this property
    
    Returns:
    
    tip text for this property suitable for displaying in the explorer/experimenter gui
  - setOutlierFactor
```
public void setOutlierFactor(double value)
```
    Sets the factor for determining the thresholds for outliers.
    
    Parameters:
    
    value - the factor.
  - getOutlierFactor
```
public double getOutlierFactor()
```
    Gets the factor for determining the thresholds for outliers.
    
    Returns:
    
    the factor.
  - extremeValuesFactorTipText
```
public java.lang.String extremeValuesFactorTipText()
```
    Returns the tip text for this property
    
    Returns:
    
    tip text for this property suitable for displaying in the explorer/experimenter gui
  - setExtremeValuesFactor
```
public void setExtremeValuesFactor(double value)
```
    Sets the factor for determining the thresholds for extreme values.
    
    Parameters:
    
    value - the factor.
  - getExtremeValuesFactor
```
public double getExtremeValuesFactor()
```
    Gets the factor for determining the thresholds for extreme values.
    
    Returns:
    
    the factor.
  - extremeValuesAsOutliersTipText
```
public java.lang.String extremeValuesAsOutliersTipText()
```
    Returns the tip text for this property
    
    Returns:
    
    tip text for this property suitable for displaying in the explorer/experimenter gui
  - setExtremeValuesAsOutliers
```
public void setExtremeValuesAsOutliers(boolean value)
```
    Set whether extreme values are also tagged as outliers.
    
    Parameters:
    
    value - whether or not to tag extreme values also as outliers.
  - getExtremeValuesAsOutliers
```
public boolean getExtremeValuesAsOutliers()
```
    Get whether extreme values are also tagged as outliers.
    
    Returns:
    
    true if extreme values are also tagged as outliers.
  - detectionPerAttributeTipText
```
public java.lang.String detectionPerAttributeTipText()
```
    Returns the tip text for this property
    
    Returns:
    
    tip text for this property suitable for displaying in the explorer/experimenter gui
  - setDetectionPerAttribute
```
public void setDetectionPerAttribute(boolean value)
```
    Set whether an Outlier/ExtremeValue attribute pair is generated for each numeric attribute ("true") or just one pair for all numeric attributes together ("false").
    
    Parameters:
    
    value - whether or not to generate indicator attribute pairs for each numeric attribute.
  - getDetectionPerAttribute
```
public boolean getDetectionPerAttribute()
```
    Gets whether an Outlier/ExtremeValue attribute pair is generated for each numeric attribute ("true") or just one pair for all numeric attributes together ("false").
    
    Returns:
    
    true if indicator attribute pairs are generated for each numeric attribute.
  - outputOffsetMultiplierTipText
```
public java.lang.String outputOffsetMultiplierTipText()
```
    Returns the tip text for this property
    
    Returns:
    
    tip text for this property suitable for displaying in the explorer/experimenter gui
  - setOutputOffsetMultiplier
```
public void setOutputOffsetMultiplier(boolean value)
```
    Set whether an additional attribute "Offset" is generated per Outlier/ExtremeValue attribute pair that lists the multiplier the value is off the median: value = median + 'multiplier' * IQR.
    
    Parameters:
    
    value - whether or not to generate the additional attribute.
  - getOutputOffsetMultiplier
```
public boolean getOutputOffsetMultiplier()
```
    Gets whether an additional attribute "Offset" is generated per Outlier/ExtremeValue attribute pair that lists the multiplier the value is off the median: value = median + 'multiplier' * IQR.
    
    Returns:
    
    true if the additional attribute is generated.
  - getCapabilities
```
public Capabilities getCapabilities()
```
    Returns the Capabilities of this filter.
    
    Specified by:
    
    getCapabilities in interface CapabilitiesHandler
    
    Overrides:
    
    getCapabilities in class Filter
    
    Returns:
    
    the capabilities of this object
    
    See Also:
    
    Capabilities
  - determineOutputFormat
```
protected Instances determineOutputFormat(Instances inputFormat)
                                   throws java.lang.Exception
```
    Determines the output format based on the input format and returns this. In case the output format cannot be returned immediately, i.e., hasImmediateOutputFormat() returns false, then this method will called from batchFinished() after the call of preprocess(Instances), in which, e.g., statistics for the actual processing step can be gathered.
    
    Specified by:
    
    determineOutputFormat in class SimpleFilter
    
    Parameters:
    
    inputFormat - the input format to base the output format on
    
    Returns:
    
    the output format
    
    Throws:
    
    java.lang.Exception - in case the determination goes wrong
    
    See Also:
    
    SimpleBatchFilter.hasImmediateOutputFormat(), SimpleBatchFilter.batchFinished()
  - computeThresholds
```
protected void computeThresholds(Instances instances)
```
    computes the thresholds for outliers and extreme values
    
    Parameters:
    
    instances - the data to work on
  - getValues
```
public double[] getValues(InterquartileRange.ValueType type)
```
    Returns the values for the specified type.
    
    Parameters:
    
    type - the type of values to return
    
    Returns:
    
    the values
  - isOutlier
```
protected boolean isOutlier(Instance inst,
                            int index)
```
    returns whether the instance has an outlier in the specified attribute or not
    
    Parameters:
    
    inst - the instance to test
    
    index - the attribute index
    
    Returns:
    
    true if the instance is an outlier
  - isOutlier
```
protected boolean isOutlier(Instance inst)
```
    returns whether the instance is an outlier or not
    
    Parameters:
    
    inst - the instance to test
    
    Returns:
    
    true if the instance is an outlier
  - isExtremeValue
```
protected boolean isExtremeValue(Instance inst,
                                 int index)
```
    returns whether the instance has an extreme value in the specified attribute or not
    
    Parameters:
    
    inst - the instance to test
    
    index - the attribute index
    
    Returns:
    
    true if the instance is an extreme value
  - isExtremeValue
```
protected boolean isExtremeValue(Instance inst)
```
    returns whether the instance is an extreme value or not
    
    Parameters:
    
    inst - the instance to test
    
    Returns:
    
    true if the instance is an extreme value
  - calculateMultiplier
```
protected double calculateMultiplier(Instance inst,
                                     int index)
```
    returns the mulitplier of the IQR the instance is off the median for this particular attribute.
    
    Parameters:
    
    inst - the instance to test
    
    index - the attribute index
    
    Returns:
    
    the multiplier
  - process
```
protected Instances process(Instances instances)
                     throws java.lang.Exception
```
    Processes the given data (may change the provided dataset) and returns the modified version. This method is called in batchFinished(). This implementation only calls process(Instance) for each instance in the given dataset.
    
    Specified by:
    
    process in class SimpleFilter
    
    Parameters:
    
    instances - the data to process
    
    Returns:
    
    the modified data
    
    Throws:
    
    java.lang.Exception - in case the processing goes wrong
    
    See Also:
    
    SimpleBatchFilter.batchFinished()
  - getRevision
```
public java.lang.String getRevision()
```
    Returns the revision string.
    
    Specified by:
    
    getRevision in interface RevisionHandler
    
    Overrides:
    
    getRevision in class Filter
    
    Returns:
    
    the revision
  - main
```
public static void main(java.lang.String[] args)
```
    Main method for testing this class.
    
    Parameters:
    
    args - should contain arguments to the filter: use -h for help

Class InterquartileRange

Nested Class Summary

Field Summary

Fields inherited from class weka.filters.Filter

Constructor Summary

Method Summary

Methods inherited from class weka.filters.SimpleBatchFilter

Methods inherited from class weka.filters.SimpleFilter

Methods inherited from class weka.filters.Filter

Methods inherited from class java.lang.Object

Field Detail

NON_NUMERIC

m_Attributes

m_AttributeIndices

m_OutlierFactor

m_ExtremeValuesFactor

m_ExtremeValuesAsOutliers

m_UpperExtremeValue

m_UpperOutlier

m_LowerOutlier

m_IQR

m_Median

m_LowerExtremeValue

m_DetectionPerAttribute

m_OutlierAttributePosition

m_OutputOffsetMultiplier

Constructor Detail

InterquartileRange

Method Detail

globalInfo

listOptions

setOptions

getOptions

attributeIndicesTipText

getAttributeIndices

setAttributeIndices

setAttributeIndicesArray

outlierFactorTipText

setOutlierFactor

getOutlierFactor

extremeValuesFactorTipText

setExtremeValuesFactor

getExtremeValuesFactor

extremeValuesAsOutliersTipText

setExtremeValuesAsOutliers

getExtremeValuesAsOutliers

detectionPerAttributeTipText

setDetectionPerAttribute

getDetectionPerAttribute

outputOffsetMultiplierTipText

setOutputOffsetMultiplier

getOutputOffsetMultiplier

getCapabilities

determineOutputFormat

computeThresholds

getValues

isOutlier

isOutlier

isExtremeValue

isExtremeValue

calculateMultiplier

process

getRevision

main