weka.filters

Class Filter

java.lang.Object

weka.filters.Filter

All Implemented Interfaces:

java.io.Serializable, CapabilitiesHandler, CapabilitiesIgnorer, CommandlineRunnable, OptionHandler, RevisionHandler

Direct Known Subclasses:

AbstractTimeSeries, Add, AddCluster, AddExpression, AddID, AddNoise, AddUserFields, AddValues, AllFilter, AttributeSelection, ChangeDateFormat, ClassOrder, ClusterMembership, Copy, Discretize, FirstOrder, MakeIndicator, MergeTwoValues, NominalToBinary, NominalToBinary, NominalToString, NonSparseToSparse, NumericTransform, Obfuscate, PartitionMembership, PotentialClassIgnorer, PrincipalComponents, Randomize, RandomProjection, Remove, RemoveFolds, RemoveFrequentValues, RemoveMisclassified, RemovePercentage, RemoveRange, RemoveType, RemoveUseless, RemoveWithValues, RenameNominalValues, RenameRelation, Reorder, Resample, Resample, ReservoirSample, SimpleFilter, SparseToNonSparse, SpreadSubsample, StratifiedRemoveFolds, StringToNominal, StringToWordVector, SwapValues

public abstract class Filter
extends java.lang.Object
implements java.io.Serializable, CapabilitiesHandler, RevisionHandler, OptionHandler, CapabilitiesIgnorer, CommandlineRunnable

An abstract class for instance filters: objects that take instances as input, carry out some transformation on the instance and then output the instance. The method implementations in this class assume that most of the work will be done in the methods overridden by subclasses.

A simple example of filter use. This example doesn't remove instances from the output queue until all instances have been input, so has higher memory consumption than an approach that uses output instances as they are made available:

  Filter filter = ..some type of filter..
  Instances instances = ..some instances..
  for (int i = 0; i < data.numInstances(); i++) {
    filter.input(data.instance(i));
  }
  filter.batchFinished();
  Instances newData = filter.outputFormat();
  Instance processed;
  while ((processed = filter.output()) != null) {
    newData.add(processed);
  }
  ..do something with newData..
 

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	m_Debug Whether the classifier is run in debug mode.
protected boolean	m_DoNotCheckCapabilities Whether capabilities should not be checked before classifier is built.
protected boolean	m_FirstBatchDone True if the first batch has been done
protected RelationalLocator	m_InputRelAtts Indices of relational attributes in the input format
protected StringLocator	m_InputStringAtts Indices of string attributes in the input format
protected boolean	m_NewBatch Record whether the filter is at the start of a batch
protected RelationalLocator	m_OutputRelAtts Indices of relational attributes in the output format
protected StringLocator	m_OutputStringAtts Indices of string attributes in the output format

Constructor Summary

Constructors

Constructor and Description
Filter()

Method Summary

Modifier and Type	Method and Description
static void	batchFilterFile(Filter filter, java.lang.String[] options) Method for testing filters ability to process multiple batches.
boolean	batchFinished() Signify that this batch of input to the filter is finished.
protected void	bufferInput(Instance instance) Adds the supplied input instance to the inputformat dataset for later processing.
protected void	copyValues(Instance instance, boolean isInput) Copies string/relational values contained in the instance copied to a new dataset.
protected void	copyValues(Instance instance, boolean instSrcCompat, Instances srcDataset, Instances destDataset) Takes string/relational values referenced by an Instance and copies them from a source dataset to a destination dataset.
java.lang.String	debugTipText() Returns the tip text for this property
java.lang.String	doNotCheckCapabilitiesTipText() Returns the tip text for this property
static void	filterFile(Filter filter, java.lang.String[] options) Method for testing filters.
protected void	flushInput() This will remove all buffered instances from the inputformat dataset.
Capabilities	getCapabilities() Returns the Capabilities of this filter.
Capabilities	getCapabilities(Instances data) Returns the Capabilities of this filter, customized based on the data.
Instances	getCopyOfInputFormat() Gets a copy of just the structure of the input format instances.
boolean	getDebug() Get whether debugging is turned on.
boolean	getDoNotCheckCapabilities() Get whether capabilities checking is turned off.
protected Instances	getInputFormat() Gets the currently set inputformat instances.
java.lang.String[]	getOptions() Gets the current settings of the filter.
Instances	getOutputFormat() Gets the format of the output instances.
java.lang.String	getRevision() Returns the revision string.
protected void	initInputLocators(Instances data, int[] indices) Initializes the input attribute locators.
protected void	initOutputLocators(Instances data, int[] indices) Initializes the output attribute locators.
boolean	input(Instance instance) Input an instance for filtering.
protected Instances	inputFormatPeek() Returns a reference to the current input format without copying it.
boolean	isFirstBatchDone() Returns true if the first batch of instances got processed.
boolean	isNewBatch() Returns true if the a new batch was started, either a new instance of the filter was created or the batchFinished() method got called.
boolean	isOutputFormatDefined() Returns whether the output format is ready to be collected
java.util.Enumeration<Option>	listOptions() Returns an enumeration describing the available options.
static void	main(java.lang.String[] args) Main method for testing this class.
static Filter[]	makeCopies(Filter model, int num) Creates a given number of deep copies of the given filter using serialization.
static Filter	makeCopy(Filter model) Creates a deep copy of the given filter using serialization.
boolean	mayRemoveInstanceAfterFirstBatchDone() Default implementation returns false.
int	numPendingOutput() Returns the number of instances pending output
Instance	output() Output an instance after filtering and remove from the output queue.
protected Instances	outputFormatPeek() Returns a reference to the current output format without copying it.
Instance	outputPeek() Output an instance after filtering but do not remove from the output queue.
void	postExecution() Perform any teardown stuff that might need to happen after execution.
void	preExecution() Perform any setup stuff that might need to happen before commandline execution.
protected void	push(Instance instance) Adds an output instance to the queue.
protected void	push(Instance instance, boolean copyInstance) Adds an output instance to the queue.
protected void	resetQueue() Clears the output queue.
void	run(java.lang.Object toRun, java.lang.String[] options) Execute the supplied object.
static void	runFilter(Filter filter, java.lang.String[] options) runs the filter instance with the given options.
void	setDebug(boolean debug) Set debugging mode.
void	setDoNotCheckCapabilities(boolean doNotCheckCapabilities) Set whether not to check capabilities.
boolean	setInputFormat(Instances instanceInfo) Sets the format of the input instances.
void	setOptions(java.lang.String[] options) Parses a given list of options.
protected void	setOutputFormat(Instances outputFormat) Sets the format of output instances.
protected void	testInputFormat(Instances instanceInfo) tests the data whether the filter can actually handle it
java.lang.String	toString() Returns a description of the filter, by default only the classname.
static Instances	useFilter(Instances data, Filter filter) Filters an entire set of instances through a filter and returns the new set.
static java.lang.String	wekaStaticWrapper(Sourcable filter, java.lang.String className, Instances input, Instances output) generates source code from the filter

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

m_OutputStringAtts

protected StringLocator m_OutputStringAtts

Indices of string attributes in the output format

m_InputStringAtts

protected StringLocator m_InputStringAtts

Indices of string attributes in the input format

m_OutputRelAtts

protected RelationalLocator m_OutputRelAtts

Indices of relational attributes in the output format

m_InputRelAtts

protected RelationalLocator m_InputRelAtts

Indices of relational attributes in the input format

m_NewBatch

protected boolean m_NewBatch

Record whether the filter is at the start of a batch

m_FirstBatchDone

protected boolean m_FirstBatchDone

True if the first batch has been done

m_Debug

protected boolean m_Debug

Whether the classifier is run in debug mode.

m_DoNotCheckCapabilities

protected boolean m_DoNotCheckCapabilities

Whether capabilities should not be checked before classifier is built.

Constructor Detail

Filter

public Filter()

Method Detail

isNewBatch

public boolean isNewBatch()

Returns true if the a new batch was started, either a new instance of the filter was created or the batchFinished() method got called.

Returns:

true if a new batch has been initiated

See Also:

m_NewBatch, batchFinished()

isFirstBatchDone

public boolean isFirstBatchDone()

Returns true if the first batch of instances got processed. Necessary for supervised filters, which "learn" from the first batch and then shouldn't get updated with subsequent calls of batchFinished().

Returns:

true if the first batch has been processed

See Also:

m_FirstBatchDone, batchFinished()

mayRemoveInstanceAfterFirstBatchDone

public boolean mayRemoveInstanceAfterFirstBatchDone()

Default implementation returns false. Some filters may not necessarily be able to produce an instance for output for every instance input after the first batch has been completed - such filters should override this method and return true.

Returns:

false by default

getCapabilities

public Capabilities getCapabilities()

Returns the Capabilities of this filter. Derived filters have to override this method to enable capabilities.

Specified by:

getCapabilities in interface CapabilitiesHandler

Returns:

the capabilities of this object

See Also:

Capabilities

getRevision

public java.lang.String getRevision()

Returns the revision string.

Specified by:

getRevision in interface RevisionHandler

Returns:

the revision

getCapabilities

public Capabilities getCapabilities(Instances data)

Returns the Capabilities of this filter, customized based on the data. I.e., if removes all class capabilities, in case there's not class attribute present or removes the NO_CLASS capability, in case that there's a class present.

Parameters:

data - the data to use for customization

Returns:

the capabilities of this object, based on the data

See Also:

getCapabilities()

setOutputFormat

protected void setOutputFormat(Instances outputFormat)

Sets the format of output instances. The derived class should use this method once it has determined the outputformat. The output queue is cleared.

Parameters:

outputFormat - the new output format

getInputFormat

protected Instances getInputFormat()

Gets the currently set inputformat instances. This dataset may contain buffered instances.

Returns:

the input Instances.

getCopyOfInputFormat

public Instances getCopyOfInputFormat()

Gets a copy of just the structure of the input format instances.

Returns:

a copy of the structure (attribute information) of the input format instances

inputFormatPeek

protected Instances inputFormatPeek()

Returns a reference to the current input format without copying it.

Returns:

a reference to the current input format

outputFormatPeek

protected Instances outputFormatPeek()

Returns a reference to the current output format without copying it.

Returns:

a reference to the current output format

push

protected void push(Instance instance)

Adds an output instance to the queue. The derived class should use this method for each output instance it makes available. Note that the instance is only copied before it is added to the output queue if it has a reference to a dataset.

Parameters:

instance - the instance to be added to the queue.

push

protected void push(Instance instance,
                    boolean copyInstance)

Adds an output instance to the queue. The derived class should use this method for each output instance it makes available. Note that the instance is only copied before it is added to the output queue if copyInstance has value true and if the instance has a reference to a dataset.

Parameters:

instance - the instance to be added to the queue.

copyInstance - whether instance is to be copied

resetQueue

protected void resetQueue()

Clears the output queue.

bufferInput

protected void bufferInput(Instance instance)

Adds the supplied input instance to the inputformat dataset for later processing. Use this method rather than getInputFormat().add(instance). Or else. Note that the provided instance gets copied when buffered.

Parameters:

instance - the Instance to buffer.

initInputLocators

protected void initInputLocators(Instances data,
                                 int[] indices)

Initializes the input attribute locators. If indices is null then all attributes of the data will be considered, otherwise only the ones that were provided.

Parameters:

data - the data to initialize the locators with

indices - if not null, the indices to which to restrict the locating

initOutputLocators

protected void initOutputLocators(Instances data,
                                  int[] indices)

Initializes the output attribute locators. If indices is null then all attributes of the data will be considered, otherwise only the ones that were provided.

Parameters:

data - the data to initialize the locators with

indices - if not null, the indices to which to restrict the locating

copyValues

protected void copyValues(Instance instance,
                          boolean isInput)

Copies string/relational values contained in the instance copied to a new dataset. The Instance must already be assigned to a dataset. This dataset and the destination dataset must have the same structure.

Parameters:

instance - the Instance containing the string/relational values to copy.

isInput - if true the input format and input attribute locators are used otherwise the output format and output locators

copyValues

protected void copyValues(Instance instance,
                          boolean instSrcCompat,
                          Instances srcDataset,
                          Instances destDataset)

Takes string/relational values referenced by an Instance and copies them from a source dataset to a destination dataset. The instance references are updated to be valid for the destination dataset. The instance may have the structure (i.e. number and attribute position) of either dataset (this affects where references are obtained from). Only works if the number of string/relational attributes is the same in both indices (implicitly these string/relational attributes should be semantically same but just with shifted positions).

Parameters:

instance - the instance containing references to strings/ relational values in the source dataset that will have references updated to be valid for the destination dataset.

instSrcCompat - true if the instance structure is the same as the source, or false if it is the same as the destination (i.e. which of the string/relational attribute indices contains the correct locations for this instance).

srcDataset - the dataset for which the current instance string/relational value references are valid (after any position mapping if needed)

destDataset - the dataset for which the current instance string/relational value references need to be inserted (after any position mapping if needed)

flushInput

protected void flushInput()

This will remove all buffered instances from the inputformat dataset. Use this method rather than getInputFormat().delete();

testInputFormat

protected void testInputFormat(Instances instanceInfo)
                        throws java.lang.Exception

tests the data whether the filter can actually handle it

Parameters:

instanceInfo - the data to test

Throws:

java.lang.Exception - if the test fails

setInputFormat

public boolean setInputFormat(Instances instanceInfo)
                       throws java.lang.Exception

Sets the format of the input instances. If the filter is able to determine the output format before seeing any input instances, it does so here. This default implementation clears the output format and output queue, and the new batch flag is set. Overriders should call super.setInputFormat(Instances)

Parameters:

instanceInfo - an Instances object containing the input instance structure (any instances contained in the object are ignored - only the structure is required).

Returns:

true if the outputFormat may be collected immediately

Throws:

java.lang.Exception - if the inputFormat can't be set successfully

getOutputFormat

public Instances getOutputFormat()

Gets the format of the output instances. This should only be called after input() or batchFinished() has returned true. The relation name of the output instances should be changed to reflect the action of the filter (eg: add the filter name and options).

Returns:

an Instances object containing the output instance structure only.

Throws:

java.lang.NullPointerException - if no input structure has been defined (or the output format hasn't been determined yet)

input

public boolean input(Instance instance)
              throws java.lang.Exception

Input an instance for filtering. Ordinarily the instance is processed and made available for output immediately. Some filters require all instances be read before producing output, in which case output instances should be collected after calling batchFinished(). If the input marks the start of a new batch, the output queue is cleared. This default implementation assumes all instance conversion will occur when batchFinished() is called.

Parameters:

instance - the input instance

Returns:

true if the filtered instance may now be collected with output().

Throws:

java.lang.NullPointerException - if the input format has not been defined.

java.lang.Exception - if the input instance was not of the correct format or if there was a problem with the filtering.

batchFinished

public boolean batchFinished()
                      throws java.lang.Exception

Signify that this batch of input to the filter is finished. If the filter requires all instances prior to filtering, output() may now be called to retrieve the filtered instances. Any subsequent instances filtered should be filtered based on setting obtained from the first batch (unless the inputFormat has been re-assigned or new options have been set). This default implementation assumes all instance processing occurs during inputFormat() and input().

Returns:

true if there are instances pending output

Throws:

java.lang.NullPointerException - if no input structure has been defined,

java.lang.Exception - if there was a problem finishing the batch.

output

public Instance output()

Output an instance after filtering and remove from the output queue.

Returns:

the instance that has most recently been filtered (or null if the queue is empty).

Throws:

java.lang.NullPointerException - if no output structure has been defined

outputPeek

public Instance outputPeek()

Output an instance after filtering but do not remove from the output queue.

Returns:

the instance that has most recently been filtered (or null if the queue is empty).

Throws:

java.lang.NullPointerException - if no input structure has been defined

numPendingOutput

public int numPendingOutput()

Returns the number of instances pending output

Returns:

the number of instances pending output

Throws:

java.lang.NullPointerException - if no input structure has been defined

isOutputFormatDefined

public boolean isOutputFormatDefined()

Returns whether the output format is ready to be collected

Returns:

true if the output format is set

makeCopy

public static Filter makeCopy(Filter model)
                       throws java.lang.Exception

Creates a deep copy of the given filter using serialization.

Parameters:

model - the filter to copy

Returns:

a deep copy of the filter

Throws:

java.lang.Exception - if an error occurs

makeCopies

public static Filter[] makeCopies(Filter model,
                                  int num)
                           throws java.lang.Exception

Creates a given number of deep copies of the given filter using serialization.

Parameters:

model - the filter to copy

num - the number of filter copies to create.

Returns:

an array of filters.

Throws:

java.lang.Exception - if an error occurs

useFilter

public static Instances useFilter(Instances data,
                                  Filter filter)
                           throws java.lang.Exception

Filters an entire set of instances through a filter and returns the new set.

Parameters:

data - the data to be filtered

filter - the filter to be used

Returns:

the filtered set of data

Throws:

java.lang.Exception - if the filter can't be used successfully

toString

public java.lang.String toString()

Returns a description of the filter, by default only the classname.

Overrides:

toString in class java.lang.Object

Returns:

a string describing the filter

wekaStaticWrapper

public static java.lang.String wekaStaticWrapper(Sourcable filter,
                                                 java.lang.String className,
                                                 Instances input,
                                                 Instances output)
                                          throws java.lang.Exception

generates source code from the filter

Parameters:

filter - the filter to output as source

className - the name of the generated class

input - the input data the header is generated for

output - the output data the header is generated for

Returns:

the generated source code

Throws:

java.lang.Exception - if source code cannot be generated

filterFile

public static void filterFile(Filter filter,
                              java.lang.String[] options)
                       throws java.lang.Exception

Method for testing filters.

Parameters:

filter - the filter to use

options - should contain the following arguments:
-i input_file
-o output_file
-c class_index
-z classname (for filters implementing weka.filters.Sourcable)
-decimal num (the number of decimal places to use in the output; default = 6)
or -h for help on options

Throws:

java.lang.Exception - if something goes wrong or the user requests help on command options

batchFilterFile

public static void batchFilterFile(Filter filter,
                                   java.lang.String[] options)
                            throws java.lang.Exception

Method for testing filters ability to process multiple batches.

Parameters:

filter - the filter to use

options - should contain the following arguments:
-i (first) input file
-o (first) output file
-r (second) input file
-s (second) output file
-c class_index
-z classname (for filters implementing weka.filters.Sourcable)
-decimal num (the number of decimal places to use in the output; default = 6)
or -h for help on options

Throws:

java.lang.Exception - if something goes wrong or the user requests help on command options

runFilter

public static void runFilter(Filter filter,
                             java.lang.String[] options)

runs the filter instance with the given options.

Parameters:

filter - the filter to run

options - the commandline options

listOptions

public java.util.Enumeration<Option> listOptions()

Returns an enumeration describing the available options.

Specified by:

listOptions in interface OptionHandler

Returns:

an enumeration of all the available options.

setOptions

public void setOptions(java.lang.String[] options)
                throws java.lang.Exception

Parses a given list of options. Valid options are:

-D
If set, filter is run in debug mode and may output additional info to the console.

-do-not-check-capabilities
If set, filter capabilities are not checked before filter is built (use with caution).

Specified by:

setOptions in interface OptionHandler

Parameters:

options - the list of options as an array of strings

Throws:

java.lang.Exception - if an option is not supported

getOptions

public java.lang.String[] getOptions()

Gets the current settings of the filter.

Specified by:

getOptions in interface OptionHandler

Returns:

an array of strings suitable for passing to setOptions

setDebug

public void setDebug(boolean debug)

Set debugging mode.

Parameters:

debug - true if debug output should be printed

getDebug

public boolean getDebug()

Get whether debugging is turned on.

Returns:

true if debugging output is on

debugTipText

public java.lang.String debugTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

setDoNotCheckCapabilities

public void setDoNotCheckCapabilities(boolean doNotCheckCapabilities)

Set whether not to check capabilities.

Specified by:

setDoNotCheckCapabilities in interface CapabilitiesIgnorer

Parameters:

doNotCheckCapabilities - true if capabilities are not to be checked.

getDoNotCheckCapabilities

public boolean getDoNotCheckCapabilities()

Get whether capabilities checking is turned off.

Specified by:

getDoNotCheckCapabilities in interface CapabilitiesIgnorer

Returns:

true if capabilities checking is turned off.

doNotCheckCapabilitiesTipText

public java.lang.String doNotCheckCapabilitiesTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

preExecution

public void preExecution()
                  throws java.lang.Exception

Perform any setup stuff that might need to happen before commandline execution. Subclasses should override if they need to do something here

Specified by:

preExecution in interface CommandlineRunnable

Throws:

java.lang.Exception - if a problem occurs during setup

run

public void run(java.lang.Object toRun,
                java.lang.String[] options)
         throws java.lang.Exception

Execute the supplied object.

Specified by:

run in interface CommandlineRunnable

Parameters:

toRun - the object to execute

options - any options to pass to the object

Throws:

java.lang.Exception - if the object is not of the expected type.

postExecution

public void postExecution()
                   throws java.lang.Exception

Perform any teardown stuff that might need to happen after execution. Subclasses should override if they need to do something here

Specified by:

postExecution in interface CommandlineRunnable

Throws:

java.lang.Exception - if a problem occurs during teardown

main

public static void main(java.lang.String[] args)

Main method for testing this class.

Parameters:

args - should contain arguments to the filter: use -h for help