it.unimi.dsi.mg4j.index
Class MultiTermIndexIterator

java.lang.Object
  it.unimi.dsi.fastutil.ints.AbstractIntIterator
      it.unimi.dsi.mg4j.search.AbstractDocumentIterator
          it.unimi.dsi.mg4j.search.AbstractCompositeDocumentIterator
              it.unimi.dsi.mg4j.search.AbstractUnionDocumentIterator
                  it.unimi.dsi.mg4j.index.MultiTermIndexIterator

All Implemented Interfaces:

IntIterator, IndexIterator, DocumentIterator, Iterable<Interval>, Iterator<Integer>

public class MultiTermIndexIterator
extends AbstractUnionDocumentIterator
implements IndexIterator

A virtual index iterator that merges several component index iterators.

This class adds to AbstractUnionDocumentIterator an interval iterator generating the OR of the intervals returned for each of the documents involved. The main difference with an OrDocumentIterator built on the same array of component iterators is that this class implements IndexIterator and hence provides a count() (the sum of counts of those component iterators positioned on the current document) and a frequency(). The latter is by default the maximum frequency of a component iterator, but it can be set at construction time.

The main raison d'être of this class is support for query expansion: a blind application of OrDocumentIterator to an array of index iterators would mislead scorers such as BM25Scorer because low-frequency terms (e.g., hapax legomena) would be responsible for most of the score.

Note that DocumentIteratorVisitor has a visit method for generic index iterator and a visit method for instances of this class. This approach provides additional flexibility—a scorer, for instance, might treat an instance of this class as a standard IndexIterator, or it might choose to query which terms actually appear and do something more sophisticated (for instance, using weights).

Nested Class Summary

Nested classes/interfaces inherited from class it.unimi.dsi.mg4j.search.AbstractCompositeDocumentIterator

AbstractCompositeDocumentIterator.AbstractCompositeIndexIntervalIterator, AbstractCompositeDocumentIterator.AbstractCompositeIntervalIterator

Nested classes/interfaces inherited from class it.unimi.dsi.mg4j.search.AbstractDocumentIterator

AbstractDocumentIterator.AbstractIntervalIterator

Field Summary

protected int	id The id of this iterator.
protected String	term The term of this iterator.

Fields inherited from class it.unimi.dsi.mg4j.search.AbstractUnionDocumentIterator

front, frontSize, queue, refArray

Fields inherited from class it.unimi.dsi.mg4j.search.AbstractCompositeDocumentIterator

documentIterator, indexIterator, indices, n, soleIndex

Fields inherited from class it.unimi.dsi.mg4j.search.AbstractDocumentIterator

ahead, curr, weight

Fields inherited from interface it.unimi.dsi.mg4j.search.DocumentIterator

END_OF_LIST

Constructor Summary

protected

MultiTermIndexIterator(int defaultFrequency, IndexIterator... indexIterator) Creates a new document iterator that merges the given array of iterators.

Method Summary

<T> T	accept(DocumentIteratorVisitor<T> visitor) Accepts a visitor.
<T> T	acceptDeep(DocumentIteratorVisitor<T> visitor)
<T> T	acceptDeepOnTruePaths(DocumentIteratorVisitor<T> visitor)
<T> T	acceptOnTruePaths(DocumentIteratorVisitor<T> visitor) Invokes AbstractUnionDocumentIterator.acceptOnTruePaths(DocumentIteratorVisitor) only on component iterators positioned on the current document.
int	count() The count is the sum of counts of those component iterators positioned on the current document.
int	frequency() The frequency is either the default frequency set at construction time, or the maximum frequency of the component iterators.
int	front(IndexIterator[] indexIterator) Fills the given array with the index iterators composing the current front.
protected IntervalIterator	getComposedIntervalIterator(Index index)
static IndexIterator	getInstance(Index index, IndexIterator... indexIterator) Returns an index iterator that merges the given array of iterators.
static IndexIterator	getInstance(IndexIterator... indexIterator) Returns an index iterator that merges the given array of iterators.
static IndexIterator	getInstance(int defaultFrequency, Index index, IndexIterator... indexIterator) Returns an index iterator that merges the given array of iterators.
static IndexIterator	getInstance(int defaultFrequency, IndexIterator... indexIterator) Returns an index iterator that merges the given array of iterators.
int	id() Returns the id of this index iterator.
IndexIterator	id(int id) Sets the id of this index iterator.
Index	index() Returns the index over which this iterator is built.
protected int	nextDocumentInternal()
Payload	payload() This method is not implemented by this class.
int[]	positionArray() Returns the positions at which the term appears in the current document in an array.
IntIterator	positions() Returns the positions at which the term appears in the current document.
int	positions(int[] position) Stores the positions at which the term appears in the current document in a given array.
int	skipTo(int n) Skips all documents smaller than n.
String	term() Returns the term whose inverted list is returned by this index iterator.
IndexIterator	term(CharSequence term) Sets the term whose inverted list is returned by this index iterator.
int	termNumber() Returns the number of the term whose inverted list is returned by this index iterator.
IndexIterator	weight(double weight) Sets the weight of this index iterator.

Methods inherited from class it.unimi.dsi.mg4j.search.AbstractUnionDocumentIterator

computeFront, intervalIterator, intervalIterators

Methods inherited from class it.unimi.dsi.mg4j.search.AbstractCompositeDocumentIterator

dispose, indices, intervalIterator, toString

Methods inherited from class it.unimi.dsi.mg4j.search.AbstractDocumentIterator

document, ensureOnADocument, fromNextDocument, hasNext, iterator, nextDocument, nextInt, toNextDocument, weight

Methods inherited from class it.unimi.dsi.fastutil.ints.AbstractIntIterator

next, remove, skip

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface it.unimi.dsi.mg4j.search.DocumentIterator

dispose, document, indices, intervalIterator, intervalIterator, intervalIterators, iterator, nextDocument, nextInt, weight

Methods inherited from interface it.unimi.dsi.fastutil.ints.IntIterator

skip

Methods inherited from interface java.util.Iterator

hasNext, next, remove

Field Detail

term

protected String term

The term of this iterator.

id

protected int id

The id of this iterator.

Constructor Detail

MultiTermIndexIterator

protected MultiTermIndexIterator(int defaultFrequency,
                                 IndexIterator... indexIterator)
                          throws IOException

Creates a new document iterator that merges the given array of iterators.

Parameters:

defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).

indexIterator - the iterators to be joined.

Throws:

IOException

Method Detail

getInstance

public static IndexIterator getInstance(IndexIterator... indexIterator)
                                 throws IOException

Returns an index iterator that merges the given array of iterators. This method requires that at least one iterator is provided. The frequency is computed as a max, and index() will return the result of the same method on the first iterator.

Parameters:

indexIterator - the iterators to be joined (at least one).

Returns:

a merged index iterator.

Throws:

IllegalArgumentException - if no iterators were provided.

IOException

getInstance

public static IndexIterator getInstance(Index index,
                                        IndexIterator... indexIterator)
                                 throws IOException

Returns an index iterator that merges the given array of iterators.

Note that the special case of the empty and of the singleton arrays are handled efficiently. The frequency is computed as a max, and index() will return index.

Parameters:

index - the index that wil be returned by index().

indexIterator - the iterators to be joined.

Returns:

a merged index iterator.

Throws:

IOException

getInstance

public static IndexIterator getInstance(int defaultFrequency,
                                        IndexIterator... indexIterator)
                                 throws IOException

Returns an index iterator that merges the given array of iterators. This method requires that at least one iterator is provided.

Parameters:

defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).

indexIterator - the iterators to be joined (at least one).

Returns:

a merged index iterator.

Throws:

IllegalArgumentException - if no iterators were provided, or they run on different indices.

IOException

getInstance

public static IndexIterator getInstance(int defaultFrequency,
                                        Index index,
                                        IndexIterator... indexIterator)
                                 throws IOException

Returns an index iterator that merges the given array of iterators.

Note that the special case of the empty and of the singleton arrays are handled efficiently.

Parameters:

defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).

index - the index that wil be returned by index().

indexIterator - the iterators to be joined.

Returns:

a merged index iterator.

Throws:

IllegalArgumentException - if there is some iterator on an index different from index.

IOException

getComposedIntervalIterator

protected IntervalIterator getComposedIntervalIterator(Index index)

Specified by:

getComposedIntervalIterator in class AbstractUnionDocumentIterator

skipTo

public int skipTo(int n)
           throws IOException

Description copied from interface: DocumentIterator

Skips all documents smaller than n. If Iterator.hasNext() has been called returning true but DocumentIterator.nextDocument() has not been called afterwards, then a call to DocumentIterator.skipTo(int) will be implicitly preceded by a call to DocumentIterator.nextDocument() (the only consequence is that skipping to the current document after a call to Iterator.hasNext() will return the next document).

Define the current document k associated with this document iterator as follows:

• -1, if DocumentIterator.nextDocument() and this method have never been called;
• DocumentIterator.END_OF_LIST, if a call to this method returned DocumentIterator.END_OF_LIST, or DocumentIterator.nextDocument() returned -1;
• the last value returned by a call to DocumentIterator.nextDocument() or this method, otherwise.

If k is larger than or equal to n, then this method does nothing and returns k. Otherwise, a call to this method is equivalent to

 while( ( k = nextDocument() ) < n && k != -1 );
 return k == -1 ? END_OF_LIST : k;
 

Thus, when a result k ≠ DocumentIterator.END_OF_LIST is returned, the state of this iterator will be exactly the same as after a call to DocumentIterator.nextDocument() that returned k. In particular, the first document larger than or equal to n (when returned by this method) will not be returned by the next call to DocumentIterator.nextDocument().

Specified by:

skipTo in interface DocumentIterator

Overrides:

skipTo in class AbstractUnionDocumentIterator

Parameters:

n - a document pointer.

Returns:

a document pointer larger than or equal to n if available, DocumentIterator.END_OF_LIST otherwise.

Throws:

IOException

nextDocumentInternal

protected int nextDocumentInternal()
                            throws IOException

Overrides:

nextDocumentInternal in class AbstractUnionDocumentIterator

Throws:

IOException

count

public int count()
          throws IOException

The count is the sum of counts of those component iterators positioned on the current document.

Specified by:

count in interface IndexIterator

Returns:

the sum of counts.

Throws:

IOException

front

public int front(IndexIterator[] indexIterator)

Fills the given array with the index iterators composing the current front.

This method is essentially a safe exposure of the front of the queue merging the component index iterators. After a call to AbstractDocumentIterator.nextDocument(), you can use this method to know which terms actually appear in the current document. You can use the public field AbstractCompositeDocumentIterator.n to size the argument array appropriately.

Parameters:

indexIterator - an array, at least large as the number of component index iterators, that will be partially filled with the index iterators corresponding to terms appearing in the current document.

Returns:

the number of iterators written into indexIterator.

frequency

public int frequency()
              throws IOException

The frequency is either the default frequency set at construction time, or the maximum frequency of the component iterators.

Specified by:

frequency in interface IndexIterator

Returns:

the frequency.

Throws:

IOException

term

public IndexIterator term(CharSequence term)

Description copied from interface: IndexIterator

Sets the term whose inverted list is returned by this index iterator.

Usually, the term is automatically set by Index.documents(CharSequence) or by IndexReader.documents(CharSequence), but you can use this method to ensure that IndexIterator.term() doesn't throw an exception.

Specified by:

term in interface IndexIterator

Parameters:

term - a character sequence (that will be defensively copied) that will be assumed to be the term whose inverted list is returned by this index iterator.

Returns:

this index iterator.

term

public String term()

Description copied from interface: IndexIterator

Returns the term whose inverted list is returned by this index iterator.

Usually, the term is automatically set by IndexReader.documents(CharSequence) or IndexReader.documents(int), but you can supply your own term with IndexIterator.term(CharSequence).

Specified by:

term in interface IndexIterator

Returns:

the term over which this iterator is built, as a compact mutable string.

See Also:

IndexIterator.termNumber()

termNumber

public int termNumber()

Description copied from interface: IndexIterator

Returns the number of the term whose inverted list is returned by this index iterator.

Usually, the term number is automatically set by IndexReader.documents(CharSequence) or IndexReader.documents(int).

Specified by:

termNumber in interface IndexIterator

Returns:

the number of the term over which this iterator is built.

See Also:

IndexIterator.term()

id

public IndexIterator id(int id)

Description copied from interface: IndexIterator

Sets the id of this index iterator.

The id is an integer associated with each index iterator. It has no specific semantics, and can be used differently in different contexts. A typical usage pattern, for instance, is using it to assign a unique number to the index iterators contained in a composite document iterator (say, numbering consecutively the leaves of the composite).

Specified by:

id in interface IndexIterator

Parameters:

id - the new id for this index iterator.

Returns:

this index iterator.

id

public int id()

Description copied from interface: IndexIterator

Returns the id of this index iterator.

Specified by:

id in interface IndexIterator

Returns:

the id of this index iterator.

See Also:

IndexIterator.id(int)

index

public Index index()

Description copied from interface: IndexIterator

Returns the index over which this iterator is built.

Specified by:

index in interface IndexIterator

Returns:

the index over which this iterator is built.

payload

public Payload payload()

This method is not implemented by this class.

Specified by:

payload in interface IndexIterator

Returns:

the payload associated with the current document.

positionArray

public int[] positionArray()
                    throws IOException

Description copied from interface: IndexIterator

Returns the positions at which the term appears in the current document in an array.

Implementations are allowed to return the same array across different calls to this method.

Specified by:

positionArray in interface IndexIterator

Returns:

an array whose first IndexIterator.count() elements contain the document positions.

Throws:

IOException

positions

public IntIterator positions()
                      throws IOException

Description copied from interface: IndexIterator

Returns the positions at which the term appears in the current document.

Specified by:

positions in interface IndexIterator

Returns:

the positions of the current document in which the current term appears.

Throws:

IOException

positions

public int positions(int[] position)
              throws IOException

Description copied from interface: IndexIterator

Stores the positions at which the term appears in the current document in a given array.

If the array is not large enough (i.e., it does not contain IndexIterator.count() elements), this method will return a negative number (the opposite of the count).

Specified by:

positions in interface IndexIterator

Parameters:

position - an array that will be used to store positions.

Returns:

the count; it will have the sign changed if positions cannot hold all positions.

Throws:

IOException

weight

public IndexIterator weight(double weight)

Description copied from interface: DocumentIterator

Sets the weight of this index iterator.

Specified by:

weight in interface IndexIterator

Specified by:

weight in interface DocumentIterator

Overrides:

weight in class AbstractDocumentIterator

Parameters:

weight - the weight of this index iterator.

Returns:

this document iterator.

See Also:

DocumentIterator.weight(double)

accept

public <T> T accept(DocumentIteratorVisitor<T> visitor)
         throws IOException

Description copied from interface: DocumentIterator

Accepts a visitor.

A document iterator is usually structured as composite, with operators as internal nodes and IndexIterators as leaves. This method implements the visitor pattern.

Specified by:

accept in interface DocumentIterator

Overrides:

accept in class AbstractCompositeDocumentIterator

Parameters:

visitor - the visitor.

Returns:

an object resulting from the visit, or null if the visit was interrupted.

Throws:

IOException

acceptOnTruePaths

public <T> T acceptOnTruePaths(DocumentIteratorVisitor<T> visitor)
                    throws IOException

Description copied from class: AbstractUnionDocumentIterator

Invokes AbstractUnionDocumentIterator.acceptOnTruePaths(DocumentIteratorVisitor) only on component iterators positioned on the current document.

Specified by:

acceptOnTruePaths in interface DocumentIterator

Overrides:

acceptOnTruePaths in class AbstractUnionDocumentIterator

Parameters:

visitor - a visitor.

Returns:

true if the visit should continue.

Throws:

IOException

See Also:

DocumentIterator.accept(DocumentIteratorVisitor), CounterCollectionVisitor

acceptDeep

public <T> T acceptDeep(DocumentIteratorVisitor<T> visitor)
             throws IOException

Throws:

IOException

acceptDeepOnTruePaths

public <T> T acceptDeepOnTruePaths(DocumentIteratorVisitor<T> visitor)
                        throws IOException

Throws:

IOException