Package ai.djl.modality.nlp

Interface Vocabulary

  • All Known Implementing Classes:
    DefaultVocabulary
    public interface Vocabulary
    Vocabulary is a collection of tokens. The primary purpose of a vocabulary is the map a token to an index.

    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      boolean contains​(java.lang.String token)
      Check if the vocabulary contains a token.
      long getIndex​(java.lang.String token)
      Returns the index of the given token.
      java.lang.String getToken​(long index)
      Returns the token corresponding to the given index.
      long size()
      Returns the size of the Vocabulary.

    • Method Detail

      • getToken

        java.lang.String getToken​(long index)
        Returns the token corresponding to the given index.
        Parameters:
        index - the index
        Returns:
        the token corresponding to the given index

      • contains

        boolean contains​(java.lang.String token)
        Check if the vocabulary contains a token.
        Parameters:
        token - String token to be checked
        Returns:
        whether this vocabulary contains the token

      • getIndex

        long getIndex​(java.lang.String token)
        Returns the index of the given token.
        Parameters:
        token - the token
        Returns:
        the index of the given token.