Package ai.djl.modality.nlp.bert

Class BertToken

  • public class BertToken
extends java.lang.Object
    BertToken contains all the information for Bert model after encoding question and paragraph.

    • Constructor Summary

      Constructors 
      Constructor Description
      BertToken​(java.util.List<java.lang.String> tokens, java.util.List<java.lang.Long> tokenType, java.util.List<java.lang.Long> attentionMask, int validLength)
      Creates an instance of BertToken which includes information for Bert model.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.util.List<java.lang.Long> getAttentionMask()
      Gets the mask to avoid performing attention on padding token indices.
      java.util.List<java.lang.String> getTokens()
      Gets the indices of input sequence tokens in the vocabulary.
      java.util.List<java.lang.Long> getTokenTypes()
      Gets segment token indices to indicate first and second portions of the inputs.
      int getValidLength()
      Gets the length of the original sentence which has question and paragraph.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • BertToken

        public BertToken​(java.util.List<java.lang.String> tokens,
                 java.util.List<java.lang.Long> tokenType,
                 java.util.List<java.lang.Long> attentionMask,
                 int validLength)
        Creates an instance of BertToken which includes information for Bert model.
        Parameters:
        tokens - indices of input sequence tokens in the vocabulary.
        tokenType - segment token indices to indicate first and second portions of the inputs.
        attentionMask - mask to avoid performing attention on padding token indices.
        validLength - length that indicates the original input sequence.

    • Method Detail

      • getTokens

        public java.util.List<java.lang.String> getTokens()
        Gets the indices of input sequence tokens in the vocabulary.
        Returns:
        indices of input sequence tokens

      • getTokenTypes

        public java.util.List<java.lang.Long> getTokenTypes()
        Gets segment token indices to indicate first and second portions of the inputs.
        Returns:
        segment token indices

      • getAttentionMask

        public java.util.List<java.lang.Long> getAttentionMask()
        Gets the mask to avoid performing attention on padding token indices.
        Returns:
        mask that performs attention on non-padding token indices

      • getValidLength

        public int getValidLength()
        Gets the length of the original sentence which has question and paragraph.
        Returns:
        length of the original sentence which has question and paragraph