ai.djl.mxnet.zoo.nlp.qa

Class BertDataParser

java.lang.Object

ai.djl.mxnet.zoo.nlp.qa.BertDataParser

public class BertDataParser
extends java.lang.Object

This is the Utility for pre-processing data for the Bert Model.

You can use this utility to parse vocabulary JSON into Java Array and Dictionary, clean and tokenize sentences, and pad the text.

Constructor Summary

Constructors

Constructor and Description
BertDataParser()

Method Summary

Modifier and Type	Method and Description
static java.util.List<java.lang.String>	formTokens(java.util.List<java.lang.String> question, java.util.List<java.lang.String> answer, int seqLength) Forms tokens with separation that can be used for BERT.
static java.util.List<java.lang.Float>	getTokenTypes(java.util.List<java.lang.String> question, java.util.List<java.lang.String> answer, int seqLength) Forms the token types List [0000...1111...000] where all questions are 0 and answers are 1.
java.util.List<java.lang.String>	idx2token(java.util.List<java.lang.Integer> indexes) Converts indexes to tokens.
static <E> java.util.List<E>	pad(java.util.List<E> tokens, E padItem, int num) Pads the tokens to the required length.
static BertDataParser	parse(java.io.InputStream is) Parses the Vocabulary to JSON files.
java.util.List<java.lang.Integer>	token2idx(java.util.List<java.lang.String> tokens) Converts tokens to indexes.
static java.util.List<java.lang.String>	tokenizer(java.lang.String input) Tokenizes the input, splits all kinds of whitespace, and separates the end of sentence symbol.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BertDataParser

public BertDataParser()

Method Detail

parse

public static BertDataParser parse(java.io.InputStream is)

Parses the Vocabulary to JSON files. [PAD], [CLS], [SEP], [MASK], [UNK] are reserved tokens.

Parameters:

is - the InputStream for the vocab.json

Returns:

an instance of BertDataParser

Throws:

java.lang.IllegalStateException - if failed read from InputStream

tokenizer

public static java.util.List<java.lang.String> tokenizer(java.lang.String input)

Tokenizes the input, splits all kinds of whitespace, and separates the end of sentence symbol.

Parameters:

input - the input string

Returns:

a list of tokens

pad

public static <E> java.util.List<E> pad(java.util.List<E> tokens,
                                        E padItem,
                                        int num)

Pads the tokens to the required length.

Type Parameters:

E - the type of the List

Parameters:

tokens - the input tokens

padItem - the things to pad at the end

num - the total length after padding

Returns:

a list of padded tokens

getTokenTypes

public static java.util.List<java.lang.Float> getTokenTypes(java.util.List<java.lang.String> question,
                                                            java.util.List<java.lang.String> answer,
                                                            int seqLength)

Forms the token types List [0000...1111...000] where all questions are 0 and answers are 1.

Parameters:

question - the question tokens

answer - the answer tokens

seqLength - the sequence length

Returns:

a list of tokenTypes

formTokens

public static java.util.List<java.lang.String> formTokens(java.util.List<java.lang.String> question,
                                                          java.util.List<java.lang.String> answer,
                                                          int seqLength)

Forms tokens with separation that can be used for BERT.

Parameters:

question - the question tokens

answer - the answer tokens

seqLength - the sequence length

Returns:

a list of tokenTypes

token2idx

public java.util.List<java.lang.Integer> token2idx(java.util.List<java.lang.String> tokens)

Converts tokens to indexes.

Parameters:

tokens - the input tokens

Returns:

a list of indexes

idx2token

public java.util.List<java.lang.String> idx2token(java.util.List<java.lang.Integer> indexes)

Converts indexes to tokens.

Parameters:

indexes - the list of indexes

Returns:

a list of tokens