Package ai.djl.basicdataset.nlp

Class StanfordQuestionAnsweringDataset

java.lang.Object

ai.djl.training.dataset.RandomAccessDataset

ai.djl.basicdataset.nlp.TextDataset

ai.djl.basicdataset.nlp.StanfordQuestionAnsweringDataset

All Implemented Interfaces:

RawDataset<java.lang.Object>, ai.djl.training.dataset.Dataset

public class StanfordQuestionAnsweringDataset
extends TextDataset
implements RawDataset<java.lang.Object>

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	StanfordQuestionAnsweringDataset.Builder	A builder for a StanfordQuestionAnsweringDataset.

Nested classes/interfaces inherited from class ai.djl.basicdataset.nlp.TextDataset

TextDataset.Sample

Nested classes/interfaces inherited from class ai.djl.training.dataset.RandomAccessDataset

ai.djl.training.dataset.RandomAccessDataset.BaseBuilder<T extends ai.djl.training.dataset.RandomAccessDataset.BaseBuilder<T>>

Nested classes/interfaces inherited from interface ai.djl.training.dataset.Dataset

ai.djl.training.dataset.Dataset.Usage

Field Summary

Fields inherited from class ai.djl.basicdataset.nlp.TextDataset

manager, mrl, prepared, samples, sourceTextData, targetTextData, usage

Fields inherited from class ai.djl.training.dataset.RandomAccessDataset

dataBatchifier, device, labelBatchifier, limit, pipeline, prefetchNumber, sampler, targetPipeline

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	StanfordQuestionAnsweringDataset(StanfordQuestionAnsweringDataset.Builder builder)	Creates a new instance of StanfordQuestionAnsweringDataset.

Method Summary

Modifier and Type	Method	Description
protected long	availableSize()	Returns the number of records available to be read in this Dataset.
static StanfordQuestionAnsweringDataset.Builder	builder()	Creates a new builder to build a StanfordQuestionAnsweringDataset.
ai.djl.training.dataset.Record	get(ai.djl.ndarray.NDManager manager, long index)	Gets the Record for the given index from the dataset.
java.lang.Object	getData()	Get data from the SQuAD dataset.
void	prepare(ai.djl.util.Progress progress)	Prepares the dataset for use with tracked progress.
protected void	preprocess(java.util.List<java.lang.String> newTextData, boolean source)	Performs pre-processing steps on text data such as tokenising, applying TextProcessors, creating vocabulary, and word embeddings.

Methods inherited from class ai.djl.basicdataset.nlp.TextDataset

getProcessedText, getRawText, getSamples, getTextEmbedding, getVocabulary

Methods inherited from class ai.djl.training.dataset.RandomAccessDataset

getData, getData, getData, getData, randomSplit, size, subDataset, toArray

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface ai.djl.training.dataset.Dataset

getData, getData, prepare

Constructor Detail

StanfordQuestionAnsweringDataset

protected StanfordQuestionAnsweringDataset(StanfordQuestionAnsweringDataset.Builder builder)

Creates a new instance of StanfordQuestionAnsweringDataset.

Parameters:

builder - the builder object to build from

Method Detail

builder

public static StanfordQuestionAnsweringDataset.Builder builder()

Creates a new builder to build a StanfordQuestionAnsweringDataset.

Returns:

a new builder

prepare

public void prepare(ai.djl.util.Progress progress)
             throws java.io.IOException,
                    ai.djl.modality.nlp.embedding.EmbeddingException

Prepares the dataset for use with tracked progress. In this method the JSON file will be parsed. The question, context, title will be added to sourceTextData and the answers will be added to targetTextData. Both of them will then be preprocessed.

Specified by:

prepare in interface ai.djl.training.dataset.Dataset

Parameters:

progress - the progress tracker

Throws:

java.io.IOException - for various exceptions depending on the dataset

ai.djl.modality.nlp.embedding.EmbeddingException - if there are exceptions during the embedding process

get

public ai.djl.training.dataset.Record get(ai.djl.ndarray.NDManager manager,
                                          long index)

Gets the Record for the given index from the dataset.

Specified by:

get in class ai.djl.training.dataset.RandomAccessDataset

Parameters:

manager - the manager used to create the arrays

index - the index of the requested data item

Returns:

a Record that contains the data and label of the requested data item. The data NDList contains three NDArrays representing the embedded title, context and question, which are named accordingly. The label NDList contains multiple NDArrays corresponding to each embedded answer.

availableSize

protected long availableSize()

Returns the number of records available to be read in this Dataset. In this implementation, the actual size of available records are the size of questionInfoList.

Specified by:

availableSize in class ai.djl.training.dataset.RandomAccessDataset

Returns:

the number of records available to be read in this Dataset

getData

public java.lang.Object getData()
                         throws java.io.IOException

Get data from the SQuAD dataset. This method will directly return the whole dataset as an object

Specified by:

getData in interface RawDataset<java.lang.Object>

Returns:

an object of Object class in the structure of JSON, e.g. Map<String, List<Map<...>>>

Throws:

java.io.IOException - when IO operation fails in loading a resource

preprocess

protected void preprocess(java.util.List<java.lang.String> newTextData,
                          boolean source)
                   throws ai.djl.modality.nlp.embedding.EmbeddingException

Performs pre-processing steps on text data such as tokenising, applying TextProcessors, creating vocabulary, and word embeddings. Since the record number in this dataset is not equivalent to the length of sourceTextData and targetTextData, the limit should be processed.

Overrides:

preprocess in class TextDataset

Parameters:

newTextData - list of all unprocessed sentences in the dataset

source - whether the text data provided is source or target

Throws:

ai.djl.modality.nlp.embedding.EmbeddingException - if there is an error while embedding input