ai.djl.basicdataset

Class TatoebaEnglishFrenchDataset.Builder

java.lang.Object

ai.djl.training.dataset.RandomAccessDataset.BaseBuilder<TatoebaEnglishFrenchDataset.Builder>

ai.djl.basicdataset.TatoebaEnglishFrenchDataset.Builder

Enclosing class:

TatoebaEnglishFrenchDataset

public static class TatoebaEnglishFrenchDataset.Builder
extends ai.djl.training.dataset.RandomAccessDataset.BaseBuilder<TatoebaEnglishFrenchDataset.Builder>

A builder for a TatoebaEnglishFrenchDataset.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	includeValidLength
protected java.util.List<ai.djl.modality.nlp.preprocess.TextProcessor>	sourceTextProcessors
protected java.util.List<ai.djl.modality.nlp.preprocess.TextProcessor>	targetTextProcessors
protected ai.djl.modality.nlp.preprocess.Tokenizer	tokenizer
protected boolean	trainEmbedding
protected ai.djl.modality.nlp.WordEmbedding	wordEmbedding

Fields inherited from class ai.djl.training.dataset.RandomAccessDataset.BaseBuilder

batchifier, device, executor, limit, pipeline, prefetchNumber, sampler, targetPipeline

Method Summary

Modifier and Type	Method and Description
TatoebaEnglishFrenchDataset	build() Builds the TatoebaEnglishFrenchDataset.
TatoebaEnglishFrenchDataset.Builder	optArtifact(ai.djl.repository.Artifact artifact) Sets the optional artifact.
TatoebaEnglishFrenchDataset.Builder	optRepository(ai.djl.repository.Repository repository) Sets the optional repository.
TatoebaEnglishFrenchDataset.Builder	optSourceTextProcessor(ai.djl.modality.nlp.preprocess.TextProcessor sourceTextProcessor) Sets a TextProcessor to be used on the source language input.
TatoebaEnglishFrenchDataset.Builder	optSourceTextProcessors(java.util.List<ai.djl.modality.nlp.preprocess.TextProcessor> sourceTextProcessors) Sets the list of TextProcessors to be used on the source language input.
TatoebaEnglishFrenchDataset.Builder	optTargetTextProcessor(ai.djl.modality.nlp.preprocess.TextProcessor targetTextProcessor) Sets a TextProcessor to be used on the target language input.
TatoebaEnglishFrenchDataset.Builder	optTargetTextProcessors(java.util.List<ai.djl.modality.nlp.preprocess.TextProcessor> targetTextProcessors) Sets the list of TextProcessors to be used on the target language input.
TatoebaEnglishFrenchDataset.Builder	optUsage(ai.djl.training.dataset.Dataset.Usage usage) Sets the optional usage.
TatoebaEnglishFrenchDataset.Builder	self()
TatoebaEnglishFrenchDataset.Builder	setEmbedding(ai.djl.modality.nlp.WordEmbedding wordEmbedding, boolean trainEmbedding) Sets the required implementation of WordEmbedding to source the embeddings from.
TatoebaEnglishFrenchDataset.Builder	setTokenizer(ai.djl.modality.nlp.preprocess.Tokenizer tokenizer) Sets a Tokenizer to tokenize the input sentences.
TatoebaEnglishFrenchDataset.Builder	setValidLength(boolean includeValidLength) Sets the required parameter whether to include the valid length as part of data in the Record.

Methods inherited from class ai.djl.training.dataset.RandomAccessDataset.BaseBuilder

getSampler, optBatchier, optDevice, optExecutor, optLimit, optPipeline, optTargetPipeline, setSampling, setSampling, setSampling

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sourceTextProcessors

protected java.util.List<ai.djl.modality.nlp.preprocess.TextProcessor> sourceTextProcessors

targetTextProcessors

protected java.util.List<ai.djl.modality.nlp.preprocess.TextProcessor> targetTextProcessors

wordEmbedding

protected ai.djl.modality.nlp.WordEmbedding wordEmbedding

trainEmbedding

protected boolean trainEmbedding

includeValidLength

protected boolean includeValidLength

tokenizer

protected ai.djl.modality.nlp.preprocess.Tokenizer tokenizer

Method Detail

self

public TatoebaEnglishFrenchDataset.Builder self()

Specified by:

self in class ai.djl.training.dataset.RandomAccessDataset.BaseBuilder<TatoebaEnglishFrenchDataset.Builder>

optUsage

public TatoebaEnglishFrenchDataset.Builder optUsage(ai.djl.training.dataset.Dataset.Usage usage)

Sets the optional usage.

Parameters:

usage - the usage

Returns:

this builder

optRepository

public TatoebaEnglishFrenchDataset.Builder optRepository(ai.djl.repository.Repository repository)

Sets the optional repository.

Parameters:

repository - the repository

Returns:

this builder

optArtifact

public TatoebaEnglishFrenchDataset.Builder optArtifact(ai.djl.repository.Artifact artifact)

Sets the optional artifact.

Parameters:

artifact - the artifact

Returns:

this builder

setEmbedding

public TatoebaEnglishFrenchDataset.Builder setEmbedding(ai.djl.modality.nlp.WordEmbedding wordEmbedding,
                                                        boolean trainEmbedding)

Sets the required implementation of WordEmbedding to source the embeddings from.

Parameters:

wordEmbedding - the implementation of WordEmbedding to source the embeddings from

trainEmbedding - whether the embeddings need further training

Returns:

this builder

setValidLength

public TatoebaEnglishFrenchDataset.Builder setValidLength(boolean includeValidLength)

Sets the required parameter whether to include the valid length as part of data in the Record.

Parameters:

includeValidLength - whether to include the valid length as part of data

Returns:

this builder

setTokenizer

public TatoebaEnglishFrenchDataset.Builder setTokenizer(ai.djl.modality.nlp.preprocess.Tokenizer tokenizer)

Sets a Tokenizer to tokenize the input sentences.

Parameters:

tokenizer - the Tokenizer to be set

Returns:

this builder

optSourceTextProcessors

public TatoebaEnglishFrenchDataset.Builder optSourceTextProcessors(java.util.List<ai.djl.modality.nlp.preprocess.TextProcessor> sourceTextProcessors)

Sets the list of TextProcessors to be used on the source language input. The order of TextProcessor in the list can make a difference.

Parameters:

sourceTextProcessors - the list of TextProcessors to be set

Returns:

this builder

optSourceTextProcessor

public TatoebaEnglishFrenchDataset.Builder optSourceTextProcessor(ai.djl.modality.nlp.preprocess.TextProcessor sourceTextProcessor)

Sets a TextProcessor to be used on the source language input. The order in which TextProcessor is added can make a difference.

Parameters:

sourceTextProcessor - the TextProcessor to be set

Returns:

this builder

optTargetTextProcessors

public TatoebaEnglishFrenchDataset.Builder optTargetTextProcessors(java.util.List<ai.djl.modality.nlp.preprocess.TextProcessor> targetTextProcessors)

Sets the list of TextProcessors to be used on the target language input. The order of TextProcessor * in the list can make a difference.

Parameters:

targetTextProcessors - the list of TextProcessors to be set

Returns:

this builder

optTargetTextProcessor

public TatoebaEnglishFrenchDataset.Builder optTargetTextProcessor(ai.djl.modality.nlp.preprocess.TextProcessor targetTextProcessor)

Sets a TextProcessor to be used on the target language input. The order in which TextProcessor is added can make a difference.

Parameters:

targetTextProcessor - the TextProcessor to be set

Returns:

this builder

build

public TatoebaEnglishFrenchDataset build()

Builds the TatoebaEnglishFrenchDataset.

Returns:

the TatoebaEnglishFrenchDataset