Preprocess (marytts-lang-en 5.2.1 API)

java.lang.Object
- marytts.modules.InternalModule
- - marytts.language.en.Preprocess

All Implemented Interfaces:

MaryModule
```
public class Preprocess
extends InternalModule
```
Author:
Tristan Hamilton
Can process following formats:
- cardinal (handled by real number)
- ordinal
- year (as a 4 digit number or any number followed by AD/BC variation)
- currency
- numberandword together
- dashes (read each number singly) or (split into two words)
- underscores
- decimal point, minus symbol (real numbers) also handles %, however Jtokeniser splits % into separate tokens
- time
- dates (in format mm/dd/yyyy)
- acronyms (only split into single characters, never expanded)
- abbreviations (list of known expansions in resource preprocess/abbrev.dat, a properties file separated by whitespace. If an abbrev has two different expansions then the capitalized version comes first, followed by a comma)
- contractions → first check lexicon, if not then → split and check if map contains contraction, if not then just remove apostrophe else → split before apostrophe into two tokens, use map to manually add ph → for 's if word ends in c,f,k,p,t then add ph = s otherwise ph = z
- ampersand &, "at" @ symbol, → symbols
- urls → note that jtokeniser splits off http[s]?://
- number ranges "18-35"
- words without vowels → first check lexicon, if not then separate into single character tokens
- #hashtags
- single "A/a" character → if there is no next token or the next token is punctuation or next token string.length == 1
- should also as a last processing attempt, split by punctuation,symbols,etc. and attempt to process these tokens separately
- durations hours:minutes:seconds(:milliseconds)
- numbers followed by an s
- punctuation → add ph attribute to tag to prevent phonemisation
May include:
- roman numerals

Field Summary

Fields
Modifier and Type Field and Description

protected String cardinalRule

protected String ordinalRule

protected String yearRule
- Fields inherited from class marytts.modules.InternalModule
  logger, state
- Fields inherited from interface marytts.modules.MaryModule
  MODULE_OFFLINE, MODULE_RUNNING

Fields
Modifier and Type	Field and Description
`protected String`	`cardinalRule`
`protected String`	`ordinalRule`
`protected String`	`yearRule`

Constructor Summary

Constructors
Constructor and Description

Preprocess()

Constructors
Constructor and Description
`Preprocess()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`expand(Document doc)` processes a document in mary xml format, from Tokens to Words which can be phonemised.
`protected String`	`expandAbbreviation(String abbrev, boolean isCapital)`
`protected String`	`expandAcronym(String acronym)`
`protected String`	`expandConsonants(String consonants)` add a space between each char of a string
`protected String`	`expandDate(String date)`
`protected String`	`expandDuration(String duration)`
`protected String`	`expandHashtag(String hashtag)`
`protected String`	`expandMoney(String money, String currency)`
`protected String`	`expandNumber(double number)`
`protected String`	`expandNumberS(String numberS)` expands a digit followed by an s.
`protected String`	`expandOrdinal(double number)`
`protected String`	`expandRange(String range)`
`protected String`	`expandRealNumber(String number)`
`protected String`	`expandTime(String time, boolean isNextTokenTime)`
`protected String`	`expandURL(String email)` expand a URL string partially by splitting by @, / and .
`protected String`	`expandWordNumber(String wordnumseq)`
`protected String`	`expandYear(double number)`
`protected String`	`expandYearBCAD(String year)`
`protected static String`	`getOrdinalRuleName(com.ibm.icu.text.RuleBasedNumberFormat rbnf)` Try to extract the rule name for "expand ordinal" from the given RuleBasedNumberFormat.
`protected static String`	`getYearRuleName(com.ibm.icu.text.RuleBasedNumberFormat rbnf)` Try to extract the rule name for "expand year" from the given RuleBasedNumberFormat.
`static Map<Object,Object>`	`loadAbbrevMap()`
`MaryData`	`process(MaryData d)`
`protected String`	`splitContraction(String contraction)`

Methods inherited from class marytts.modules.InternalModule
getInputType, getLocale, getOutputType, getState, inputType, name, outputType, powerOnSelfTest, shutdown, startup

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - cardinalRule
```
protected final String cardinalRule
```
  - ordinalRule
```
protected final String ordinalRule
```
  - yearRule
```
protected final String yearRule
```
- Constructor Detail
  - Preprocess
```
public Preprocess()
```
- Method Detail
  - process
```
public MaryData process(MaryData d)
                 throws Exception
```
    Specified by:
    
    process in interface MaryModule
    
    Overrides:
    
    process in class InternalModule
    
    Throws:
    
    Exception
  - expand
```
protected void expand(Document doc)
               throws ParseException,
                      IOException,
                      MaryConfigurationException
```
    processes a document in mary xml format, from Tokens to Words which can be phonemised.
    
    Parameters:
    
    doc - doc
    
    Throws:
    
    ParseException - parse exception
    
    IOException - IO Exception
    
    MaryConfigurationException - mary configuration exception
  - expandNumber
```
protected String expandNumber(double number)
```
  - expandOrdinal
```
protected String expandOrdinal(double number)
```
  - expandYear
```
protected String expandYear(double number)
```
  - expandDuration
```
protected String expandDuration(String duration)
```
  - expandAcronym
```
protected String expandAcronym(String acronym)
```
  - expandURL
```
protected String expandURL(String email)
```
    expand a URL string partially by splitting by @, / and . symbols (but retaining them)
    
    Parameters:
    
    email - email
    
    Returns:
    
    Arrays.toString(tokens).replaceAll("[,\\]\\[]", "")
  - expandYearBCAD
```
protected String expandYearBCAD(String year)
```
  - expandConsonants
```
protected String expandConsonants(String consonants)
```
    add a space between each char of a string
    
    Parameters:
    
    consonants - consonants
    
    Returns:
    
    Joiner.on(" ").join(Lists.charactersOf(consonants))
  - expandHashtag
```
protected String expandHashtag(String hashtag)
```
  - expandRange
```
protected String expandRange(String range)
```
  - expandNumberS
```
protected String expandNumberS(String numberS)
```
    expands a digit followed by an s. e.g. 7s and 8s and the 60s
    
    Parameters:
    
    numberS - numberS
    
    Returns:
    
    number
  - splitContraction
```
protected String splitContraction(String contraction)
```
  - expandAbbreviation
```
protected String expandAbbreviation(String abbrev,
                                    boolean isCapital)
```
    Parameters:
    
    abbrev - the token to be expanded
    
    isCapital - whether the following token begins with a capital letter
    
    Returns:
    
    abbrev
  - expandDate
```
protected String expandDate(String date)
                     throws ParseException
```
    Throws:
    
    ParseException
  - expandTime
```
protected String expandTime(String time,
                            boolean isNextTokenTime)
```
    Parameters:
    
    time - the token to be expanded
    
    isNextTokenTime - whether the following token contains am or pm
    
    Returns:
    
    theTime
  - expandRealNumber
```
protected String expandRealNumber(String number)
```
  - expandWordNumber
```
protected String expandWordNumber(String wordnumseq)
```
  - expandMoney
```
protected String expandMoney(String money,
                             String currency)
```
  - getOrdinalRuleName
```
protected static String getOrdinalRuleName(com.ibm.icu.text.RuleBasedNumberFormat rbnf)
```
    Try to extract the rule name for "expand ordinal" from the given RuleBasedNumberFormat.
    The rule name is locale sensitive, but usually starts with "%spellout-ordinal".
    
    Parameters:
    
    rbnf - The RuleBasedNumberFormat from where we will try to extract the rule name.
    
    Returns:
    
    The rule name for "ordinal spell out".
  - getYearRuleName
```
protected static String getYearRuleName(com.ibm.icu.text.RuleBasedNumberFormat rbnf)
```
    Try to extract the rule name for "expand year" from the given RuleBasedNumberFormat.
    The rule name is locale sensitive, but usually starts with "%spellout-numbering-year".
    
    Parameters:
    
    rbnf - The RuleBasedNumberFormat from where we will try to extract the rule name.
    
    Returns:
    
    The rule name for "year spell out".
  - loadAbbrevMap
```
public static Map<Object,Object> loadAbbrevMap()
                                        throws IOException
```
    Throws:
    
    IOException

Class Preprocess

Field Summary

Fields inherited from class marytts.modules.InternalModule

Fields inherited from interface marytts.modules.MaryModule

Constructor Summary

Method Summary

Methods inherited from class marytts.modules.InternalModule

Methods inherited from class java.lang.Object

Field Detail

cardinalRule

ordinalRule

yearRule

Constructor Detail

Preprocess

Method Detail

process

expand

expandNumber

expandOrdinal

expandYear

expandDuration

expandAcronym

expandURL

expandYearBCAD

expandConsonants

expandHashtag

expandRange

expandNumberS

splitContraction

expandAbbreviation

expandDate

expandTime

expandRealNumber

expandWordNumber

expandMoney

getOrdinalRuleName

getYearRuleName

loadAbbrevMap