com.hankcs.hanlp.tokenizer.lexical

Class AbstractLexicalAnalyzer

java.lang.Object

com.hankcs.hanlp.seg.Segment

com.hankcs.hanlp.seg.CharacterBasedSegment

com.hankcs.hanlp.tokenizer.lexical.AbstractLexicalAnalyzer

All Implemented Interfaces:

LexicalAnalyzer, NERecognizer, POSTagger, Segmenter

Direct Known Subclasses:

CRFLexicalAnalyzer, HMMLexicalAnalyzer, PerceptronLexicalAnalyzer

public class AbstractLexicalAnalyzer
extends CharacterBasedSegment
implements LexicalAnalyzer

词法分析器基类（中文分词、词性标注和命名实体识别）

Author:

hankcs

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	enableRuleBasedSegment 是否执行规则分词（英文数字标点等的规则预处理）。规则永远是丑陋的，默认关闭。
protected NERecognizer	neRecognizer
protected POSTagger	posTagger
protected Segmenter	segmenter
protected static byte[]	typeTable 字符类型表

Fields inherited from class com.hankcs.hanlp.seg.Segment

config, customDictionary

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractLexicalAnalyzer()
	AbstractLexicalAnalyzer(Segmenter segmenter)
	AbstractLexicalAnalyzer(Segmenter segmenter, POSTagger posTagger)
	AbstractLexicalAnalyzer(Segmenter segmenter, POSTagger posTagger, NERecognizer neRecognizer)

Method Summary

Modifier and Type	Method and Description
protected boolean	acceptCustomWord(int begin, int end, CoreDictionary.Attribute value) Deprecated. 自1.6.7起废弃，强制模式下为最长匹配，否则按分词结果合并
Sentence	analyze(String sentence) 对句子进行词法分析
protected List<CoreDictionary.Attribute>	combineWithCustomDictionary(List<String> vertexList) 使用用户词典合并粗分结果
AbstractLexicalAnalyzer	enableRuleBasedSegment(boolean enableRuleBasedSegment) 是否执行规则分词（英文数字标点等的规则预处理）。规则永远是丑陋的，默认关闭。
NERTagSet	getNERTagSet()
String[]	recognize(String[] wordArray, String[] posArray) 命名实体识别
protected List<Term>	roughSegSentence(char[] sentence) 单纯的分词模型实现该方法，仅输出词
List<String>	segment(String sentence) 中文分词
List<String>	segment(String sentence, String normalized) 这个方法会查询用户词典
void	segment(String sentence, String normalized, List<String> wordList)
protected void	segment(String sentence, String normalized, List<String> wordList, List<CoreDictionary.Attribute> attributeList) 分词
protected void	segmentAfterRule(String sentence, String normalized, List<String> wordList) 丑陋的规则系统
protected List<Term>	segSentence(char[] sentence) 以下方法用于纯分词模型 分词、词性标注联合模型则直接重载segSentence
String[]	tag(List<String> wordList) 词性标注
String[]	tag(String... words) 词性标注

Methods inherited from class com.hankcs.hanlp.seg.CharacterBasedSegment

guessAttribute, toVertexList

Methods inherited from class com.hankcs.hanlp.seg.Segment

atomSegment, combineByCustomDictionary, combineByCustomDictionary, combineByCustomDictionary, combineByCustomDictionary, convert, enableAllNamedEntityRecognize, enableCustomDictionary, enableCustomDictionary, enableCustomDictionaryForcing, enableIndexMode, enableIndexMode, enableJapaneseNameRecognize, enableMultithreading, enableMultithreading, enableNameRecognize, enableNumberQuantifierRecognize, enableOffset, enableOrganizationRecognize, enablePartOfSpeechTagging, enablePlaceRecognize, enableTranslatedNameRecognize, mergeNumberQuantifier, quickAtomSegment, seg, seg, seg2sentence, seg2sentence, simpleAtomSegment

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

segmenter

protected Segmenter segmenter

posTagger

protected POSTagger posTagger

neRecognizer

protected NERecognizer neRecognizer

typeTable

protected static byte[] typeTable

字符类型表

enableRuleBasedSegment

protected boolean enableRuleBasedSegment

是否执行规则分词（英文数字标点等的规则预处理）。规则永远是丑陋的，默认关闭。

Constructor Detail

AbstractLexicalAnalyzer

protected AbstractLexicalAnalyzer()

AbstractLexicalAnalyzer

public AbstractLexicalAnalyzer(Segmenter segmenter)

AbstractLexicalAnalyzer

public AbstractLexicalAnalyzer(Segmenter segmenter,
                               POSTagger posTagger)

AbstractLexicalAnalyzer

public AbstractLexicalAnalyzer(Segmenter segmenter,
                               POSTagger posTagger,
                               NERecognizer neRecognizer)

Method Detail

segment

protected void segment(String sentence,
                       String normalized,
                       List<String> wordList,
                       List<CoreDictionary.Attribute> attributeList)

分词

Parameters:

sentence - 文本

normalized - 正规化后的文本

wordList - 储存单词列表

attributeList - 储存用户词典中的词性，设为null表示不查询用户词典

segment

public void segment(String sentence,
                    String normalized,
                    List<String> wordList)

Specified by:

segment in interface Segmenter

segment

public List<String> segment(String sentence)

中文分词

Specified by:

segment in interface Segmenter

Parameters:

sentence -

Returns:

recognize

public String[] recognize(String[] wordArray,
                          String[] posArray)

Description copied from interface: NERecognizer

命名实体识别

Specified by:

recognize in interface NERecognizer

Parameters:

wordArray - 单词

posArray - 词性

Returns:

BMES-NER标签

tag

public String[] tag(String... words)

Description copied from interface: POSTagger

词性标注

Specified by:

tag in interface POSTagger

Parameters:

words - 单词

Returns:

词性数组

tag

public String[] tag(List<String> wordList)

Description copied from interface: POSTagger

词性标注

Specified by:

tag in interface POSTagger

Parameters:

wordList - 单词

Returns:

词性数组

getNERTagSet

public NERTagSet getNERTagSet()

Specified by:

getNERTagSet in interface NERecognizer

analyze

public Sentence analyze(String sentence)

Description copied from interface: LexicalAnalyzer

对句子进行词法分析

Specified by:

analyze in interface LexicalAnalyzer

Parameters:

sentence - 纯文本句子

Returns:

HanLP定义的结构化句子

segment

public List<String> segment(String sentence,
                            String normalized)

这个方法会查询用户词典

Parameters:

sentence -

normalized -

Returns:

acceptCustomWord

protected boolean acceptCustomWord(int begin,
                                   int end,
                                   CoreDictionary.Attribute value)

Deprecated. 自1.6.7起废弃，强制模式下为最长匹配，否则按分词结果合并

分词时查询到一个用户词典中的词语，此处控制是否接受它

Parameters:

begin - 起始位置

end - 终止位置

value - 词性

Returns:

true 表示接受

roughSegSentence

protected List<Term> roughSegSentence(char[] sentence)

Description copied from class: CharacterBasedSegment

单纯的分词模型实现该方法，仅输出词

Specified by:

roughSegSentence in class CharacterBasedSegment

Returns:

segSentence

protected List<Term> segSentence(char[] sentence)

Description copied from class: CharacterBasedSegment

以下方法用于纯分词模型 分词、词性标注联合模型则直接重载segSentence

Overrides:

segSentence in class CharacterBasedSegment

Parameters:

sentence - 待分词句子

Returns:

单词列表

segmentAfterRule

protected void segmentAfterRule(String sentence,
                                String normalized,
                                List<String> wordList)

丑陋的规则系统

Parameters:

sentence -

normalized -

wordList -

combineWithCustomDictionary

protected List<CoreDictionary.Attribute> combineWithCustomDictionary(List<String> vertexList)

使用用户词典合并粗分结果

Parameters:

vertexList - 粗分结果

Returns:

合并后的结果

enableRuleBasedSegment

public AbstractLexicalAnalyzer enableRuleBasedSegment(boolean enableRuleBasedSegment)

是否执行规则分词（英文数字标点等的规则预处理）。规则永远是丑陋的，默认关闭。

Parameters:

enableRuleBasedSegment - 是否激活

Returns:

词法分析器对象