com.hankcs.hanlp.tokenizer

Class StandardTokenizer

java.lang.Object

com.hankcs.hanlp.tokenizer.StandardTokenizer

public class StandardTokenizer
extends Object

标准分词器

Author:

hankcs

Field Summary

Fields

Modifier and Type	Field and Description
static Segment	SEGMENT 预置分词器

Constructor Summary

Constructors

Constructor and Description
StandardTokenizer()

Method Summary

Modifier and Type	Method and Description
static List<List<Term>>	seg2sentence(String text) 切分为句子形式
static List<List<Term>>	seg2sentence(String text, boolean shortest) 分词断句 输出句子形式
static List<Term>	segment(char[] text) 分词
static List<Term>	segment(String text) 分词

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SEGMENT

public static final Segment SEGMENT

预置分词器

Constructor Detail

StandardTokenizer

public StandardTokenizer()

Method Detail

segment

public static List<Term> segment(String text)

分词

Parameters:

text - 文本

Returns:

分词结果

segment

public static List<Term> segment(char[] text)

分词

Parameters:

text - 文本

Returns:

分词结果

seg2sentence

public static List<List<Term>> seg2sentence(String text)

切分为句子形式

Parameters:

text - 文本

Returns:

句子列表

seg2sentence

public static List<List<Term>> seg2sentence(String text,
                                            boolean shortest)

分词断句 输出句子形式

Parameters:

text - 待分词句子

shortest - 是否断句为最细的子句（将逗号也视作分隔符）

Returns:

句子列表，每个句子由一个单词列表组成