com.hankcs.hanlp.model.perceptron.utility

Class Utility

java.lang.Object

com.hankcs.hanlp.model.perceptron.utility.Utility

public class Utility
extends Object

Author:

hankcs

Constructor Summary

Constructors

Constructor and Description
Utility()

Method Summary

Modifier and Type	Method and Description
static Set<String>	combineNER(String[] nerArray, NERTagSet tagSet)
static void	convertPKUtoCWS(String inputFolder, String outputFile, int begin, int end) 将人民日报格式的分词语料转化为空格分割的语料
static void	convertPKUtoPOS(String inputFolder, String outputFile, int begin, int end) 将人民日报格式的分词语料转化为空格分割的语料
static List<String[]>	convertSentenceToNER(Sentence sentence, NERTagSet tagSet) 将句子转换为 （单词，词性，NER标签）三元组
static int[]	evaluateCWS(String developFile, PerceptronSegmenter segmenter)
static Map<String,double[]>	evaluateNER(NERecognizer recognizer, String goldFile)
static void	normalize(Sentence sentence)
static String	normalize(String text)
static double[]	prf(int[] stat)
static double[]	prf(int goldTotal, int predTotal, int correct)
static void	printNERScore(Map<String,double[]> scores)
static String[][]	reshapeNER(List<String[]> ner)
static void	reshapeNER(List<String[]> collector, String[] wordArray, String[] posArray, String[] tagArray)
static void	shuffleArray(int[] ar) Fisher–Yates shuffle
static <T> void	shuffleArray(T[] ar)
static String[]	toWordArray(List<Word> wordList)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Utility

public Utility()

Method Detail

prf

public static double[] prf(int[] stat)

prf

public static double[] prf(int goldTotal,
                           int predTotal,
                           int correct)

shuffleArray

public static void shuffleArray(int[] ar)

Fisher–Yates shuffle

Parameters:

ar -

shuffleArray

public static <T> void shuffleArray(T[] ar)

normalize

public static String normalize(String text)

convertPKUtoCWS

public static void convertPKUtoCWS(String inputFolder,
                                   String outputFile,
                                   int begin,
                                   int end)
                            throws IOException

将人民日报格式的分词语料转化为空格分割的语料

Parameters:

inputFolder - 输入人民日报语料的上级目录(该目录下的所有文件都是一篇人民日报分词文章)

outputFile - 输出一整个CRF训练格式的语料

begin - 取多少个文档之后

end -

Throws:

IOException - 转换过程中的IO异常

convertPKUtoPOS

public static void convertPKUtoPOS(String inputFolder,
                                   String outputFile,
                                   int begin,
                                   int end)
                            throws IOException

将人民日报格式的分词语料转化为空格分割的语料

Parameters:

inputFolder - 输入人民日报语料的上级目录(该目录下的所有文件都是一篇人民日报分词文章)

outputFile - 输出一整个CRF训练格式的语料

begin - 取多少个文档之后

end -

Throws:

IOException - 转换过程中的IO异常

toWordArray

public static String[] toWordArray(List<Word> wordList)

evaluateCWS

public static int[] evaluateCWS(String developFile,
                                PerceptronSegmenter segmenter)
                         throws IOException

Throws:

IOException

convertSentenceToNER

public static List<String[]> convertSentenceToNER(Sentence sentence,
                                                  NERTagSet tagSet)

将句子转换为 （单词，词性，NER标签）三元组

Parameters:

sentence -

tagSet -

Returns:

normalize

public static void normalize(Sentence sentence)

evaluateNER

public static Map<String,double[]> evaluateNER(NERecognizer recognizer,
                                               String goldFile)

combineNER

public static Set<String> combineNER(String[] nerArray,
                                     NERTagSet tagSet)

reshapeNER

public static String[][] reshapeNER(List<String[]> ner)

reshapeNER

public static void reshapeNER(List<String[]> collector,
                              String[] wordArray,
                              String[] posArray,
                              String[] tagArray)

printNERScore

public static void printNERScore(Map<String,double[]> scores)