Packages

class TextClassifier extends Serializable

This example use a (pre-trained GloVe embedding) to convert word to vector, and uses it to train a text classification model on the 20 Newsgroup dataset with 20 different categories. This model can achieve around 90% accuracy after 2 epochs training.

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. TextClassifier
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new TextClassifier(param: AbstractTextClassificationParams)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def analyzeTexts(dataRdd: RDD[(String, Float)]): (Map[String, WordMeta], Map[Float, Array[Float]])

    Go through the whole data set to gather some meta info for the tokens.

    Go through the whole data set to gather some meta info for the tokens. Tokens would be discarded if the frequency ranking is less then maxWordsNum

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def buildModel(classNum: Int): Sequential[Float]

    Return a text classification model with the specific num of class

  7. def buildWord2Vec(word2Meta: Map[String, WordMeta]): Map[Float, Array[Float]]

    Load the pre-trained word2Vec

    Load the pre-trained word2Vec

    returns

    A map from word to vector

  8. def buildWord2VecWithIndex(word2Meta: Map[String, Int]): Map[Float, Array[Float]]

    Load the pre-trained word2Vec

    Load the pre-trained word2Vec

    returns

    A map from word to vector

  9. var classNum: Int
  10. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )
  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def getData(sc: SparkContext): (Array[RDD[(Array[Array[Float]], Float)]], Map[String, WordMeta], Map[Float, Array[Float]])

    Create train and val RDDs from input

  16. val gloveDir: String
  17. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  18. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  19. val log: Logger
  20. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  23. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  24. val textDataDir: String
  25. def toString(): String
    Definition Classes
    AnyRef → Any
  26. def train(): Unit

    Start to train the text classification model

  27. def trainFromData(sc: SparkContext, rdds: Array[RDD[(Array[Array[Float]], Float)]]): Module[Float]

    Train the text classification model with train and val RDDs

  28. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped