org.datavec.cli.subcommands

Class Vectorize

java.lang.Object

org.datavec.cli.subcommands.Vectorize

All Implemented Interfaces:

SubCommand

public class Vectorize
extends Object
implements SubCommand

Vectorize Command. Based on an input and output format transforms data Design Notes - current version has some artifacts from previous iterations; we're not quite ready to have a unified set of interfaces for the vectorization pipelines - we've still yet to add the timeseries class of pipelines: { timeseries, audio, video } - we've still yet to add the parallelization mechanics to support yarn/spark - after we add the timeseries and parallelization mechaincs, we can refactor into a new inheritance setup Label / RecordReader semantics - the record reader system is designed like the Hadoop RR system; it doenst know anything about contents of files - down the road we may want to change to a system where the label is treated specifically separately from the raw Collection vector - this would be a cleaner design, but in v1 (w the impending book) we dont have time for this semantics change - it might be pragmatic to use a flexible vector schema, ala CSV's vector schema system, for other data types. not sure yet. - Current breakdown for each pipeline n. Type: Input Format > Label Mechanics in Collection // notes 1. Image: ImageInputFormat > { [array of doubles], directoryLabelID } // image data, then the directory indexed as an ID int 2. CSV: LineInputFormat > { string } // label is defined in vector schema 3. Text: (cli's) TextInputFormat > { string, dirLabelString } // label is second string in Collection 4. MNIST: MnistInputFormat > { [array of doubles], classIndexID } // image data, then the class indexed as an ID int

Author:

Josh Patterson, Adam Gibson

Field Summary

Fields

Modifier and Type	Field and Description
protected String[]	args
Properties	configProps
String	configurationFile
static String	DEFAULT_INPUT_FORMAT_CLASSNAME
static String	DEFAULT_OUTPUT_FORMAT_CLASSNAME
static String	DEFAULT_VECTORIZATION_ENGINE_CLASSNAME
static String	INPUT_FORMAT
static String	NORMALIZE_DATA_FLAG
boolean	normalizeData
static String	OUTPUT_FILENAME_KEY
static String	OUTPUT_FORMAT
String	outputVectorFilename
static String	PRINT_STATS_FLAG
static String	SHUFFLE_DATA_FLAG
boolean	validCommandLineParameters
static String	VECTORIZATION_ENGINE

Constructor Summary

Constructors

Constructor and Description
Vectorize()
Vectorize(String[] args)

Method Summary

Modifier and Type	Method and Description
InputFormat	createInputFormat() Creates an input format
OutputFormat	createOutputFormat()
VectorizationEngine	createVectorizationEngine() Creates an input format
void	debugLoadedConfProperties() Dont change print stuff, its part of application console output UI
void	execute() Execute the input
void	loadConfigFile()
static void	printUsage()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OUTPUT_FILENAME_KEY

public static final String OUTPUT_FILENAME_KEY

See Also:

Constant Field Values

INPUT_FORMAT

public static final String INPUT_FORMAT

See Also:

Constant Field Values

DEFAULT_INPUT_FORMAT_CLASSNAME

public static final String DEFAULT_INPUT_FORMAT_CLASSNAME

See Also:

Constant Field Values

OUTPUT_FORMAT

public static final String OUTPUT_FORMAT

See Also:

Constant Field Values

DEFAULT_OUTPUT_FORMAT_CLASSNAME

public static final String DEFAULT_OUTPUT_FORMAT_CLASSNAME

See Also:

Constant Field Values

VECTORIZATION_ENGINE

public static final String VECTORIZATION_ENGINE

See Also:

Constant Field Values

DEFAULT_VECTORIZATION_ENGINE_CLASSNAME

public static final String DEFAULT_VECTORIZATION_ENGINE_CLASSNAME

See Also:

Constant Field Values

NORMALIZE_DATA_FLAG

public static final String NORMALIZE_DATA_FLAG

See Also:

Constant Field Values

SHUFFLE_DATA_FLAG

public static final String SHUFFLE_DATA_FLAG

See Also:

Constant Field Values

PRINT_STATS_FLAG

public static final String PRINT_STATS_FLAG

See Also:

Constant Field Values

args

protected String[] args

validCommandLineParameters

public boolean validCommandLineParameters

configurationFile

public String configurationFile

configProps

public Properties configProps

outputVectorFilename

public String outputVectorFilename

normalizeData

public boolean normalizeData

Constructor Detail

Vectorize

public Vectorize()

Vectorize

public Vectorize(String[] args)

Parameters:

args - arguments for command

Method Detail

loadConfigFile

public void loadConfigFile()
                    throws IOException

Throws:

IOException

debugLoadedConfProperties

public void debugLoadedConfProperties()

Dont change print stuff, its part of application console output UI

printUsage

public static void printUsage()

execute

public void execute()
             throws Exception

Description copied from interface: SubCommand

Execute the input

Specified by:

execute in interface SubCommand

Throws:

Exception

createInputFormat

public InputFormat createInputFormat()

Creates an input format

Returns:

createOutputFormat

public OutputFormat createOutputFormat()

createVectorizationEngine

public VectorizationEngine createVectorizationEngine()

Creates an input format

Returns:

VectorizationEngine