hex

Class DataInfo

java.lang.Object

water.Iced<T>

water.Keyed

hex.DataInfo

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, water.Freezable

public class DataInfo
extends water.Keyed

Created by tomasnykodym on 1/29/15. Provides higher level interface for accessing data row-wise. Performs on the fly auto-expansion of categorical variables (to 1 hot encoding) and standardization ( or normalize/demean/descale/none) of predictors and response. Supports sparse data, sparse columns can be transformed to sparse rows on the fly with some (significant) memory overhead, as the data of the whole chunk(s) will be copied.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	DataInfo.Row
class	DataInfo.Rows
static class	DataInfo.TransformType

Nested classes/interfaces inherited from class water.Keyed

water.Keyed.BinarySerializer<X extends water.Keyed>

Field Summary

Fields

Modifier and Type	Field and Description
int[]	_activeCols
water.fvec.Frame	_adaptedFrame
int[][]	_catLvls
int[]	_catMissing
int[]	_catModes
int[]	_catOffsets
int	_cats
boolean	_fold
boolean	_imputeMissing
boolean	_intercept
double[]	_normMul
double[]	_normRespMul
double[]	_normRespSub
double[]	_normSub
double[]	_numMeans
int	_nums
boolean	_offset
int[]	_permutation
DataInfo.TransformType	_predictor_transform
DataInfo.TransformType	_response_transform
int	_responses
boolean	_skipMissing
boolean	_useAllFactorLevels
boolean	_valid
boolean	_weights

Fields inherited from class water.Keyed

_key, EMPTY_KEY_LIST

Constructor Summary

Constructors

Constructor and Description
DataInfo(water.Key selfKey, water.fvec.Frame train, water.fvec.Frame valid, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket)
DataInfo(water.Key selfKey, water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket, boolean weight, boolean offset, boolean fold)
DataInfo(water.Key selfKey, water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket, boolean weight, boolean offset, boolean fold, boolean intercept)

Method Summary

Methods

Modifier and Type	Method and Description
protected long	checksum_impl()
java.lang.String[]	coefNames()
double	computeSparseOffset(double[] coefficients)
DataInfo	deep_clone()
double[]	denormalizeBeta(double[] beta)
void	dropWeights()
DataInfo.Row	extractDenseRow(water.fvec.Chunk[] chunks, int rid, DataInfo.Row row)
DataInfo.Row[]	extractDenseRowsVertical(water.fvec.Chunk[] chunks) Extract (dense) rows from given chunks, one Vec at a time - should be slightly faster than per-row
DataInfo.Row[]	extractSparseRows(water.fvec.Chunk[] chunks, double offset) Extract (sparse) rows from given chunks.
DataInfo	filterExpandedColumns(int[] cols)
int	foldChunkId()
int	fullN()
int	getCategoricalId(int cid, int val)
water.fvec.Vec	getWeightsVec()
static int	imputeCat(water.fvec.Vec v)
int	largestCat()
static DataInfo	makeEmpty(int fullN)
int[]	mapNames(java.lang.String[] names)
DataInfo.Row	newDenseRow()
DataInfo.Row	newDenseRow(double[] numVals)
int	numStart()
int	offsetChunkId()
int	responseChunkId()
DataInfo.Rows	rows(water.fvec.Chunk[] chks)
DataInfo.Rows	rows(water.fvec.Chunk[] chks, boolean sparse)
void	setPredictorTransform(DataInfo.TransformType t)
void	setResponseTransform(DataInfo.TransformType t)
water.fvec.Vec	setWeights(java.lang.String name, water.fvec.Vec vec)
void	unScaleNumericals(double[] in, double[] out) Undo the standardization/normalization of numerical columns
void	unScaleResponses(double[] in, double[] out) Undo the standardization/normalization of numerical columns
void	updateWeightedSigmaAndMean(double[] sigmas, double[] mean)
DataInfo	validDinfo(water.fvec.Frame valid)
int	weightChunkId()

Methods inherited from class water.Keyed

checksum, getBinarySerializer, getPublishedKeys, remove_impl, remove, remove, remove, remove

Methods inherited from class water.Iced

clone, frozenType, read_impl, read, readExternal, readJSON_impl, readJSON, toJsonString, write_impl, write, writeExternal, writeJSON_impl, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_activeCols

public int[] _activeCols

_adaptedFrame

public water.fvec.Frame _adaptedFrame

_responses

public int _responses

_predictor_transform

public DataInfo.TransformType _predictor_transform

_response_transform

public DataInfo.TransformType _response_transform

_useAllFactorLevels

public boolean _useAllFactorLevels

_nums

public int _nums

_cats

public int _cats

_catOffsets

public int[] _catOffsets

_catMissing

public int[] _catMissing

_catModes

public int[] _catModes

_permutation

public int[] _permutation

_normMul

public double[] _normMul

_normSub

public double[] _normSub

_normRespMul

public double[] _normRespMul

_normRespSub

public double[] _normRespSub

_numMeans

public double[] _numMeans

_intercept

public boolean _intercept

_offset

public final boolean _offset

_weights

public boolean _weights

_fold

public final boolean _fold

_skipMissing

public final boolean _skipMissing

_imputeMissing

public final boolean _imputeMissing

_valid

public boolean _valid

_catLvls

public final int[][] _catLvls

Constructor Detail

DataInfo

public DataInfo(water.Key selfKey,
        water.fvec.Frame train,
        water.fvec.Frame valid,
        boolean useAllFactorLevels,
        DataInfo.TransformType predictor_transform,
        boolean skipMissing,
        boolean imputeMissing,
        boolean missingBucket)

DataInfo

public DataInfo(water.Key selfKey,
        water.fvec.Frame train,
        water.fvec.Frame valid,
        int nResponses,
        boolean useAllFactorLevels,
        DataInfo.TransformType predictor_transform,
        DataInfo.TransformType response_transform,
        boolean skipMissing,
        boolean imputeMissing,
        boolean missingBucket,
        boolean weight,
        boolean offset,
        boolean fold)

DataInfo

public DataInfo(water.Key selfKey,
        water.fvec.Frame train,
        water.fvec.Frame valid,
        int nResponses,
        boolean useAllFactorLevels,
        DataInfo.TransformType predictor_transform,
        DataInfo.TransformType response_transform,
        boolean skipMissing,
        boolean imputeMissing,
        boolean missingBucket,
        boolean weight,
        boolean offset,
        boolean fold,
        boolean intercept)

Method Detail

setWeights

public water.fvec.Vec setWeights(java.lang.String name,
                        water.fvec.Vec vec)

dropWeights

public void dropWeights()

responseChunkId

public int responseChunkId()

foldChunkId

public int foldChunkId()

offsetChunkId

public int offsetChunkId()

weightChunkId

public int weightChunkId()

checksum_impl

protected long checksum_impl()

Specified by:

checksum_impl in class water.Keyed

deep_clone

public DataInfo deep_clone()

makeEmpty

public static DataInfo makeEmpty(int fullN)

validDinfo

public DataInfo validDinfo(water.fvec.Frame valid)

denormalizeBeta

public double[] denormalizeBeta(double[] beta)

imputeCat

public static int imputeCat(water.fvec.Vec v)

filterExpandedColumns

public DataInfo filterExpandedColumns(int[] cols)

updateWeightedSigmaAndMean

public void updateWeightedSigmaAndMean(double[] sigmas,
                              double[] mean)

setPredictorTransform

public void setPredictorTransform(DataInfo.TransformType t)

setResponseTransform

public void setResponseTransform(DataInfo.TransformType t)

fullN

public final int fullN()

largestCat

public final int largestCat()

numStart

public final int numStart()

coefNames

public final java.lang.String[] coefNames()

mapNames

public int[] mapNames(java.lang.String[] names)

unScaleNumericals

public final void unScaleNumericals(double[] in,
                     double[] out)

Undo the standardization/normalization of numerical columns

Parameters:

in - input values

out - output values (can be the same as input)

unScaleResponses

public final void unScaleResponses(double[] in,
                    double[] out)

Undo the standardization/normalization of numerical columns

Parameters:

in - input values

out - output values (can be the same as input)

getCategoricalId

public final int getCategoricalId(int cid,
                   int val)

extractDenseRow

public final DataInfo.Row extractDenseRow(water.fvec.Chunk[] chunks,
                           int rid,
                           DataInfo.Row row)

getWeightsVec

public water.fvec.Vec getWeightsVec()

newDenseRow

public DataInfo.Row newDenseRow()

newDenseRow

public DataInfo.Row newDenseRow(double[] numVals)

computeSparseOffset

public double computeSparseOffset(double[] coefficients)

rows

public DataInfo.Rows rows(water.fvec.Chunk[] chks)

rows

public DataInfo.Rows rows(water.fvec.Chunk[] chks,
                 boolean sparse)

extractSparseRows

public final DataInfo.Row[] extractSparseRows(water.fvec.Chunk[] chunks,
                               double offset)

Extract (sparse) rows from given chunks. Note: 0 remains 0 - _normSub of DataInfo isn't used (mean shift during standarization is not reverted) - UNLESS offset is specified (for GLM only) Essentially turns the dataset 90 degrees.

Parameters:

chunks - - chunk of dataset

offset - - adjustment for 0s if running with on-the-fly standardization (i.e. zeros are not really zeros because of centering)

Returns:

array of sparse rows

extractDenseRowsVertical

public final DataInfo.Row[] extractDenseRowsVertical(water.fvec.Chunk[] chunks)

Extract (dense) rows from given chunks, one Vec at a time - should be slightly faster than per-row

Parameters:

chunks - - chunk of dataset

Returns:

array of dense rows