org.datavec.arrow

Class ArrowConverter

java.lang.Object

org.datavec.arrow.ArrowConverter

public class ArrowConverter
extends Object

Interop between datavec primitives and arrow. This allows for datavec schemas and primitives to be converted to the arrow format.

Author:

Adam Gibson

Constructor Summary

Constructors

Constructor and Description
ArrowConverter()

Method Summary

Modifier and Type	Method and Description
static org.apache.arrow.vector.types.pojo.Field	booleanField(String name) Shortcut
static org.apache.arrow.vector.BitVector	booleanVectorOf(org.apache.arrow.memory.BufferAllocator allocator, String name, int length)
static org.nd4j.linalg.api.ndarray.INDArray	convertArrowVector(org.apache.arrow.vector.FieldVector fieldVector, ColumnType type) Convert a field vector to a column vector
static List<org.apache.arrow.vector.FieldVector>	convertToArrowVector(org.nd4j.linalg.api.ndarray.INDArray from, List<String> name, ColumnType type, org.apache.arrow.memory.BufferAllocator bufferAllocator) Convert an INDArray to a list of column vectors or a singleton list when either a row vector or a column vector
static org.apache.arrow.vector.types.pojo.Field	doubleField(String name) Shortcut method for creating a double field with 64 bit floating point
static org.apache.arrow.vector.Float8Vector	doubleVectorOf(org.apache.arrow.memory.BufferAllocator allocator, String name, int length)
static org.apache.arrow.vector.types.pojo.Field	field(String name, org.apache.arrow.vector.types.pojo.ArrowType arrowType) Shortcut method for returning a field given an arrow type and name with no sub fields
static org.apache.arrow.vector.types.pojo.Field	floatField(String name) Shortcut method for creating a double field with 32 bit floating point
static org.apache.arrow.vector.Float4Vector	floatVectorOf(org.apache.arrow.memory.BufferAllocator allocator, String name, int length)
static Writable	fromEntry(int item, org.apache.arrow.vector.FieldVector from, ColumnType columnType) Based on an input ColumnType get an entry from a FieldVector
static org.apache.arrow.vector.types.pojo.Field	getFieldForColumn(String name, ColumnType columnType) Create a field given the input ColumnType and name
static org.apache.arrow.vector.types.pojo.Field	intField(String name) Shortcut method for creating a double field with 32 bit integer field
static org.apache.arrow.vector.IntVector	intVectorOf(org.apache.arrow.memory.BufferAllocator allocator, String name, int length)
static org.apache.arrow.vector.types.pojo.Field	longField(String name) Shortcut method for creating a long field with 64 bit long field
static org.apache.arrow.vector.BigIntVector	longVectorOf(org.apache.arrow.memory.BufferAllocator allocator, String name, int length)
static org.apache.arrow.vector.VarBinaryVector	ndarrayVectorOf(org.apache.arrow.memory.BufferAllocator allocator, String name, int length) Create an ndarray vector that stores structs of INDArray based on the org.apache.arrow.flatbuf.Tensor format
static org.apache.arrow.vector.dictionary.DictionaryProvider	providerForVectors(List<org.apache.arrow.vector.FieldVector> vectors, List<org.apache.arrow.vector.types.pojo.Field> fields) Provide a value look up dictionary based on the given set of input FieldVector s for reading and writing to arrow streams
static org.nd4j.linalg.primitives.Pair<Schema,ArrowWritableRecordBatch>	readFromBytes(byte[] input) Read a datavec schema and record set from the given bytes (usually expected to be an arrow format file)
static org.nd4j.linalg.primitives.Pair<Schema,ArrowWritableRecordBatch>	readFromFile(File input) Read a datavec schema and record set from the given arrow file.
static org.nd4j.linalg.primitives.Pair<Schema,ArrowWritableRecordBatch>	readFromFile(FileInputStream input) Read a datavec schema and record set from the given arrow file.
static void	setValue(ColumnType columnType, org.apache.arrow.vector.FieldVector fieldVector, Object value, int row) Set the value of the specified column vector at the specified row based on the given value.
static org.apache.arrow.vector.types.pojo.Field	stringField(String name)
static org.apache.arrow.vector.VarCharVector	stringVectorOf(org.apache.arrow.memory.BufferAllocator allocator, String name, int length)
static org.apache.arrow.vector.TimeStampMilliVector	timeVectorOf(org.apache.arrow.memory.BufferAllocator allocator, String name, int length)
static org.nd4j.linalg.api.ndarray.INDArray	toArray(ArrowWritableRecordBatch arrowWritableRecordBatch) Create an ndarray from a matrix.
static org.nd4j.linalg.api.ndarray.INDArray	toArray(ArrowWritableRecordTimeSeriesBatch arrowWritableRecordBatch) Create an ndarray from a matrix.
static List<org.apache.arrow.vector.FieldVector>	toArrowColumns(org.apache.arrow.memory.BufferAllocator bufferAllocator, Schema schema, List<List<Writable>> dataVecRecord) Given a buffer allocator and datavec schema, convert the passed in batch of records to a set of arrow columns
static List<org.apache.arrow.vector.FieldVector>	toArrowColumnsString(org.apache.arrow.memory.BufferAllocator bufferAllocator, Schema schema, List<List<String>> dataVecRecord) Convert a set of input strings to arrow columns
static List<org.apache.arrow.vector.FieldVector>	toArrowColumnsStringSingle(org.apache.arrow.memory.BufferAllocator bufferAllocator, Schema schema, List<String> dataVecRecord) Convert a set of input strings to arrow columns
static List<org.apache.arrow.vector.FieldVector>	toArrowColumnsStringTimeSeries(org.apache.arrow.memory.BufferAllocator bufferAllocator, Schema schema, List<List<List<String>>> dataVecRecord) Convert a set of input strings to arrow columns for a time series.
static List<org.apache.arrow.vector.FieldVector>	toArrowColumnsTimeSeries(org.apache.arrow.memory.BufferAllocator bufferAllocator, Schema schema, List<List<List<Writable>>> dataVecRecord) Convert a set of input strings to arrow columns for a time series.
static <T> List<org.apache.arrow.vector.FieldVector>	toArrowColumnsTimeSeriesHelper(org.apache.arrow.memory.BufferAllocator bufferAllocator, Schema schema, List<List<List<T>>> dataVecRecord) Convert a set of input strings to arrow columns for a time series.
static org.apache.arrow.vector.types.pojo.Schema	toArrowSchema(Schema schema) Convert a data vec Schema to an arrow Schema
static ArrowWritableRecordBatch	toArrowWritables(List<org.apache.arrow.vector.FieldVector> fieldVectors, Schema schema) Convert the input field vectors (the input data) and the given schema to a proper list of writables.
static List<Writable>	toArrowWritablesSingle(List<org.apache.arrow.vector.FieldVector> fieldVectors, Schema schema) Return a singular record based on the converted writables result.
static List<List<List<Writable>>>	toArrowWritablesTimeSeries(List<org.apache.arrow.vector.FieldVector> fieldVectors, Schema schema, int timeSeriesLength) Convert the input field vectors (the input data) and the given schema to a proper list of writables.
static Schema	toDatavecSchema(org.apache.arrow.vector.types.pojo.Schema schema) Convert an Schema to a datavec Schema
static org.apache.arrow.vector.BitVector	vectorFor(org.apache.arrow.memory.BufferAllocator allocator, String name, boolean[] data)
static org.apache.arrow.vector.TimeStampMilliVector	vectorFor(org.apache.arrow.memory.BufferAllocator allocator, String name, Date[] data)
static org.apache.arrow.vector.Float8Vector	vectorFor(org.apache.arrow.memory.BufferAllocator allocator, String name, double[] data)
static org.apache.arrow.vector.Float4Vector	vectorFor(org.apache.arrow.memory.BufferAllocator allocator, String name, float[] data)
static org.apache.arrow.vector.VarBinaryVector	vectorFor(org.apache.arrow.memory.BufferAllocator bufferAllocator, String name, org.nd4j.linalg.api.ndarray.INDArray[] data) Returns a vector representing a tensor view of each ndarray.
static org.apache.arrow.vector.IntVector	vectorFor(org.apache.arrow.memory.BufferAllocator allocator, String name, int[] data)
static org.apache.arrow.vector.BigIntVector	vectorFor(org.apache.arrow.memory.BufferAllocator allocator, String name, long[] data)
static org.apache.arrow.vector.VarCharVector	vectorFor(org.apache.arrow.memory.BufferAllocator allocator, String name, String[] data)
static void	writeRecordBatchTo(org.apache.arrow.memory.BufferAllocator bufferAllocator, List<List<Writable>> recordBatch, Schema inputSchema, OutputStream outputStream) Write the records to the given output stream
static void	writeRecordBatchTo(List<List<Writable>> recordBatch, Schema inputSchema, OutputStream outputStream) Write the records to the given output stream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ArrowConverter

public ArrowConverter()

Method Detail

toArray

public static org.nd4j.linalg.api.ndarray.INDArray toArray(ArrowWritableRecordTimeSeriesBatch arrowWritableRecordBatch)

Create an ndarray from a matrix. The included batch must be all the same number of rows in order to work. The reason for this is INDArray must be all the same dimensions. Note that the input columns must also be numerical. If they aren't numerical already, consider using an TransformProcess to transform the data output from ArrowRecordReader in to the proper format for usage with this method for direct conversion.

Parameters:

arrowWritableRecordBatch - the incoming batch. This is typically output from an ArrowRecordReader

Returns:

an INDArray representative of the input data

toArray

public static org.nd4j.linalg.api.ndarray.INDArray toArray(ArrowWritableRecordBatch arrowWritableRecordBatch)

Create an ndarray from a matrix. The included batch must be all the same number of rows in order to work. The reason for this is INDArray must be all the same dimensions. Note that the input columns must also be numerical. If they aren't numerical already, consider using an TransformProcess to transform the data output from ArrowRecordReader in to the proper format for usage with this method for direct conversion.

Parameters:

arrowWritableRecordBatch - the incoming batch. This is typically output from an ArrowRecordReader

Returns:

an INDArray representative of the input data

convertArrowVector

public static org.nd4j.linalg.api.ndarray.INDArray convertArrowVector(org.apache.arrow.vector.FieldVector fieldVector,
                                                                      ColumnType type)

Convert a field vector to a column vector

Parameters:

fieldVector - the field vector to convert

type - the type of the column vector

Returns:

the converted ndarray

convertToArrowVector

public static List<org.apache.arrow.vector.FieldVector> convertToArrowVector(org.nd4j.linalg.api.ndarray.INDArray from,
                                                                             List<String> name,
                                                                             ColumnType type,
                                                                             org.apache.arrow.memory.BufferAllocator bufferAllocator)

Convert an INDArray to a list of column vectors or a singleton list when either a row vector or a column vector

Parameters:

from - the input array

name - the name of the vector

type - the type of the vector

bufferAllocator - the allocator to use

Returns:

the list of field vectors

writeRecordBatchTo

public static void writeRecordBatchTo(List<List<Writable>> recordBatch,
                                      Schema inputSchema,
                                      OutputStream outputStream)

Write the records to the given output stream

Parameters:

recordBatch - the record batch to write

inputSchema - the input schema

outputStream - the output stream to write to

writeRecordBatchTo

public static void writeRecordBatchTo(org.apache.arrow.memory.BufferAllocator bufferAllocator,
                                      List<List<Writable>> recordBatch,
                                      Schema inputSchema,
                                      OutputStream outputStream)

Write the records to the given output stream

Parameters:

recordBatch - the record batch to write

inputSchema - the input schema

outputStream - the output stream to write to

toArrowWritablesTimeSeries

public static List<List<List<Writable>>> toArrowWritablesTimeSeries(List<org.apache.arrow.vector.FieldVector> fieldVectors,
                                                                    Schema schema,
                                                                    int timeSeriesLength)

Convert the input field vectors (the input data) and the given schema to a proper list of writables.

Parameters:

fieldVectors - the field vectors to use

schema - the schema to use

timeSeriesLength - the length of the time series

Returns:

the equivalent datavec batch given the input data

toArrowWritables

public static ArrowWritableRecordBatch toArrowWritables(List<org.apache.arrow.vector.FieldVector> fieldVectors,
                                                        Schema schema)

Convert the input field vectors (the input data) and the given schema to a proper list of writables.

Parameters:

fieldVectors - the field vectors to use

schema - the schema to use

Returns:

the equivalent datavec batch given the input data

toArrowWritablesSingle

public static List<Writable> toArrowWritablesSingle(List<org.apache.arrow.vector.FieldVector> fieldVectors,
                                                    Schema schema)

Return a singular record based on the converted writables result.

Parameters:

fieldVectors - the field vectors to use

schema - the schema to use for input

Returns:

readFromFile

public static org.nd4j.linalg.primitives.Pair<Schema,ArrowWritableRecordBatch> readFromFile(FileInputStream input)
                                                                                     throws IOException

Read a datavec schema and record set from the given arrow file.

Parameters:

input - the input to read

Returns:

the associated datavec schema and record

Throws:

IOException

readFromFile

public static org.nd4j.linalg.primitives.Pair<Schema,ArrowWritableRecordBatch> readFromFile(File input)
                                                                                     throws IOException

Read a datavec schema and record set from the given arrow file.

Parameters:

input - the input to read

Returns:

the associated datavec schema and record

Throws:

IOException

readFromBytes

public static org.nd4j.linalg.primitives.Pair<Schema,ArrowWritableRecordBatch> readFromBytes(byte[] input)
                                                                                      throws IOException

Read a datavec schema and record set from the given bytes (usually expected to be an arrow format file)

Parameters:

input - the input to read

Returns:

the associated datavec schema and record

Throws:

IOException

toArrowSchema

public static org.apache.arrow.vector.types.pojo.Schema toArrowSchema(Schema schema)

Convert a data vec Schema to an arrow Schema

Parameters:

schema - the input schema

Returns:

the schema for arrow

toDatavecSchema

public static Schema toDatavecSchema(org.apache.arrow.vector.types.pojo.Schema schema)

Convert an Schema to a datavec Schema

Parameters:

schema - the input arrow schema

Returns:

the equivalent datavec schema

field

public static org.apache.arrow.vector.types.pojo.Field field(String name,
                                                             org.apache.arrow.vector.types.pojo.ArrowType arrowType)

Shortcut method for returning a field given an arrow type and name with no sub fields

Parameters:

name - the name of the field

arrowType - the arrow type of the field

Returns:

the resulting field

getFieldForColumn

public static org.apache.arrow.vector.types.pojo.Field getFieldForColumn(String name,
                                                                         ColumnType columnType)

Create a field given the input ColumnType and name

Parameters:

name - the name of the field

columnType - the column type to add

Returns:

doubleField

public static org.apache.arrow.vector.types.pojo.Field doubleField(String name)

Shortcut method for creating a double field with 64 bit floating point

Parameters:

name - the name of the field

Returns:

the created field

floatField

public static org.apache.arrow.vector.types.pojo.Field floatField(String name)

Shortcut method for creating a double field with 32 bit floating point

Parameters:

name - the name of the field

Returns:

the created field

intField

public static org.apache.arrow.vector.types.pojo.Field intField(String name)

Shortcut method for creating a double field with 32 bit integer field

Parameters:

name - the name of the field

Returns:

the created field

longField

public static org.apache.arrow.vector.types.pojo.Field longField(String name)

Shortcut method for creating a long field with 64 bit long field

Parameters:

name - the name of the field

Returns:

the created field

stringField

public static org.apache.arrow.vector.types.pojo.Field stringField(String name)

Parameters:

name -

Returns:

booleanField

public static org.apache.arrow.vector.types.pojo.Field booleanField(String name)

Shortcut

Parameters:

name -

Returns:

providerForVectors

public static org.apache.arrow.vector.dictionary.DictionaryProvider providerForVectors(List<org.apache.arrow.vector.FieldVector> vectors,
                                                                                       List<org.apache.arrow.vector.types.pojo.Field> fields)

Provide a value look up dictionary based on the given set of input FieldVector s for reading and writing to arrow streams

Parameters:

vectors - the vectors to use as a lookup

Returns:

the associated DictionaryProvider for the given input FieldVector list

toArrowColumns

public static List<org.apache.arrow.vector.FieldVector> toArrowColumns(org.apache.arrow.memory.BufferAllocator bufferAllocator,
                                                                       Schema schema,
                                                                       List<List<Writable>> dataVecRecord)

Given a buffer allocator and datavec schema, convert the passed in batch of records to a set of arrow columns

Parameters:

bufferAllocator - the buffer allocator to use

schema - the schema to convert

dataVecRecord - the data vec record batch to convert

Returns:

the converted list of FieldVector

toArrowColumnsTimeSeries

public static List<org.apache.arrow.vector.FieldVector> toArrowColumnsTimeSeries(org.apache.arrow.memory.BufferAllocator bufferAllocator,
                                                                                 Schema schema,
                                                                                 List<List<List<Writable>>> dataVecRecord)

Convert a set of input strings to arrow columns for a time series.

Parameters:

bufferAllocator - the buffer allocator to use

schema - the schema to use

dataVecRecord - the collection of input strings to process

Returns:

the created vectors

toArrowColumnsTimeSeriesHelper

public static <T> List<org.apache.arrow.vector.FieldVector> toArrowColumnsTimeSeriesHelper(org.apache.arrow.memory.BufferAllocator bufferAllocator,
                                                                                           Schema schema,
                                                                                           List<List<List<T>>> dataVecRecord)

Convert a set of input strings to arrow columns for a time series.

Parameters:

bufferAllocator - the buffer allocator to use

schema - the schema to use

dataVecRecord - the collection of input strings to process

Returns:

the created vectors

toArrowColumnsStringSingle

public static List<org.apache.arrow.vector.FieldVector> toArrowColumnsStringSingle(org.apache.arrow.memory.BufferAllocator bufferAllocator,
                                                                                   Schema schema,
                                                                                   List<String> dataVecRecord)

Convert a set of input strings to arrow columns

Parameters:

bufferAllocator - the buffer allocator to use

schema - the schema to use

dataVecRecord - the collection of input strings to process

Returns:

the created vectors

toArrowColumnsStringTimeSeries

public static List<org.apache.arrow.vector.FieldVector> toArrowColumnsStringTimeSeries(org.apache.arrow.memory.BufferAllocator bufferAllocator,
                                                                                       Schema schema,
                                                                                       List<List<List<String>>> dataVecRecord)

Convert a set of input strings to arrow columns for a time series.

Parameters:

bufferAllocator - the buffer allocator to use

schema - the schema to use

dataVecRecord - the collection of input strings to process

Returns:

the created vectors

toArrowColumnsString

public static List<org.apache.arrow.vector.FieldVector> toArrowColumnsString(org.apache.arrow.memory.BufferAllocator bufferAllocator,
                                                                             Schema schema,
                                                                             List<List<String>> dataVecRecord)

Convert a set of input strings to arrow columns

Parameters:

bufferAllocator - the buffer allocator to use

schema - the schema to use

dataVecRecord - the collection of input strings to process

Returns:

the created vectors

setValue

public static void setValue(ColumnType columnType,
                            org.apache.arrow.vector.FieldVector fieldVector,
                            Object value,
                            int row)

Set the value of the specified column vector at the specified row based on the given value. The value will be converted relative to the specified column type. Note that the passed in value may only be a Writable or a String

Parameters:

columnType - the column type of the value

fieldVector - the field vector to set

value - the value to set (Writable or String types)

row - the row of the item

vectorFor

public static org.apache.arrow.vector.TimeStampMilliVector vectorFor(org.apache.arrow.memory.BufferAllocator allocator,
                                                                     String name,
                                                                     Date[] data)

Parameters:

allocator -

name -

data -

Returns:

timeVectorOf

public static org.apache.arrow.vector.TimeStampMilliVector timeVectorOf(org.apache.arrow.memory.BufferAllocator allocator,
                                                                        String name,
                                                                        int length)

Parameters:

allocator -

name -

length - the length of the vector

Returns:

vectorFor

public static org.apache.arrow.vector.VarBinaryVector vectorFor(org.apache.arrow.memory.BufferAllocator bufferAllocator,
                                                                String name,
                                                                org.nd4j.linalg.api.ndarray.INDArray[] data)

Returns a vector representing a tensor view of each ndarray. Each ndarray will be a "row" represented as a tensor object with in the return VarBinaryVector

Parameters:

bufferAllocator - the buffer allocator to use

name - the name of the column

data - the input arrays

Returns:

vectorFor

public static org.apache.arrow.vector.VarCharVector vectorFor(org.apache.arrow.memory.BufferAllocator allocator,
                                                              String name,
                                                              String[] data)

Parameters:

allocator -

name -

data -

Returns:

ndarrayVectorOf

public static org.apache.arrow.vector.VarBinaryVector ndarrayVectorOf(org.apache.arrow.memory.BufferAllocator allocator,
                                                                      String name,
                                                                      int length)

Create an ndarray vector that stores structs of INDArray based on the org.apache.arrow.flatbuf.Tensor format

Parameters:

allocator - the allocator to use

name - the name of the vector

length - the number of vectors to store

Returns:

stringVectorOf

public static org.apache.arrow.vector.VarCharVector stringVectorOf(org.apache.arrow.memory.BufferAllocator allocator,
                                                                   String name,
                                                                   int length)

Parameters:

allocator -

name -

length - the length of the vector

Returns:

vectorFor

public static org.apache.arrow.vector.Float4Vector vectorFor(org.apache.arrow.memory.BufferAllocator allocator,
                                                             String name,
                                                             float[] data)

Parameters:

allocator -

name -

data -

Returns:

floatVectorOf

public static org.apache.arrow.vector.Float4Vector floatVectorOf(org.apache.arrow.memory.BufferAllocator allocator,
                                                                 String name,
                                                                 int length)

Parameters:

allocator -

name -

length - the length of the vector

Returns:

vectorFor

public static org.apache.arrow.vector.Float8Vector vectorFor(org.apache.arrow.memory.BufferAllocator allocator,
                                                             String name,
                                                             double[] data)

Parameters:

allocator -

name -

data -

Returns:

doubleVectorOf

public static org.apache.arrow.vector.Float8Vector doubleVectorOf(org.apache.arrow.memory.BufferAllocator allocator,
                                                                  String name,
                                                                  int length)

Parameters:

allocator -

name -

length - the length of the vector

Returns:

vectorFor

public static org.apache.arrow.vector.BitVector vectorFor(org.apache.arrow.memory.BufferAllocator allocator,
                                                          String name,
                                                          boolean[] data)

Parameters:

allocator -

name -

data -

Returns:

booleanVectorOf

public static org.apache.arrow.vector.BitVector booleanVectorOf(org.apache.arrow.memory.BufferAllocator allocator,
                                                                String name,
                                                                int length)

Parameters:

allocator -

name -

Returns:

vectorFor

public static org.apache.arrow.vector.IntVector vectorFor(org.apache.arrow.memory.BufferAllocator allocator,
                                                          String name,
                                                          int[] data)

Parameters:

allocator -

name -

data -

Returns:

intVectorOf

public static org.apache.arrow.vector.IntVector intVectorOf(org.apache.arrow.memory.BufferAllocator allocator,
                                                            String name,
                                                            int length)

Parameters:

allocator -

name -

Returns:

vectorFor

public static org.apache.arrow.vector.BigIntVector vectorFor(org.apache.arrow.memory.BufferAllocator allocator,
                                                             String name,
                                                             long[] data)

Parameters:

allocator -

name -

data -

Returns:

longVectorOf

public static org.apache.arrow.vector.BigIntVector longVectorOf(org.apache.arrow.memory.BufferAllocator allocator,
                                                                String name,
                                                                int length)

Parameters:

allocator -

name -

length - the number of rows in the column vector

Returns:

fromEntry

public static Writable fromEntry(int item,
                                 org.apache.arrow.vector.FieldVector from,
                                 ColumnType columnType)

Based on an input ColumnType get an entry from a FieldVector

Parameters:

item - the row of the item to get from the column vector

from - the column vector from

columnType - the column type

Returns:

the resulting writable