io.delta.kernel.client

Interface JsonHandler

@Evolving
public interface JsonHandler

Provides JSON handling functionality to Delta Kernel. Delta Kernel can use this client to parse JSON strings into ColumnarBatch or read content from JSON files. Connectors can leverage this interface to provide their best implementation of the JSON parsing capability to Delta Kernel.

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
StructType	deserializeStructType(String structTypeJson) Deserialize the Delta schema from structTypeJson according to the Delta Protocol schema serialization rules .
ColumnarBatch	parseJson(ColumnVector jsonStringVector, StructType outputSchema, java.util.Optional<ColumnVector> selectionVector) Parse the given json strings and return the fields requested by outputSchema as columns in a ColumnarBatch.
CloseableIterator<ColumnarBatch>	readJsonFiles(CloseableIterator<FileStatus> fileIter, StructType physicalSchema, java.util.Optional<Predicate> predicate) Read and parse the JSON format file at given locations and return the data as a ColumnarBatch with the columns requested by physicalSchema.

Method Detail

parseJson

ColumnarBatch parseJson(ColumnVector jsonStringVector,
                        StructType outputSchema,
                        java.util.Optional<ColumnVector> selectionVector)

Parse the given json strings and return the fields requested by outputSchema as columns in a ColumnarBatch.

There are a couple special cases that should be handled for specific data types:

• FloatType and DoubleType: handle non-numeric numbers encoded as strings
  • NaN: "NaN"
  • Positive infinity: "+INF", "Infinity", "+Infinity"
  • Negative infinity: "-INF", "-Infinity""
• DateType: handle dates encoded as strings in the format "yyyy-MM-dd"
• TimestampType: handle timestamps encoded as strings in the format "yyyy-MM-dd'T'HH:mm:ss.SSSXXX"

Parameters:

jsonStringVector - String ColumnVector of valid JSON strings.

outputSchema - Schema of the data to return from the parsed JSON. If any requested fields are missing in the JSON string, a null is returned for that particular field in the returned Row. The type for each given field is expected to match the type in the JSON.

selectionVector - Optional selection vector indicating which rows to parse the JSON. If present, only the selected rows should be parsed. Unselected rows should be all null in the returned batch.

Returns:

a ColumnarBatch of schema outputSchema with one row for each entry in jsonStringVector

deserializeStructType

StructType deserializeStructType(String structTypeJson)

Deserialize the Delta schema from structTypeJson according to the Delta Protocol schema serialization rules .

Parameters:

structTypeJson - the JSON formatted schema string to parse

Returns:

the parsed StructType

readJsonFiles

CloseableIterator<ColumnarBatch> readJsonFiles(CloseableIterator<FileStatus> fileIter,
                                               StructType physicalSchema,
                                               java.util.Optional<Predicate> predicate)
                                        throws java.io.IOException

Read and parse the JSON format file at given locations and return the data as a ColumnarBatch with the columns requested by physicalSchema.

Parameters:

fileIter - Iterator of files to read data from.

physicalSchema - Select list of columns to read from the JSON file.

predicate - Optional predicate which the JSON reader can optionally use to prune rows that don't satisfy the predicate. Because pruning is optional and may be incomplete, caller is still responsible apply the predicate on the data returned by this method.

Returns:

an iterator of ColumnarBatchs containing the data in columnar format. It is the responsibility of the caller to close the iterator. The data returned is in the same as the order of files given in scanFileIter

Throws:

java.io.IOException - if an I/O error occurs during the read.