ai.databand.log

Class DbndLogger

java.lang.Object

ai.databand.log.DbndLogger

public class DbndLogger
extends java.lang.Object

Report metrics and dataframes to Databand. This class is safe to use in any circumstances and won't break client code.

Constructor Summary

Constructors

Constructor and Description
DbndLogger()

Method Summary

Modifier and Type	Method and Description
static void	logDataframe(java.lang.String key, java.lang.Object value, boolean withHistograms) Report Spark dataframe.
static void	logDataframe(java.lang.String key, java.lang.Object value, HistogramRequest histogramRequest) Report Spark dataframe.
static void	logDatasetOperation(java.lang.String path, DatasetOperationType type, org.apache.spark.sql.Dataset<?> data) Report success dataset operation.
static void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, org.apache.spark.sql.Dataset<?> data) Report dataset operation.
static void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, org.apache.spark.sql.Dataset<?> data, boolean withPreview) Report dataset operation.
static void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, org.apache.spark.sql.Dataset<?> data, boolean withPreview, boolean withSchema) Report dataset operation.
static void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, org.apache.spark.sql.Dataset<?> data, boolean withPreview, boolean withSchema, java.lang.Boolean withPartition) Report dataset operation.
static void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, org.apache.spark.sql.Dataset<?> data, java.lang.Throwable error) Report dataset operation and error operation if occurred.
static void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, org.apache.spark.sql.Dataset<?> data, java.lang.Throwable error, boolean withPartition) Report dataset operation and error operation if occurred.
static void	logMetric(java.lang.String key, java.lang.Object value) Report single metric.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DbndLogger

public DbndLogger()

Method Detail

logMetric

public static void logMetric(java.lang.String key,
                             java.lang.Object value)

Report single metric.

Parameters:

key - metric key

value - metric value. Value will be converted using following rules: integers and doubles will be converted to integers and doubles strings will be converted to strings spark datasets will generate preview toString() will be called for the rest of the types

logDataframe

public static void logDataframe(java.lang.String key,
                                java.lang.Object value,
                                boolean withHistograms)

Report Spark dataframe. Descriptive statistics and histograms will be generated for each column.

Parameters:

key -

value -

withHistograms -

logDataframe

public static void logDataframe(java.lang.String key,
                                java.lang.Object value,
                                HistogramRequest histogramRequest)

Report Spark dataframe. Descriptive statistics and histograms will be generated for each column. Reporting can be customized using HistogramRequest object.

Parameters:

key -

value -

histogramRequest -

logDatasetOperation

public static void logDatasetOperation(java.lang.String path,
                                       DatasetOperationType type,
                                       org.apache.spark.sql.Dataset<?> data)

Report success dataset operation. Schema and preview will automatically be calculated.

Parameters:

path - data path (S3, filesystem, GCS etc)

type - operation type — read/write

data - spark dataset

logDatasetOperation

public static void logDatasetOperation(java.lang.String path,
                                       DatasetOperationType type,
                                       DatasetOperationStatus status,
                                       org.apache.spark.sql.Dataset<?> data)

Report dataset operation. Schema and preview will automatically be calculated.

Parameters:

path - data path (S3, filesystem, GCS etc)

type - operation type — read/write

status - operation status — success/failure

data - spark dataset

logDatasetOperation

public static void logDatasetOperation(java.lang.String path,
                                       DatasetOperationType type,
                                       DatasetOperationStatus status,
                                       org.apache.spark.sql.Dataset<?> data,
                                       java.lang.Throwable error)

Report dataset operation and error operation if occurred. Schema and preview will automatically be calculated.

Parameters:

path - data path (S3, filesystem, GCS etc)

type - operation type — read/write

status - operation status — success/failure

data - spark dataset

error - error when operation was failed

logDatasetOperation

public static void logDatasetOperation(java.lang.String path,
                                       DatasetOperationType type,
                                       DatasetOperationStatus status,
                                       org.apache.spark.sql.Dataset<?> data,
                                       java.lang.Throwable error,
                                       boolean withPartition)

Report dataset operation and error operation if occurred. Schema and preview will automatically be calculated.

Parameters:

path - data path (S3, filesystem, GCS etc)

type - operation type — read/write

status - operation status — success/failure

data - spark dataset

error - error when operation was failed

withPartition - pass true if you want to remove partition path from dataset path or false if you want to keep them

logDatasetOperation

public static void logDatasetOperation(java.lang.String path,
                                       DatasetOperationType type,
                                       DatasetOperationStatus status,
                                       org.apache.spark.sql.Dataset<?> data,
                                       boolean withPreview)

Report dataset operation. Schema will be automatically calculated.

Parameters:

path - data path (S3, filesystem, GCS etc)

type - operation type — read/write

status - operation status — success/failure

data - spark dataset

withPreview - pass true if you want to generate preview (may impact performance)

logDatasetOperation

public static void logDatasetOperation(java.lang.String path,
                                       DatasetOperationType type,
                                       DatasetOperationStatus status,
                                       org.apache.spark.sql.Dataset<?> data,
                                       boolean withPreview,
                                       boolean withSchema)

Report dataset operation.

Parameters:

path - data path (S3, filesystem, GCS etc)

type - operation type — read/write

status - operation status — success/failure

data - spark dataset

withPreview - pass true if you want to generate preview (may impact performance)

withSchema - pass true if you want to calculate dataset schema

logDatasetOperation

public static void logDatasetOperation(java.lang.String path,
                                       DatasetOperationType type,
                                       DatasetOperationStatus status,
                                       org.apache.spark.sql.Dataset<?> data,
                                       boolean withPreview,
                                       boolean withSchema,
                                       java.lang.Boolean withPartition)

Report dataset operation.

Parameters:

path - data path (S3, filesystem, GCS etc)

type - operation type — read/write

status - operation status — success/failure

data - spark dataset

withPreview - pass true if you want to generate preview (may impact performance)

withSchema - pass true if you want to calculate dataset schema

withPartition - pass true if you want to remove partition path from dataset path or false if you want to keep them