org.apache.beam.examples.complete.datatokenization.utils

Class ErrorConverters.WriteErrorsToTextIO<T,V>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<FailsafeElement<T,V>>,org.apache.beam.sdk.values.PDone>

org.apache.beam.examples.complete.datatokenization.utils.ErrorConverters.WriteErrorsToTextIO<T,V>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

ErrorConverters

public abstract static class ErrorConverters.WriteErrorsToTextIO<T,V>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<FailsafeElement<T,V>>,org.apache.beam.sdk.values.PDone>

ErrorConverters.WriteErrorsToTextIO is a PTransform that writes strings error messages to file system using TextIO and custom line format SerializableFunction to convert errors in necessary format.
Example of usage in pipeline:

 pCollection.apply("Write to TextIO",
   WriteErrorsToTextIO.<String,String>newBuilder()
     .setErrorWritePath("errors.txt")
     .setTranslateFunction((FailsafeElement<String,String> failsafeElement) -> {
       ArrayList<String> outputRow  = new ArrayList<>();
       final String message = failsafeElement.getOriginalPayload();
       String timestamp = Instant.now().toString();
       outputRow.add(timestamp);
       outputRow.add(failsafeElement.getErrorMessage());
       outputRow.add(failsafeElement.getStacktrace());
       // Only set the payload if it's populated on the message.
       if (failsafeElement.getOriginalPayload() != null) {
         outputRow.add(message);
       }

       return String.join(",",outputRow);
     })
 

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ErrorConverters.WriteErrorsToTextIO.Builder<T,V> Builder for ErrorConverters.WriteErrorsToTextIO.

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
WriteErrorsToTextIO()

Method Summary

Modifier and Type	Method and Description
abstract java.lang.String	errorWritePath()
org.apache.beam.sdk.values.PDone	expand(org.apache.beam.sdk.values.PCollection<FailsafeElement<T,V>> pCollection)
static <T,V> ErrorConverters.WriteErrorsToTextIO.Builder<T,V>	newBuilder()
abstract org.apache.beam.sdk.transforms.SerializableFunction<FailsafeElement<T,V>,java.lang.String>	translateFunction()
abstract @Nullable org.joda.time.Duration	windowDuration()

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

WriteErrorsToTextIO

public WriteErrorsToTextIO()

Method Detail

newBuilder

public static <T,V> ErrorConverters.WriteErrorsToTextIO.Builder<T,V> newBuilder()

errorWritePath

public abstract java.lang.String errorWritePath()

translateFunction

public abstract org.apache.beam.sdk.transforms.SerializableFunction<FailsafeElement<T,V>,java.lang.String> translateFunction()

windowDuration

public abstract @Nullable org.joda.time.Duration windowDuration()

expand

public org.apache.beam.sdk.values.PDone expand(org.apache.beam.sdk.values.PCollection<FailsafeElement<T,V>> pCollection)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<FailsafeElement<T,V>>,org.apache.beam.sdk.values.PDone>