Package org.apache.beam.sdk.extensions.avro.io

Class AvroIO.ParseFiles<T>

    • Field Summary

      • Fields inherited from class org.apache.beam.sdk.transforms.PTransform

        annotations, displayData, name, resourceHints

    • Constructor Summary

      Constructors 
      Constructor Description
      ParseFiles()  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.beam.sdk.values.PCollection<T> expand​(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.io.FileIO.ReadableFile> input)  
      void populateDisplayData​(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)  
      AvroIO.ParseFiles<T> withCoder​(org.apache.beam.sdk.coders.Coder<T> coder)
      Specifies the coder for the result of the parseFn.
      AvroIO.ParseFiles<T> withDesiredBundleSizeBytes​(long desiredBundleSizeBytes)
      Set a value for the bundle size for parallel reads.
      AvroIO.ParseFiles<T> withFileExceptionHandler​(org.apache.beam.sdk.io.ReadAllViaFileBasedSource.ReadFileRangesFnExceptionHandler exceptionHandler)
      Specifies if exceptions should be logged only for streaming pipelines.
      AvroIO.ParseFiles<T> withUsesReshuffle​(boolean usesReshuffle)
      Specifies if a Reshuffle should run before file reads occur.

      • Methods inherited from class org.apache.beam.sdk.transforms.PTransform

        addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate, validate

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Constructor Detail

      • ParseFiles

        public ParseFiles()

    • Method Detail

      • withCoder

        public AvroIO.ParseFiles<T> withCoder​(org.apache.beam.sdk.coders.Coder<T> coder)
        Specifies the coder for the result of the parseFn.

      • withUsesReshuffle

        public AvroIO.ParseFiles<T> withUsesReshuffle​(boolean usesReshuffle)
        Specifies if a Reshuffle should run before file reads occur.

      • withFileExceptionHandler

        public AvroIO.ParseFiles<T> withFileExceptionHandler​(org.apache.beam.sdk.io.ReadAllViaFileBasedSource.ReadFileRangesFnExceptionHandler exceptionHandler)
        Specifies if exceptions should be logged only for streaming pipelines.

      • withDesiredBundleSizeBytes

        public AvroIO.ParseFiles<T> withDesiredBundleSizeBytes​(long desiredBundleSizeBytes)
        Set a value for the bundle size for parallel reads. Default is 64 MB. You may want to use a lower value (e.g. 1 MB) for streaming applications.

      • expand

        public org.apache.beam.sdk.values.PCollection<T> expand​(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.io.FileIO.ReadableFile> input)
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.io.FileIO.ReadableFile>,​org.apache.beam.sdk.values.PCollection<T>>

      • populateDisplayData

        public void populateDisplayData​(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
        Specified by:
        populateDisplayData in interface org.apache.beam.sdk.transforms.display.HasDisplayData
        Overrides:
        populateDisplayData in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.io.FileIO.ReadableFile>,​org.apache.beam.sdk.values.PCollection<T>>