Package org.apache.beam.runners.spark.io

Class MicrobatchSource.Reader

java.lang.Object

org.apache.beam.sdk.io.Source.Reader<T>

org.apache.beam.runners.spark.io.MicrobatchSource.Reader

All Implemented Interfaces:

java.lang.AutoCloseable

Enclosing class:

MicrobatchSource<T,CheckpointMarkT extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark>

public class MicrobatchSource.Reader
extends org.apache.beam.sdk.io.Source.Reader<T>

Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.

This Reader reads until one of the following thresholds has been reached:

1. max records (per batch)
2. max read duration (per batch)

Method Summary

Modifier and Type	Method	Description
boolean	advance()
void	close()
CheckpointMarkT	getCheckpointMark()
T	getCurrent()
org.apache.beam.sdk.io.Source<T>	getCurrentSource()
org.joda.time.Instant	getCurrentTimestamp()
org.joda.time.Instant	getWatermark()
boolean	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

start

public boolean start()
              throws java.io.IOException

Specified by:

start in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

java.io.IOException

advance

public boolean advance()
                throws java.io.IOException

Specified by:

advance in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

java.io.IOException

getCurrent

public T getCurrent()
             throws java.util.NoSuchElementException

Specified by:

getCurrent in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

java.util.NoSuchElementException

getCurrentTimestamp

public org.joda.time.Instant getCurrentTimestamp()
                                          throws java.util.NoSuchElementException

Specified by:

getCurrentTimestamp in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

java.util.NoSuchElementException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

java.io.IOException

getCurrentSource

public org.apache.beam.sdk.io.Source<T> getCurrentSource()

Specified by:

getCurrentSource in class org.apache.beam.sdk.io.Source.Reader<T>

getCheckpointMark

public CheckpointMarkT getCheckpointMark()

getWatermark

public org.joda.time.Instant getWatermark()