Package com.mtfelisb.flink.connectors.elasticsearch.sink

Class ElasticsearchSink<T>

  • Type Parameters:
    T -
    All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<T>
    public class ElasticsearchSink<T>
extends org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
    ElasticsearchSink A RichSinkFunction with CheckpointedFunction implemented; The ElasticsearchSink uses BulkRequest internally to each incoming event and flushes if a specified threshold is reached, or if the checkpoint is enabled, when the snapshot is triggered.
    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

        org.apache.flink.streaming.api.functions.sink.SinkFunction.Context

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()
      close the close method when triggered by Flink will destroy the ElasticsearchClient connection
      void initializeState​(org.apache.flink.runtime.state.FunctionInitializationContext functionInitializationContext)
      initializeState
      void invoke​(T value, org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
      invoke buffers every data element in the stream until the threshold is reached
      void open​(org.apache.flink.configuration.Configuration parameters)
      open creates the connection with the Elasticsearch cluster relying on the provided factory
      void snapshotState​(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)
      snapshotState used as a hook; that means every checkpoint will trigger a bulk request to Elasticsearch

      • Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

        getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

      • Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

        finish, invoke, writeWatermark

    • Constructor Detail

      • ElasticsearchSink

        public ElasticsearchSink​(INetworkConfigFactory networkConfigFactory,
                         Emitter<T> emitter,
                         Long threshold,
                         IBulkRequestFactory bulkRequestFactory)
        ElasticsearchSink
        Parameters:
        networkConfigFactory - a factory to create the network conn with Elasticsearch
        emitter - user defined operations to be sent in bulk requests
        threshold - used defined limiting the bulk request size

    • Method Detail

      • open

        public void open​(org.apache.flink.configuration.Configuration parameters)
          throws Exception
        open creates the connection with the Elasticsearch cluster relying on the provided factory
        Specified by:
        open in interface org.apache.flink.api.common.functions.RichFunction
        Overrides:
        open in class org.apache.flink.api.common.functions.AbstractRichFunction
        Parameters:
        parameters - The configuration containing the parameters attached to the contract.
        Throws:
        Exception

      • invoke

        public void invoke​(T value,
                   org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
            throws Exception
        invoke buffers every data element in the stream until the threshold is reached
        Specified by:
        invoke in interface org.apache.flink.streaming.api.functions.sink.SinkFunction<T>
        Parameters:
        value - The input record.
        context - Additional context about the input record.
        Throws:
        Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)
                   throws Exception
        snapshotState used as a hook; that means every checkpoint will trigger a bulk request to Elasticsearch
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
        Parameters:
        functionSnapshotContext - the context for drawing a snapshot of the operator
        Throws:
        Exception

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.FunctionInitializationContext functionInitializationContext)
                     throws Exception
        initializeState
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
        Parameters:
        functionInitializationContext - the context for initializing the operator
        Throws:
        Exception

      • close

        public void close()
           throws Exception
        close the close method when triggered by Flink will destroy the ElasticsearchClient connection
        Specified by:
        close in interface org.apache.flink.api.common.functions.RichFunction
        Overrides:
        close in class org.apache.flink.api.common.functions.AbstractRichFunction
        Throws:
        Exception