Package org.apache.pulsar.io.kinesis

Class KinesisSink

java.lang.Object
org.apache.pulsar.io.aws.AbstractAwsConnector
org.apache.pulsar.io.kinesis.KinesisSink
All Implemented Interfaces:
AutoCloseable, Sink<org.apache.pulsar.client.api.schema.GenericObject>
public class KinesisSink extends AbstractAwsConnector implements Sink<org.apache.pulsar.client.api.schema.GenericObject>
A Kinesis sink which can be configured by KinesisSinkConfig. 
 KinesisSinkConfig accepts
 1. awsEndpoint: kinesis end-point url can be found at : https://docs.aws.amazon.com/general/latest/gr/rande.html
 2. awsRegion: appropriate aws region eg: us-west-1, us-west-2
 3. awsKinesisStreamName: kinesis stream name
 4. awsCredentialPluginName: Fully-Qualified class name of implementation of AwsCredentialProviderPlugin.
    - It is a factory class which creates an AWSCredentialsProvider that will be used by KinesisProducer
    - If it is empty then KinesisSink creates default AWSCredentialsProvider
      which accepts json-map of credentials in awsCredentialPluginParam
      eg: awsCredentialPluginParam = {"accessKey":"my-access-key","secretKey":"my-secret-key"}
 5. awsCredentialPluginParam: json-parameters to initialize AwsCredentialProviderPlugin
 6. messageFormat: enum:["ONLY_RAW_PAYLOAD","FULL_MESSAGE_IN_JSON","FULL_MESSAGE_IN_FB"]
   a. ONLY_RAW_PAYLOAD:     publishes raw payload to stream
   b. FULL_MESSAGE_IN_JSON: publish full message (encryptionCtx + properties + payload) in json format
   json-schema:
   {"type":"object","properties":{"encryptionCtx":{"type":"object","properties":{"metadata":{"type":"object","additionalProperties":{"type":"string"}},"uncompressedMessageSize":{"type":"integer"},"keysMetadataMap":{"type":"object","additionalProperties":{"type":"object","additionalProperties":{"type":"string"}}},"keysMapBase64":{"type":"object","additionalProperties":{"type":"string"}},"encParamBase64":{"type":"string"},"compressionType":{"type":"string","enum":["NONE","LZ4","ZLIB"]},"batchSize":{"type":"integer"},"algorithm":{"type":"string"}}},"payloadBase64":{"type":"string"},"properties":{"type":"object","additionalProperties":{"type":"string"}}}}
   Example:
   {"payloadBase64":"cGF5bG9hZA==","properties":{"prop1":"value"},"encryptionCtx":{"keysMapBase64":{"key1":"dGVzdDE=","key2":"dGVzdDI="},"keysMetadataMap":{"key1":{"ckms":"cmks-1","version":"v1"},"key2":{"ckms":"cmks-2","version":"v2"}},"metadata":{"ckms":"cmks-1","version":"v1"},"encParamBase64":"cGFyYW0=","algorithm":"algo","compressionType":"LZ4","uncompressedMessageSize":10,"batchSize":10}}
   c. FULL_MESSAGE_IN_FB: publish full message (encryptionCtx + properties + payload) in flatbuffer format
   d. FULL_MESSAGE_IN_JSON_EXPAND_VALUE: publish full message (topic + key + value + properties + event time) in JSON format using the schema to expand the value in JSON.

  • Field Details

  • Constructor Details

    • KinesisSink

      public KinesisSink()

  • Method Details

    • write

      public void write(org.apache.pulsar.functions.api.Record<org.apache.pulsar.client.api.schema.GenericObject> record) throws Exception
      Specified by:
      write in interface Sink<org.apache.pulsar.client.api.schema.GenericObject>
      Throws:
      Exception

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable

    • open

      public void open(Map<String,Object> config, SinkContext sinkContext)
      Specified by:
      open in interface Sink<org.apache.pulsar.client.api.schema.GenericObject>

    • createKinesisMessage

      public ByteBuffer createKinesisMessage(KinesisSinkConfig.MessageFormat msgFormat, org.apache.pulsar.functions.api.Record<org.apache.pulsar.client.api.schema.GenericObject> record) throws com.fasterxml.jackson.core.JsonProcessingException
      Throws:
      com.fasterxml.jackson.core.JsonProcessingException