程序包 org.apache.pulsar.io.kinesis

类 KinesisSink

java.lang.Object
org.apache.pulsar.io.aws.AbstractAwsConnector
org.apache.pulsar.io.kinesis.KinesisSink
所有已实现的接口:
AutoCloseable, org.apache.pulsar.io.core.Sink<byte[]>
public class KinesisSink extends org.apache.pulsar.io.aws.AbstractAwsConnector implements org.apache.pulsar.io.core.Sink<byte[]>
A Kinesis sink which can be configured by KinesisSinkConfig. 
 KinesisSinkConfig accepts
 1. awsEndpoint: kinesis end-point url can be found at : https://docs.aws.amazon.com/general/latest/gr/rande.html
 2. awsRegion: appropriate aws region eg: us-west-1, us-west-2
 3. awsKinesisStreamName: kinesis stream name
 4. awsCredentialPluginName: Fully-Qualified class name of implementation of AwsCredentialProviderPlugin.
    - It is a factory class which creates an AWSCredentialsProvider that will be used by KinesisProducer
    - If it is empty then KinesisSink creates default AWSCredentialsProvider
      which accepts json-map of credentials in awsCredentialPluginParam
      eg: awsCredentialPluginParam = {"accessKey":"my-access-key","secretKey":"my-secret-key"}
 5. awsCredentialPluginParam: json-parameters to initialize AwsCredentialProviderPlugin
 6. messageFormat: enum:["ONLY_RAW_PAYLOAD","FULL_MESSAGE_IN_JSON"]
   a. ONLY_RAW_PAYLOAD:     publishes raw payload to stream
   b. FULL_MESSAGE_IN_JSON: publish full message (encryptionCtx + properties + payload) in json format
   json-schema:
   {"type":"object","properties":{"encryptionCtx":{"type":"object","properties":{"metadata":{"type":"object","additionalProperties":{"type":"string"}},"uncompressedMessageSize":{"type":"integer"},"keysMetadataMap":{"type":"object","additionalProperties":{"type":"object","additionalProperties":{"type":"string"}}},"keysMapBase64":{"type":"object","additionalProperties":{"type":"string"}},"encParamBase64":{"type":"string"},"compressionType":{"type":"string","enum":["NONE","LZ4","ZLIB"]},"batchSize":{"type":"integer"},"algorithm":{"type":"string"}}},"payloadBase64":{"type":"string"},"properties":{"type":"object","additionalProperties":{"type":"string"}}}}
   Example:
   {"payloadBase64":"cGF5bG9hZA==","properties":{"prop1":"value"},"encryptionCtx":{"keysMapBase64":{"key1":"dGVzdDE=","key2":"dGVzdDI="},"keysMetadataMap":{"key1":{"ckms":"cmks-1","version":"v1"},"key2":{"ckms":"cmks-2","version":"v2"}},"metadata":{"ckms":"cmks-1","version":"v1"},"encParamBase64":"cGFyYW0=","algorithm":"algo","compressionType":"LZ4","uncompressedMessageSize":10,"batchSize":10}}

  • 字段详细资料

    • METRICS_TOTAL_INCOMING

      public static final String METRICS_TOTAL_INCOMING
      另请参阅:

    • METRICS_TOTAL_INCOMING_BYTES

      public static final String METRICS_TOTAL_INCOMING_BYTES
      另请参阅:

    • METRICS_TOTAL_SUCCESS

      public static final String METRICS_TOTAL_SUCCESS
      另请参阅:

    • METRICS_TOTAL_FAILURE

      public static final String METRICS_TOTAL_FAILURE
      另请参阅:

  • 构造器详细资料

    • KinesisSink

      public KinesisSink()

  • 方法详细资料

    • write

      public void write(org.apache.pulsar.functions.api.Record<byte[]> record) throws Exception
      指定者:
      write 在接口中 org.apache.pulsar.io.core.Sink<byte[]>
      抛出:
      Exception

    • close

      public void close() throws IOException
      指定者:
      close 在接口中 AutoCloseable
      抛出:
      IOException

    • open

      public void open(Map<String,Object> config, org.apache.pulsar.io.core.SinkContext sinkContext) throws Exception
      指定者:
      open 在接口中 org.apache.pulsar.io.core.Sink<byte[]>
      抛出:
      Exception

    • createKinesisMessage

      public static ByteBuffer createKinesisMessage(KinesisSinkConfig.MessageFormat msgFormat, org.apache.pulsar.functions.api.Record<byte[]> record)