Package org.apache.pinot.spi.stream

Class StreamConfigProperties

  • public class StreamConfigProperties
extends Object
    Defines the keys for the stream config properties map

    • Field Detail

      • SEGMENT_FLUSH_THRESHOLD_TIME

        public static final String SEGMENT_FLUSH_THRESHOLD_TIME
        Time threshold that will keep the realtime segment open for before we complete the segment
        See Also:
        Constant Field Values

      • DEPRECATED_SEGMENT_FLUSH_THRESHOLD_ROWS

        public static final String DEPRECATED_SEGMENT_FLUSH_THRESHOLD_ROWS
        Deprecated.
        because the property key is confusing (says size but is actually rows). Use SEGMENT_FLUSH_THRESHOLD_ROWS Row count flush threshold for realtime segments. This behaves in a similar way for HLC and LLC. For HLC, since there is only one consumer per server, this size is used as the size of the consumption buffer and determines after how many rows we flush to disk. For example, if this threshold is set to two million rows, then a high level consumer would have a buffer size of two million. For LLC, this size is divided across all the segments assigned to a given server and is set on a per segment basis. Assuming a low level consumer server is assigned four stream partitions to consume from and a flush size of two million, then each consuming segment would have a flush size of five hundred thousand rows, for a total of two million rows in memory. Keep in mind that this NOT a hard threshold, as other tables can also be assigned to this server, and that in certain conditions (eg. if the number of servers, replicas of partitions changes) where partition to server assignment changes, it's possible to end up with more (or less) than this number of rows in memory. If this value is set to 0, then the consumers adjust the number of rows consumed by a partition such that the size of the completed segment is the desired size (see REALTIME_DESIRED_SEGMENT_SIZE), unless REALTIME_SEGMENT_FLUSH_TIME is reached first)
        See Also:
        Constant Field Values

      • DEPRECATED_SEGMENT_FLUSH_DESIRED_SIZE

        public static final String DEPRECATED_SEGMENT_FLUSH_DESIRED_SIZE
        Deprecated.
        because the property key is confusing (desired size is not indicative of segment size). Use SEGMENT_FLUSH_THRESHOLD_SEGMENT_SIZE The desired size of a completed realtime segment. This config is used only if REALTIME_SEGMENT_FLUSH_SIZE is set to 0. Default value of REALTIME_SEGMENT_FLUSH_SIZE is "200M". Values are parsed using DataSize class. The value for this configuration should be chosen based on the amount of memory available on consuming machines, the number of completed segments that are expected to be resident on the machine and the amount of memory used by consuming machines. In other words: numPartitionsInMachine * (consumingPartitionMemory + numPartitionsRetained * REALTIME_DESIRED_SEGMENT_SIZE) must be less than or equal to the total memory available to store pinot data. Note that consumingPartitionMemory will vary depending on the rows that are consumed. Not included here is any heap memory used (currently inverted index uses heap memory for consuming partitions).
        See Also:
        Constant Field Values

      • SEGMENT_FLUSH_THRESHOLD_SEGMENT_SIZE

        public static final String SEGMENT_FLUSH_THRESHOLD_SEGMENT_SIZE
        See Also:
        Constant Field Values

      • SEGMENT_FLUSH_AUTOTUNE_INITIAL_ROWS

        public static final String SEGMENT_FLUSH_AUTOTUNE_INITIAL_ROWS
        The initial num rows to use for segment size auto tuning. By default 100_000 is used.
        See Also:
        Constant Field Values

    • Method Detail

      • constructStreamProperty

        public static String constructStreamProperty​(String streamType,
                                             String property)
        Helper method to create a stream specific property

      • getPropertySuffix

        public static String getPropertySuffix​(String incoming,
                                       String propertyPrefix)