org.graylog2.shared.journal

Class LocalKafkaJournal

java.lang.Object

com.google.common.util.concurrent.AbstractIdleService

org.graylog2.shared.journal.LocalKafkaJournal

All Implemented Interfaces:

com.google.common.util.concurrent.Service, Journal

@Singleton
public class LocalKafkaJournal
extends com.google.common.util.concurrent.AbstractIdleService
implements Journal

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	LocalKafkaJournal.DirtyLogFlusher
class	LocalKafkaJournal.LogRetentionCleaner Java implementation of the Kafka log retention cleaner.
class	LocalKafkaJournal.OffsetFileFlusher
class	LocalKafkaJournal.RecoveryCheckpointFlusher

Nested classes/interfaces inherited from interface org.graylog2.shared.journal.Journal

Journal.Entry, Journal.JournalReadEntry

Nested classes/interfaces inherited from interface com.google.common.util.concurrent.Service

com.google.common.util.concurrent.Service.Listener, com.google.common.util.concurrent.Service.State

Field Summary

Fields

Modifier and Type	Field and Description
static long	DEFAULT_COMMITTED_OFFSET
static String	GAUGE_UNCOMMITTED_MESSAGES
static String	METER_READ_MESSAGES
static String	METER_WRITTEN_MESSAGES
static int	NOTIFY_ON_UTILIZATION_PERCENTAGE
static int	THRESHOLD_THROTTLING_DISABLED

Constructor Summary

Constructors

Constructor and Description
LocalKafkaJournal(Path journalDirectory, ScheduledExecutorService scheduler, com.github.joschi.jadconfig.util.Size segmentSize, org.joda.time.Duration segmentAge, com.github.joschi.jadconfig.util.Size retentionSize, org.joda.time.Duration retentionAge, long flushInterval, org.joda.time.Duration flushAge, int throttleThresholdPercentage, com.codahale.metrics.MetricRegistry metricRegistry, ServerStatus serverStatus)
LocalKafkaJournal(Path journalDirectory, ScheduledExecutorService scheduler, com.github.joschi.jadconfig.util.Size segmentSize, org.joda.time.Duration segmentAge, com.github.joschi.jadconfig.util.Size retentionSize, org.joda.time.Duration retentionAge, long flushInterval, org.joda.time.Duration flushAge, int throttleThresholdPercentage, com.codahale.metrics.MetricRegistry metricRegistry, ServerStatus serverStatus, String metricPrefix)

Method Summary

Modifier and Type	Method and Description
int	cleanupLogs()
Journal.Entry	createEntry(byte[] idBytes, byte[] messageBytes) Creates an opaque object which can be passed to write(java.util.List) for a bulk journal write.
protected void	flushDirtyLogs() A Java transliteration of what the scala implementation does, which unfortunately is declared as private
long	getCommittedOffset()
long	getCommittedReadOffset() Returns the highest journal offset that has been writting to persistent storage by Graylog.
long	getLogEndOffset() returns the offset for the next value to be inserted in the entire journal.
long	getLogStartOffset() Returns the first valid offset in the entire journal.
long	getNextReadOffset()
int	getPurgedSegmentsInLastRetention()
Iterable<org.graylog.shaded.kafka09.log.LogSegment>	getSegments()
ThrottleState	getThrottleState() For informational purposes this method provides access to the current state of the journal.
void	markJournalOffsetCommitted(long offset) Upon fully processing, and persistently storing, a batch of messages, the system should mark the message with the highest offset as committed.
int	numberOfSegments() Returns the number of segments this journal consists of.
List<Journal.JournalReadEntry>	read(long requestedMaximumCount)
List<Journal.JournalReadEntry>	read(long readOffset, long requestedMaximumCount)
void	setThrottleState(ThrottleState state)
protected void	shutDown()
long	size() Returns the journal size in bytes, exluding index files.
protected void	startUp()
void	truncateTo(long offset) Discards all data in the journal prior to the given offset.
long	write(byte[] idBytes, byte[] messageBytes) Writes a single message to the journal and returns the new write position
long	write(List<Journal.Entry> entries) Writes the list of entries to the journal.

Methods inherited from class com.google.common.util.concurrent.AbstractIdleService

addListener, awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, executor, failureCause, isRunning, serviceName, startAsync, state, stopAsync, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.google.common.util.concurrent.Service

awaitRunning, awaitTerminated

Field Detail

DEFAULT_COMMITTED_OFFSET

public static final long DEFAULT_COMMITTED_OFFSET

See Also:

Constant Field Values

NOTIFY_ON_UTILIZATION_PERCENTAGE

public static final int NOTIFY_ON_UTILIZATION_PERCENTAGE

See Also:

Constant Field Values

THRESHOLD_THROTTLING_DISABLED

public static final int THRESHOLD_THROTTLING_DISABLED

See Also:

Constant Field Values

METER_WRITTEN_MESSAGES

public static final String METER_WRITTEN_MESSAGES

See Also:

Constant Field Values

METER_READ_MESSAGES

public static final String METER_READ_MESSAGES

See Also:

Constant Field Values

GAUGE_UNCOMMITTED_MESSAGES

public static final String GAUGE_UNCOMMITTED_MESSAGES

See Also:

Constant Field Values

Constructor Detail

LocalKafkaJournal

@Inject
public LocalKafkaJournal(@Named(value="message_journal_dir")
                                 Path journalDirectory,
                                 @Named(value="scheduler")
                                 ScheduledExecutorService scheduler,
                                 @Named(value="message_journal_segment_size")
                                 com.github.joschi.jadconfig.util.Size segmentSize,
                                 @Named(value="message_journal_segment_age")
                                 org.joda.time.Duration segmentAge,
                                 @Named(value="message_journal_max_size")
                                 com.github.joschi.jadconfig.util.Size retentionSize,
                                 @Named(value="message_journal_max_age")
                                 org.joda.time.Duration retentionAge,
                                 @Named(value="message_journal_flush_interval")
                                 long flushInterval,
                                 @Named(value="message_journal_flush_age")
                                 org.joda.time.Duration flushAge,
                                 @Named(value="lb_throttle_threshold_percentage")
                                 int throttleThresholdPercentage,
                                 com.codahale.metrics.MetricRegistry metricRegistry,
                                 ServerStatus serverStatus)

LocalKafkaJournal

public LocalKafkaJournal(Path journalDirectory,
                         ScheduledExecutorService scheduler,
                         com.github.joschi.jadconfig.util.Size segmentSize,
                         org.joda.time.Duration segmentAge,
                         com.github.joschi.jadconfig.util.Size retentionSize,
                         org.joda.time.Duration retentionAge,
                         long flushInterval,
                         org.joda.time.Duration flushAge,
                         int throttleThresholdPercentage,
                         com.codahale.metrics.MetricRegistry metricRegistry,
                         ServerStatus serverStatus,
                         String metricPrefix)

Parameters:

throttleThresholdPercentage - The journal utilization percent at which throttling will be triggered. Expressed as an integer between 1 and 100. The value -1 disables throttling.

Method Detail

getPurgedSegmentsInLastRetention

public int getPurgedSegmentsInLastRetention()

createEntry

public Journal.Entry createEntry(byte[] idBytes,
                                 byte[] messageBytes)

Creates an opaque object which can be passed to write(java.util.List) for a bulk journal write.

Specified by:

createEntry in interface Journal

Parameters:

idBytes - a byte array which represents the key for the entry

messageBytes - the journal entry's payload, i.e. the message itself

Returns:

a journal entry to be passed to write(java.util.List)

write

public long write(List<Journal.Entry> entries)

Writes the list of entries to the journal.

Specified by:

write in interface Journal

Parameters:

entries - journal entries to be written

Returns:

the last position written to in the journal

write

public long write(byte[] idBytes,
                  byte[] messageBytes)

Writes a single message to the journal and returns the new write position

Specified by:

write in interface Journal

Parameters:

idBytes - byte array congaing the message id

messageBytes - encoded message payload

Returns:

the last position written to in the journal

read

public List<Journal.JournalReadEntry> read(long requestedMaximumCount)

Specified by:

read in interface Journal

read

public List<Journal.JournalReadEntry> read(long readOffset,
                                           long requestedMaximumCount)

markJournalOffsetCommitted

public void markJournalOffsetCommitted(long offset)

Upon fully processing, and persistently storing, a batch of messages, the system should mark the message with the highest offset as committed. A background job will write the last position to disk periodically.

Specified by:

markJournalOffsetCommitted in interface Journal

Parameters:

offset - the offset of the latest committed message

flushDirtyLogs

protected void flushDirtyLogs()

A Java transliteration of what the scala implementation does, which unfortunately is declared as private

getCommittedOffset

public long getCommittedOffset()

getNextReadOffset

public long getNextReadOffset()

startUp

protected void startUp()
                throws Exception

Specified by:

startUp in class com.google.common.util.concurrent.AbstractIdleService

Throws:

Exception

shutDown

protected void shutDown()
                 throws Exception

Specified by:

shutDown in class com.google.common.util.concurrent.AbstractIdleService

Throws:

Exception

cleanupLogs

public int cleanupLogs()

getSegments

public Iterable<org.graylog.shaded.kafka09.log.LogSegment> getSegments()

size

public long size()

Returns the journal size in bytes, exluding index files.

Returns:

journal size in bytes

numberOfSegments

public int numberOfSegments()

Returns the number of segments this journal consists of.

Returns:

number of segments

getCommittedReadOffset

public long getCommittedReadOffset()

Returns the highest journal offset that has been writting to persistent storage by Graylog.

Every message at an offset prior to this one can be considered as processed and does not need to be held in the journal any longer. By default Graylog will try to aggressively flush the journal to consume a smaller amount of disk space.

Returns:

the offset of the last message which has been successfully processed.

truncateTo

public void truncateTo(long offset)

Discards all data in the journal prior to the given offset.

Parameters:

offset - offset to truncate to, so that no offset in the journal is larger than this.

getLogStartOffset

public long getLogStartOffset()

Returns the first valid offset in the entire journal.

Returns:

first offset

getLogEndOffset

public long getLogEndOffset()

returns the offset for the next value to be inserted in the entire journal.

Returns:

the next offset value (last valid offset is this number - 1)

getThrottleState

public ThrottleState getThrottleState()

For informational purposes this method provides access to the current state of the journal.

Returns:

the journal state for throttling purposes

setThrottleState

public void setThrottleState(ThrottleState state)