org.graylog2.shared.journal

Class KafkaJournal

java.lang.Object

com.google.common.util.concurrent.AbstractIdleService

org.graylog2.shared.journal.KafkaJournal

All Implemented Interfaces:

com.google.common.util.concurrent.Service, Journal

@Singleton
public class KafkaJournal
extends com.google.common.util.concurrent.AbstractIdleService
implements Journal

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	KafkaJournal.DirtyLogFlusher
class	KafkaJournal.LogRetentionCleaner Java implementation of the Kafka log retention cleaner.
class	KafkaJournal.OffsetFileFlusher
class	KafkaJournal.RecoveryCheckpointFlusher

Nested classes/interfaces inherited from interface org.graylog2.shared.journal.Journal

Journal.Entry, Journal.JournalReadEntry

Nested classes/interfaces inherited from interface com.google.common.util.concurrent.Service

com.google.common.util.concurrent.Service.Listener, com.google.common.util.concurrent.Service.State

Field Summary

Fields

Modifier and Type	Field and Description
static long	DEFAULT_COMMITTED_OFFSET
static int	NOTIFY_ON_UTILIZATION_PERCENTAGE

Constructor Summary

Constructors

Constructor and Description

KafkaJournal(File journalDirectory, ScheduledExecutorService scheduler, com.github.joschi.jadconfig.util.Size segmentSize, org.joda.time.Duration segmentAge, com.github.joschi.jadconfig.util.Size retentionSize, org.joda.time.Duration retentionAge, long flushInterval, org.joda.time.Duration flushAge, int throttleThresholdPercentage, com.codahale.metrics.MetricRegistry metricRegistry, ServerStatus serverStatus)

Method Summary

Modifier and Type	Method and Description
int	cleanupLogs()
Journal.Entry	createEntry(byte[] idBytes, byte[] messageBytes) Creates an opaque object which can be passed to write(java.util.List) for a bulk journal write.
protected void	flushDirtyLogs() A Java transliteration of what the scala implementation does, which unfortunately is declared as private
long	getCommittedOffset()
long	getCommittedReadOffset() Returns the highest journal offset that has been writting to persistent storage by Graylog.
long	getLogEndOffset() returns the offset for the next value to be inserted in the entire journal.
long	getLogStartOffset() Returns the first valid offset in the entire journal.
long	getNextReadOffset()
int	getPurgedSegmentsInLastRetention()
Iterable<kafka.log.LogSegment>	getSegments()
ThrottleState	getThrottleState() For informational purposes this method provides access to the current state of the journal.
void	markJournalOffsetCommitted(long offset) Upon fully processing, and persistently storing, a batch of messages, the system should mark the message with the highest offset as committed.
int	numberOfSegments() Returns the number of segments this journal consists of.
List<Journal.JournalReadEntry>	read(long requestedMaximumCount)
List<Journal.JournalReadEntry>	read(long readOffset, long requestedMaximumCount)
void	setThrottleState(ThrottleState state)
protected void	shutDown()
long	size() Returns the journal size in bytes, exluding index files.
protected void	startUp()
void	truncateTo(long offset) Discards all data in the journal prior to the given offset.
long	write(byte[] idBytes, byte[] messageBytes) Writes a single message to the journal and returns the new write position
long	write(List<Journal.Entry> entries) Writes the list of entries to the journal.

Methods inherited from class com.google.common.util.concurrent.AbstractIdleService

addListener, awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, executor, failureCause, isRunning, serviceName, startAsync, state, stopAsync, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_COMMITTED_OFFSET

public static final long DEFAULT_COMMITTED_OFFSET

See Also:

Constant Field Values

NOTIFY_ON_UTILIZATION_PERCENTAGE

public static final int NOTIFY_ON_UTILIZATION_PERCENTAGE

See Also:

Constant Field Values

Constructor Detail

KafkaJournal

@Inject
public KafkaJournal(@Named(value="message_journal_dir")
                            File journalDirectory,
                            @Named(value="scheduler")
                            ScheduledExecutorService scheduler,
                            @Named(value="message_journal_segment_size")
                            com.github.joschi.jadconfig.util.Size segmentSize,
                            @Named(value="message_journal_segment_age")
                            org.joda.time.Duration segmentAge,
                            @Named(value="message_journal_max_size")
                            com.github.joschi.jadconfig.util.Size retentionSize,
                            @Named(value="message_journal_max_age")
                            org.joda.time.Duration retentionAge,
                            @Named(value="message_journal_flush_interval")
                            long flushInterval,
                            @Named(value="message_journal_flush_age")
                            org.joda.time.Duration flushAge,
                            @Named(value="lb_throttle_threshold_percentage")
                            int throttleThresholdPercentage,
                            com.codahale.metrics.MetricRegistry metricRegistry,
                            ServerStatus serverStatus)

Method Detail

getPurgedSegmentsInLastRetention

public int getPurgedSegmentsInLastRetention()

createEntry

public Journal.Entry createEntry(byte[] idBytes,
                                 byte[] messageBytes)

Creates an opaque object which can be passed to write(java.util.List) for a bulk journal write.

Specified by:

createEntry in interface Journal

Parameters:

idBytes - a byte array which represents the key for the entry

messageBytes - the journal entry's payload, i.e. the message itself

Returns:

a journal entry to be passed to write(java.util.List)

write

public long write(List<Journal.Entry> entries)

Writes the list of entries to the journal.

Specified by:

write in interface Journal

Parameters:

entries - journal entries to be written

Returns:

the last position written to in the journal

write

public long write(byte[] idBytes,
                  byte[] messageBytes)

Writes a single message to the journal and returns the new write position

Specified by:

write in interface Journal

Parameters:

idBytes - byte array congaing the message id

messageBytes - encoded message payload

Returns:

the last position written to in the journal

read

public List<Journal.JournalReadEntry> read(long requestedMaximumCount)

Specified by:

read in interface Journal

read

public List<Journal.JournalReadEntry> read(long readOffset,
                                           long requestedMaximumCount)

markJournalOffsetCommitted

public void markJournalOffsetCommitted(long offset)

Upon fully processing, and persistently storing, a batch of messages, the system should mark the message with the highest offset as committed. A background job will write the last position to disk periodically.

Specified by:

markJournalOffsetCommitted in interface Journal

Parameters:

offset - the offset of the latest committed message

flushDirtyLogs

protected void flushDirtyLogs()

A Java transliteration of what the scala implementation does, which unfortunately is declared as private

getCommittedOffset

public long getCommittedOffset()

getNextReadOffset

public long getNextReadOffset()

startUp

protected void startUp()
                throws Exception

Specified by:

startUp in class com.google.common.util.concurrent.AbstractIdleService

Throws:

Exception

shutDown

protected void shutDown()
                 throws Exception

Specified by:

shutDown in class com.google.common.util.concurrent.AbstractIdleService

Throws:

Exception

cleanupLogs

public int cleanupLogs()

getSegments

public Iterable<kafka.log.LogSegment> getSegments()

size

public long size()

Returns the journal size in bytes, exluding index files.

Returns:

journal size in bytes

numberOfSegments

public int numberOfSegments()

Returns the number of segments this journal consists of.

Returns:

number of segments

getCommittedReadOffset

public long getCommittedReadOffset()

Returns the highest journal offset that has been writting to persistent storage by Graylog.

Every message at an offset prior to this one can be considered as processed and does not need to be held in the journal any longer. By default Graylog will try to aggressively flush the journal to consume a smaller amount of disk space.

Returns:

the offset of the last message which has been successfully processed.

truncateTo

public void truncateTo(long offset)

Discards all data in the journal prior to the given offset.

Parameters:

offset - offset to truncate to, so that no offset in the journal is larger than this.

getLogStartOffset

public long getLogStartOffset()

Returns the first valid offset in the entire journal.

Returns:

first offset

getLogEndOffset

public long getLogEndOffset()

returns the offset for the next value to be inserted in the entire journal.

Returns:

the next offset value (last valid offset is this number - 1)

getThrottleState

public ThrottleState getThrottleState()

For informational purposes this method provides access to the current state of the journal.

Returns:

the journal state for throttling purposes

setThrottleState

public void setThrottleState(ThrottleState state)