ch.qos.logback.ext.loggly

Class LogglyBatchAppender<E>

java.lang.Object

ch.qos.logback.core.spi.ContextAwareBase

ch.qos.logback.core.UnsynchronizedAppenderBase<E>

ch.qos.logback.ext.loggly.AbstractLogglyAppender<E>

ch.qos.logback.ext.loggly.LogglyBatchAppender<E>

All Implemented Interfaces:

ch.qos.logback.core.Appender<E>, ch.qos.logback.core.spi.ContextAware, ch.qos.logback.core.spi.FilterAttachable<E>, ch.qos.logback.core.spi.LifeCycle, LogglyBatchAppenderMBean

public class LogglyBatchAppender<E>
extends AbstractLogglyAppender<E>
implements LogglyBatchAppenderMBean

Logback batch appender for Loggly HTTP API.

Note:Loggly's Syslog API is much more scalable than the HTTP API which should mostly be used in low-volume or non-production systems. The HTTP API can be very convenient to workaround firewalls.

If the LogglyBatchAppender saturates and discards log messages, the following warning message is appended to both Loggly and System.err:
"$date - OutputStream is full, discard previous logs"

Configuration settings

Property Name	Type	Description
inputKey	String	Loggly input key. "inputKey" or endpointUrl is required. Sample "12345678-90ab-cdef-1234-567890abcdef"
endpointUrl	String	Loggly HTTP API endpoint URL. "inputKey" or endpointUrl is required. Sample: "https://logs.loggly.com/inputs/12345678-90ab-cdef-1234-567890abcdef"
pattern	String	Pattern used for Loggly log messages. Default value is: %d{"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",UTC} %-5level [%thread] %logger: %m%n.
proxyHost	String	hostname of a proxy server. If blank, no proxy is used (See URL.openConnection(java.net.Proxy).
proxyPort	int	port of a proxy server. Must be a valid int but is ignored if proxyHost is blank or null.
jmxMonitoring	boolean	Enable registration of a monitoring MBean named "ch.qos.logback:type=LogglyBatchAppender,name=LogglyBatchAppender@#hashcode#". Default: true.
maxNumberOfBuckets	int	Max number of buckets of in the byte buffer. Default value: 8.
maxBucketSizeInKilobytes	int	Max size of each bucket. Default value: 1024 Kilobytes (1MB).
flushIntervalInSeconds	int	Interval of the buffer flush to Loggly API. Default value: 3.

Default configuration consumes up to 8 buffers of 1024 Kilobytes (1MB) each, which seems very reasonable even for small JVMs. If logs are discarded, try first to shorten the flushIntervalInSeconds parameter to "2s" or event "1s".

Configuration Sample

 <configuration scan="true" scanPeriod="30 seconds" debug="true">
   <if condition='isDefined("logback.loggly.inputKey")'>
     <then>
       <appender name="loggly" class="ch.qos.logback.ext.loggly.LogglyBatchAppender">
         <inputKey>${logback.loggly.inputKey}</inputKey>
         <pattern>%d{yyyy/MM/dd HH:mm:ss,SSS} [${HOSTNAME}] [%thread] %-5level %logger{36} - %m %throwable{5}%n</pattern>
         <proxyHost>${logback.loggly.proxy.host:-}</proxyHost>
         <proxyPort>${logback.loggly.proxy.port:-8080}</proxyPort>
         <debug>${logback.loggly.debug:-false}</debug>
       </appender>
       <root level="WARN">
         <appender-ref ref="loggly"/>
       </root>
     </then>
   </if>
 </configuration>
 

Implementation decisions

• Why buffer the generated log messages as bytes instead of using the ch.qos.logback.core.read.CyclicBufferAppender and buffering the ILoggingEvent ? Because it is much easier to control the size in memory
• Why buffer in a byte array instead of directly writing in a BufferedOutputStream on the HttpURLConnection ? Because the Loggly API may not like such kind of streaming approach.

Author:

Cyrille Le Clerc

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	LogglyBatchAppender.LogglyExporter

Field Summary

Fields

Modifier and Type	Field and Description
protected AtomicLong	sendDurationInNanos
protected AtomicInteger	sendExceptionCount
protected AtomicInteger	sendSuccessCount
protected AtomicLong	sentBytes

Fields inherited from class ch.qos.logback.ext.loggly.AbstractLogglyAppender

DEFAULT_ENDPOINT_PREFIX, DEFAULT_LAYOUT_PATTERN, endpointUrl, inputKey, layout, layoutCreatedImplicitly, proxy, UTF_8

Fields inherited from class ch.qos.logback.core.UnsynchronizedAppenderBase

name, started

Fields inherited from class ch.qos.logback.core.spi.ContextAwareBase

context

Constructor Summary

Constructors

Constructor and Description
LogglyBatchAppender()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	append(E eventObject)
long	getCurrentLogEntriesBufferSizeInBytes() Size in bytes of the log entries that have not yet been sent to Loggly.
int	getDiscardedBucketsCount() Number of discarded buckets
int	getFlushIntervalInSeconds()
protected HttpURLConnection	getHttpConnection(URL url) Creates a configured HTTP connection to a URL (does not open the connection)
long	getSendDurationInNanos() Duration spent sending logs to Loggly.
int	getSendExceptionCount() Number of failing invocations to Loggly's send logs API.
int	getSendSuccessCount() Number of successful invocations to Loggly's send logs API.
long	getSentBytes() Number of bytes sent to Loggly.
boolean	isDebug()
void	processLogEntries() Send log entries to Loggly
protected void	processLogEntries(InputStream in) Send log entries to Loggly
void	setDebug(boolean debug) Enable debugging
void	setFlushIntervalInSeconds(int flushIntervalInSeconds)
void	setJmxMonitoring(boolean jmxMonitoring)
void	setMaxBucketSizeInKilobytes(int maxBucketSizeInKilobytes)
void	setMaxNumberOfBuckets(int maxNumberOfBuckets)
void	setMbeanServer(MBeanServer mbeanServer)
void	start()
void	stop()

Methods inherited from class ch.qos.logback.ext.loggly.AbstractLogglyAppender

buildEndpointUrl, createLayout, ensureLayout, getEndpointUrl, getHttpReadTimeoutInMillis, getInputKey, getLayout, getPattern, getProxyHost, getProxyPort, readResponseBody, setEndpointUrl, setHttpReadTimeoutInMillis, setInputKey, setLayout, setPattern, setProxyHost, setProxyPort, setProxyPort, toBytes

Methods inherited from class ch.qos.logback.core.UnsynchronizedAppenderBase

addFilter, clearAllFilters, doAppend, getCopyOfAttachedFiltersList, getFilterChainDecision, getName, isStarted, setName, toString

Methods inherited from class ch.qos.logback.core.spi.ContextAwareBase

addError, addError, addInfo, addInfo, addStatus, addWarn, addWarn, getContext, getDeclaredOrigin, getStatusManager, setContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface ch.qos.logback.core.spi.ContextAware

addError, addError, addInfo, addInfo, addStatus, addWarn, addWarn, getContext, setContext

Field Detail

sendDurationInNanos

protected final AtomicLong sendDurationInNanos

sentBytes

protected final AtomicLong sentBytes

sendSuccessCount

protected final AtomicInteger sendSuccessCount

sendExceptionCount

protected final AtomicInteger sendExceptionCount

Constructor Detail

LogglyBatchAppender

public LogglyBatchAppender()

Method Detail

append

protected void append(E eventObject)

Specified by:

append in class ch.qos.logback.core.UnsynchronizedAppenderBase<E>

start

public void start()

Specified by:

start in interface ch.qos.logback.core.spi.LifeCycle

Overrides:

start in class AbstractLogglyAppender<E>

stop

public void stop()

Specified by:

stop in interface ch.qos.logback.core.spi.LifeCycle

Overrides:

stop in class AbstractLogglyAppender<E>

processLogEntries

public void processLogEntries()

Send log entries to Loggly

Specified by:

processLogEntries in interface LogglyBatchAppenderMBean

getHttpConnection

protected HttpURLConnection getHttpConnection(URL url)
                                       throws IOException

Creates a configured HTTP connection to a URL (does not open the connection)

Parameters:

url - target URL

Returns:

the newly created HTTP connection

Throws:

IOException

processLogEntries

protected void processLogEntries(InputStream in)
                          throws IOException

Send log entries to Loggly

Throws:

IOException

getFlushIntervalInSeconds

public int getFlushIntervalInSeconds()

setFlushIntervalInSeconds

public void setFlushIntervalInSeconds(int flushIntervalInSeconds)

getSentBytes

public long getSentBytes()

Description copied from interface: LogglyBatchAppenderMBean

Number of bytes sent to Loggly.

Specified by:

getSentBytes in interface LogglyBatchAppenderMBean

getSendDurationInNanos

public long getSendDurationInNanos()

Description copied from interface: LogglyBatchAppenderMBean

Duration spent sending logs to Loggly.

Specified by:

getSendDurationInNanos in interface LogglyBatchAppenderMBean

getSendSuccessCount

public int getSendSuccessCount()

Description copied from interface: LogglyBatchAppenderMBean

Number of successful invocations to Loggly's send logs API.

Specified by:

getSendSuccessCount in interface LogglyBatchAppenderMBean

getSendExceptionCount

public int getSendExceptionCount()

Description copied from interface: LogglyBatchAppenderMBean

Number of failing invocations to Loggly's send logs API.

Specified by:

getSendExceptionCount in interface LogglyBatchAppenderMBean

getDiscardedBucketsCount

public int getDiscardedBucketsCount()

Description copied from interface: LogglyBatchAppenderMBean

Number of discarded buckets

Specified by:

getDiscardedBucketsCount in interface LogglyBatchAppenderMBean

getCurrentLogEntriesBufferSizeInBytes

public long getCurrentLogEntriesBufferSizeInBytes()

Description copied from interface: LogglyBatchAppenderMBean

Size in bytes of the log entries that have not yet been sent to Loggly.

Specified by:

getCurrentLogEntriesBufferSizeInBytes in interface LogglyBatchAppenderMBean

setDebug

public void setDebug(boolean debug)

Description copied from interface: LogglyBatchAppenderMBean

Enable debugging

Specified by:

setDebug in interface LogglyBatchAppenderMBean

isDebug

public boolean isDebug()

Specified by:

isDebug in interface LogglyBatchAppenderMBean

setJmxMonitoring

public void setJmxMonitoring(boolean jmxMonitoring)

setMbeanServer

public void setMbeanServer(MBeanServer mbeanServer)

setMaxNumberOfBuckets

public void setMaxNumberOfBuckets(int maxNumberOfBuckets)

setMaxBucketSizeInKilobytes

public void setMaxBucketSizeInKilobytes(int maxBucketSizeInKilobytes)