org.apache.hadoop.hive.ql.log.syslog

Class SyslogParser

java.lang.Object

org.apache.hadoop.hive.ql.log.syslog.SyslogParser

All Implemented Interfaces:

Closeable, AutoCloseable

public class SyslogParser
extends Object
implements Closeable

A Syslog protocol parser. It should be capable of parsing RFC 3164 (BSD syslog) streams as well as RFC 5424 (defined in 2009.). Adapted from https://github.com/spotify/flume-syslog-source2 and modified it for several assumptions about the way hive logs using syslog format (specifically RFC5424). This implementation also parses structured data, returns all parsed fields as map and also un-escapes messages. This parser also gracefully handles some corner cases where 'msg' can be empty or line can start with '<' but not a valid RFC5424 format etc. Assumption: 1) This parser assumes the linebreaks '\n' in stack traces for example are replaced by '\r' to make single line message. The reader will do replacement of '\r' with '\n' at the time of read. 2) This parser assumes structured data values are html escaped. So it will html unescape when parsing structured data. (hive writes log lines directly to stderr that look like rfc5424 layout starting with '<' so the expectation from log4j2 is to escape those lines using html escaping). 3) Read event returns List<Object> conforming to sys.logs table schema in hive. The schema for sys.logs table is expected to be (facility STRING, severity STRING, version STRING, ts TIMESTAMP, hostname STRING, app_name STRING, proc_id STRING, msg_id STRING, structured_data map<STRING,STRING>, msg BINARY, unmatched BINARY) 4) Timestamps are in UTC This parser is tested with Log4j2's RFC5424 layout generated using the following properties appenders = console appender.console.layout.type = Rfc5424Layout appender.console.layout.appName = ${env:APP_NAME} appender.console.layout.facility = USER appender.console.layout.includeMDC = true appender.console.layout.mdcId = mdc appender.console.layout.messageId = ${env:MSG_ID} appender.console.layout.newLine = true appender.console.layout.newLineEscape = \\r appender.console.layout.exceptionPattern = %ex{full} appender.console.layout.loggerfields.type = LoggerFields appender.console.layout.loggerfields.pairs1.type = KeyValuePair appender.console.layout.loggerfields.pairs1.key = level appender.console.layout.loggerfields.pairs1.value = %p appender.console.layout.loggerfields.pairs2.type = KeyValuePair appender.console.layout.loggerfields.pairs2.key = thread appender.console.layout.loggerfields.pairs2.value = %enc{%t} appender.console.layout.loggerfields.pairs3.type = KeyValuePair appender.console.layout.loggerfields.pairs3.key = class appender.console.layout.loggerfields.pairs3.value = %c{2}

Constructor Summary

Constructors

Constructor and Description
SyslogParser()
SyslogParser(InputStream in) Construct a new Syslog protocol parser.
SyslogParser(InputStream in, boolean parseTag, Charset encoding) Construct a new Syslog protocol parser.

Method Summary

Modifier and Type	Method and Description
void	close() Free the resources used by this parser.
List<Object>	readEvent() Read the next Syslog message from the stream.
void	setInputStream(InputStream in)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SyslogParser

public SyslogParser()

SyslogParser

public SyslogParser(InputStream in)

Construct a new Syslog protocol parser. Tags are parsed, and the encoding is assumed to be UTF-8.

SyslogParser

public SyslogParser(InputStream in,
                    boolean parseTag,
                    Charset encoding)

Construct a new Syslog protocol parser.

Parameters:

in - the stream to read data from. The InputStream#read() function is heavily used, so make sure it is buffered.

parseTag - true to parse the "tag[pid]:", false to leave it as part of the message body.

encoding - the encoding to use for various string conversions, most notably the hostname.

Method Detail

close

public void close()
           throws IOException

Free the resources used by this parser. Note that the parser cannot be reused. Closes the underlying input stream.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

readEvent

public List<Object> readEvent()
                       throws IOException

Read the next Syslog message from the stream.

Returns:

a parsed map of object, or null on EOF.

Throws:

IOException - if the underlying stream fails, or unexpected bytes are seen.

setInputStream

public void setInputStream(InputStream in)