org.apache.hadoop.hive.ql.log.syslog

Class SyslogInputFormat

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

org.apache.hadoop.mapred.TextInputFormat

org.apache.hadoop.hive.ql.log.syslog.SyslogInputFormat

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>, org.apache.hadoop.mapred.JobConfigurable

public class SyslogInputFormat
extends org.apache.hadoop.mapred.TextInputFormat

Syslog input format handles files written to sys.logs hive table. This input format can prune files based on predicates in ts (timestamp) column. Following are the assumptions/supported options for the file pruning to work. When configuring log shippers (like fluentd), the following directory layout has to be followed WAREHOUSE_DIR/sys.db/logs/dt=DATE/ns=NAMESPACE/app=APPLICATION/YYYY-mm-DD-HH-MM_i[.log][.gz] WAREHOUSE_DIR - is hive warehouse location for external tables DATE - date of the log files NAMESPACE - namespace for the applications APPLICATION - name of the application (hiveserver2, metastore etc.) YYYY - year mm - month DD - day of the month HH - hour of the day (24 hr format) MM - minute i - index (log rotation) [.log] - optional log extension [.gz] - optional gz compression extension When executing queries like select count(*) from sys.logs where ts between current_timestamp() - INTERVAL '1' DAY and current_timestamp() syslog input format can prunes the file listing based on the query filter condition. File Pruning Assumptions/Support: - All timestamps are in UTC (timestamp in file name, log lines and in query predicates all assume UTC) - File names are in YYYY-mm-DD-HH-MM_i[.log][.gz] format. - Seconds and millis in predicates gets rounded off to nearest minute (essentially ignoring them). - Time slice used by log aggregation library should match with hive.syslog.input.format.file.time.slice config. If a filename is 2019-04-02-21-00_0.log.gz and timeslice is 300s then the file 2019-04-02-21-00_0.log.gz is expected to have log lines from timestamp 2019:04:02 21:00:00 to 2019:04:02 21:05:00 timestamp. - Logs table should have 'ts' as timestamp column. - Only simple BETWEEN filter predicate is supported for 'ts' column. There cannot be >1 predicates on 'ts' column.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileInputFormat

org.apache.hadoop.mapred.FileInputFormat.Counter

Field Summary

Fields inherited from class org.apache.hadoop.mapred.FileInputFormat

INPUT_DIR_NONRECURSIVE_IGNORE_SUBDIRS, INPUT_DIR_RECURSIVE, NUM_INPUT_FILES

Constructor Summary

Constructors

Constructor and Description
SyslogInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>	getRecordReader(org.apache.hadoop.mapred.InputSplit genericSplit, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter)
protected org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.mapred.JobConf job)
List<org.apache.hadoop.fs.FileStatus>	pruneFiles(SearchArgument sarg, long timeSliceSeconds, List<org.apache.hadoop.fs.FileStatus> inputFiles)

Methods inherited from class org.apache.hadoop.mapred.TextInputFormat

configure, isSplitable

Methods inherited from class org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, getSplits, makeSplit, makeSplit, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SyslogInputFormat

public SyslogInputFormat()

Method Detail

listStatus

protected org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.mapred.JobConf job)
                                                throws IOException

Overrides:

listStatus in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

Throws:

IOException

pruneFiles

public List<org.apache.hadoop.fs.FileStatus> pruneFiles(SearchArgument sarg,
                                                        long timeSliceSeconds,
                                                        List<org.apache.hadoop.fs.FileStatus> inputFiles)

getRecordReader

public org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> getRecordReader(org.apache.hadoop.mapred.InputSplit genericSplit,
                                                                                                                          org.apache.hadoop.mapred.JobConf job,
                                                                                                                          org.apache.hadoop.mapred.Reporter reporter)
                                                                                                                   throws IOException

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

Overrides:

getRecordReader in class org.apache.hadoop.mapred.TextInputFormat

Throws:

IOException