org.apache.hadoop.hive.ql.log.syslog

Class SyslogStorageHandler

java.lang.Object

org.apache.hadoop.hive.ql.log.syslog.SyslogStorageHandler

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, HiveStorageHandler

public class SyslogStorageHandler
extends Object
implements HiveStorageHandler

SyslogStorageHandler handles log files written to logs table in sys db. sys.logs is external table partitioned by dt (date), ns (namespace) and app (application) where logs files are stored in YYYY-mm-DD-HH-MM_i[.log][.gz] format. The contents in the log files are expected to be in RFC5424 layout for machine readability and human readability (some editors can replace the unescape the linebreaks correctly to show stacktraces etc.). NOTE: This storage handler is not generic and has several assumptions that are specific for hive's own log processing. Refer SyslogInputFormat and SyslogParser for more assumptions.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

HiveStorageHandler.AcidSupportType

Field Summary

Fields inherited from interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

DEFAULT_ALLOWED_ALTER_OPS

Constructor Summary

Constructors

Constructor and Description
SyslogStorageHandler()

Method Summary

Modifier and Type	Method and Description
void	configureInputJobCredentials(TableDesc tableDesc, Map<String,String> secrets) This method is called to allow the StorageHandlers the chance to populate secret keys into the job's credentials.
void	configureInputJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc).
void	configureJobConf(TableDesc tableDesc, org.apache.hadoop.mapred.JobConf jobConf) Called just before submitting MapReduce job.
void	configureOutputJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc).
void	configureTableJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) Deprecated use configureInputJobProperties/configureOutputJobProperties methods instead.
HiveAuthorizationProvider	getAuthorizationProvider() Returns the implementation specific authorization provider
org.apache.hadoop.conf.Configuration	getConf()
Class<? extends org.apache.hadoop.mapred.InputFormat>	getInputFormatClass()
HiveMetaHook	getMetaHook()
Class<? extends org.apache.hadoop.mapred.OutputFormat>	getOutputFormatClass()
Class<? extends AbstractSerDe>	getSerDeClass()
void	setConf(org.apache.hadoop.conf.Configuration conf)
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

acidSelectColumns, acidSortColumns, acidVirtualColumns, addDynamicSplitPruningEdge, addResourcesForCreateTable, alterTableSnapshotRefOperation, alwaysUnpartitioned, appendFiles, areSnapshotsSupported, canComputeQueryUsingStats, canPerformMetadataDelete, canProvideBasicStatistics, canProvideColStatistics, canSetColStatistics, canUseTruncate, checkAndSetTableMetaRef, commitInMoveTask, createDPContext, directInsert, executeOperation, getBasicStatistics, getColStatistics, getColumnInfo, getCurrentSnapshotContext, getFileFormatPropertyKey, getLockType, getNativeProperties, getOperatorDescProperties, getPartitionKeys, getPartitionNames, getPartitionsByExpr, getPartitionTransformSpec, getRowId, getStorageFormatDescriptor, getStorageHandlerInfo, getType, getURIForAuth, hasAppendsOnly, isAllowedAlterOperation, isMetadataTableSupported, isTableMetaRefSupported, isTimeTravelAllowed, isValidMetadataTable, prepareAlterTableEnvironmentContext, setColStatistics, setTableLocationForCTAS, setTableParametersForCTLT, shouldOverwrite, showPartitions, sortColumns, storageHandlerCommit, storageHandlerCommit, supportsAcidOperations, supportsAppendData, supportsPartitionTransform, supportsSortColumns, supportsTruncateOnNonNativeTables, validatePartSpec, validateSinkDesc

Constructor Detail

SyslogStorageHandler

public SyslogStorageHandler()

Method Detail

getInputFormatClass

public Class<? extends org.apache.hadoop.mapred.InputFormat> getInputFormatClass()

Specified by:

getInputFormatClass in interface HiveStorageHandler

Returns:

Class providing an implementation of InputFormat

getOutputFormatClass

public Class<? extends org.apache.hadoop.mapred.OutputFormat> getOutputFormatClass()

Specified by:

getOutputFormatClass in interface HiveStorageHandler

Returns:

Class providing an implementation of OutputFormat

getSerDeClass

public Class<? extends AbstractSerDe> getSerDeClass()

Specified by:

getSerDeClass in interface HiveStorageHandler

Returns:

Class providing an implementation of AbstractSerDe

getMetaHook

public HiveMetaHook getMetaHook()

Specified by:

getMetaHook in interface HiveStorageHandler

Returns:

metadata hook implementation, or null if this storage handler does not need any metadata notifications

getAuthorizationProvider

public HiveAuthorizationProvider getAuthorizationProvider()

Description copied from interface: HiveStorageHandler

Returns the implementation specific authorization provider

Specified by:

getAuthorizationProvider in interface HiveStorageHandler

Returns:

authorization provider

configureInputJobProperties

public void configureInputJobProperties(TableDesc tableDesc,
                                        Map<String,String> jobProperties)

Description copied from interface: HiveStorageHandler

This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc). Key value pairs passed into jobProperties are guaranteed to be set in the job's configuration object. User's can retrieve "context" information from tableDesc. User's should avoid mutating tableDesc and only make changes in jobProperties. This method is expected to be idempotent such that a job called with the same tableDesc values should return the same key-value pairs in jobProperties. Any external state set by this method should remain the same if this method is called again. It is up to the user to determine how best guarantee this invariant. This method in particular is to create a configuration for input.

Specified by:

configureInputJobProperties in interface HiveStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureOutputJobProperties

public void configureOutputJobProperties(TableDesc tableDesc,
                                         Map<String,String> jobProperties)

Description copied from interface: HiveStorageHandler

This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc). Key value pairs passed into jobProperties are guaranteed to be set in the job's configuration object. User's can retrieve "context" information from tableDesc. User's should avoid mutating tableDesc and only make changes in jobProperties. This method is expected to be idempotent such that a job called with the same tableDesc values should return the same key-value pairs in jobProperties. Any external state set by this method should remain the same if this method is called again. It is up to the user to determine how best guarantee this invariant. This method in particular is to create a configuration for output.

Specified by:

configureOutputJobProperties in interface HiveStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureTableJobProperties

public void configureTableJobProperties(TableDesc tableDesc,
                                        Map<String,String> jobProperties)

Description copied from interface: HiveStorageHandler

Deprecated use configureInputJobProperties/configureOutputJobProperties methods instead. Configures properties for a job based on the definition of the source or target table it accesses.

Specified by:

configureTableJobProperties in interface HiveStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureJobConf

public void configureJobConf(TableDesc tableDesc,
                             org.apache.hadoop.mapred.JobConf jobConf)

Description copied from interface: HiveStorageHandler

Called just before submitting MapReduce job.

Specified by:

configureJobConf in interface HiveStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobConf - jobConf for MapReduce job

configureInputJobCredentials

public void configureInputJobCredentials(TableDesc tableDesc,
                                         Map<String,String> secrets)

Description copied from interface: HiveStorageHandler

This method is called to allow the StorageHandlers the chance to populate secret keys into the job's credentials.

Specified by:

configureInputJobCredentials in interface HiveStorageHandler

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

toString

public String toString()

Overrides:

toString in class Object