org.apache.hadoop.hive.kafka

Class KafkaInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,KafkaWritable>

org.apache.hadoop.hive.kafka.KafkaInputFormat

All Implemented Interfaces:

VectorizedInputFormatInterface, org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,KafkaWritable>

public class KafkaInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,KafkaWritable>
implements org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,KafkaWritable>, VectorizedInputFormatInterface

Kafka puller input format to read records from a Kafka Queue. The input split will contain the set of topic partition and start/end offsets. Records will be returned as bytes array.

Constructor Summary

Constructors

Constructor and Description
KafkaInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.NullWritable,KafkaWritable>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,KafkaWritable>	getRecordReader(org.apache.hadoop.mapred.InputSplit inputSplit, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.mapred.Reporter reporter)
org.apache.hadoop.mapred.InputSplit[]	getSplits(org.apache.hadoop.mapred.JobConf jobConf, int i)
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext jobContext)
VectorizedSupport.Support[]	getSupportedFeatures()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.ql.exec.vector.VectorizedInputFormatInterface

getSupportedFeatures

Constructor Detail

KafkaInputFormat

public KafkaInputFormat()

Method Detail

getSplits

public org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf jobConf,
                                                       int i)
                                                throws IOException

Specified by:

getSplits in interface org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,KafkaWritable>

Throws:

IOException

getRecordReader

public org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,KafkaWritable> getRecordReader(org.apache.hadoop.mapred.InputSplit inputSplit,
                                                                                                              org.apache.hadoop.mapred.JobConf jobConf,
                                                                                                              org.apache.hadoop.mapred.Reporter reporter)

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,KafkaWritable>

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext jobContext)
                                                       throws IOException,
                                                              InterruptedException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,KafkaWritable>

Throws:

IOException

InterruptedException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.NullWritable,KafkaWritable> createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit,
                                                                                                                    org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,KafkaWritable>

getSupportedFeatures

public VectorizedSupport.Support[] getSupportedFeatures()

Specified by:

getSupportedFeatures in interface VectorizedInputFormatInterface