org.apache.druid.indexer

Class HadoopDruidIndexerConfig

java.lang.Object

org.apache.druid.indexer.HadoopDruidIndexerConfig

public class HadoopDruidIndexerConfig
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HadoopDruidIndexerConfig.IndexJobCounters

Field Summary

Fields

Modifier and Type	Field and Description
static IndexIO	INDEX_IO
static com.fasterxml.jackson.databind.ObjectMapper	JSON_MAPPER
static Properties	PROPERTIES Hadoop tasks running in an Indexer process need a reference to the Properties instance created in PropertiesModule so that the task sees properties that were specified in Druid's config files.

Constructor Summary

Constructors

Constructor and Description
HadoopDruidIndexerConfig(HadoopIngestionSpec spec)

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.Job	addInputPaths(org.apache.hadoop.mapreduce.Job job) Job instance should have Configuration set (by calling addJobProperties(Job) or via injected system properties) before this method is called.
static HadoopDruidIndexerConfig	fromConfiguration(org.apache.hadoop.conf.Configuration conf)
static HadoopDruidIndexerConfig	fromDistributedFileSystem(String path)
static HadoopDruidIndexerConfig	fromFile(File file)
static HadoopDruidIndexerConfig	fromSpec(HadoopIngestionSpec spec)
static HadoopDruidIndexerConfig	fromString(String str)
Map<String,String>	getAllowedProperties()
String	getDataSource()
GranularitySpec	getGranularitySpec()
IndexSpec	getIndexSpec()
IndexSpec	getIndexSpecForIntermediatePersists()
List<org.joda.time.Interval>	getInputIntervals()
com.google.common.base.Optional<List<org.joda.time.Interval>>	getIntervals()
int	getMaxParseExceptions()
InputRowParser	getParser()
DimensionBasedPartitionsSpec	getPartitionsSpec()
PathSpec	getPathSpec()
HadoopIngestionSpec	getSchema()
HadoopyShardSpec	getShardSpec(Bucket bucket)
int	getTargetPartitionSize()
String	getWorkingPath()
void	intoConfiguration(org.apache.hadoop.mapreduce.Job job)
boolean	isCombineText()
boolean	isLogParseExceptions()
boolean	isUpdaterJobSpecSet()
void	setGranularitySpec(GranularitySpec granularitySpec)
void	setShardSpecs(Map<Long,List<HadoopyShardSpec>> shardSpecs)
void	verify()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

JSON_MAPPER

public static final com.fasterxml.jackson.databind.ObjectMapper JSON_MAPPER

INDEX_IO

public static final IndexIO INDEX_IO

PROPERTIES

public static final Properties PROPERTIES

Hadoop tasks running in an Indexer process need a reference to the Properties instance created in PropertiesModule so that the task sees properties that were specified in Druid's config files.

This is not strictly necessary for Peon-based tasks which have all properties, including config file properties, specified on their command line by ForkingTaskRunner (so they could use System.getProperties() only), but we always use the injected Properties for consistency.

Constructor Detail

HadoopDruidIndexerConfig

public HadoopDruidIndexerConfig(HadoopIngestionSpec spec)

Method Detail

fromSpec

public static HadoopDruidIndexerConfig fromSpec(HadoopIngestionSpec spec)

fromFile

public static HadoopDruidIndexerConfig fromFile(File file)

fromString

public static HadoopDruidIndexerConfig fromString(String str)

fromDistributedFileSystem

public static HadoopDruidIndexerConfig fromDistributedFileSystem(String path)

fromConfiguration

public static HadoopDruidIndexerConfig fromConfiguration(org.apache.hadoop.conf.Configuration conf)

getSchema

public HadoopIngestionSpec getSchema()

getPathSpec

public PathSpec getPathSpec()

getDataSource

public String getDataSource()

getGranularitySpec

public GranularitySpec getGranularitySpec()

setGranularitySpec

public void setGranularitySpec(GranularitySpec granularitySpec)

getPartitionsSpec

public DimensionBasedPartitionsSpec getPartitionsSpec()

getIndexSpec

public IndexSpec getIndexSpec()

getIndexSpecForIntermediatePersists

public IndexSpec getIndexSpecForIntermediatePersists()

setShardSpecs

public void setShardSpecs(Map<Long,List<HadoopyShardSpec>> shardSpecs)

getIntervals

public com.google.common.base.Optional<List<org.joda.time.Interval>> getIntervals()

getTargetPartitionSize

public int getTargetPartitionSize()

isUpdaterJobSpecSet

public boolean isUpdaterJobSpecSet()

isCombineText

public boolean isCombineText()

getParser

public InputRowParser getParser()

getShardSpec

public HadoopyShardSpec getShardSpec(Bucket bucket)

isLogParseExceptions

public boolean isLogParseExceptions()

getMaxParseExceptions

public int getMaxParseExceptions()

getAllowedProperties

public Map<String,String> getAllowedProperties()

addInputPaths

public org.apache.hadoop.mapreduce.Job addInputPaths(org.apache.hadoop.mapreduce.Job job)
                                              throws IOException

Job instance should have Configuration set (by calling addJobProperties(Job) or via injected system properties) before this method is called. The PathSpec may create objects which depend on the values of these configurations.

Throws:

IOException

getInputIntervals

public List<org.joda.time.Interval> getInputIntervals()

getWorkingPath

public String getWorkingPath()

intoConfiguration

public void intoConfiguration(org.apache.hadoop.mapreduce.Job job)

verify

public void verify()