org.apache.druid.indexer

Class JobHelper

java.lang.Object

org.apache.druid.indexer.JobHelper

public class JobHelper
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	JobHelper.DataPusher Simple interface for retry operations

Field Summary

Fields

Modifier and Type	Field and Description
static String	INDEX_ZIP

Constructor Summary

Constructors

Constructor and Description
JobHelper()

Method Summary

Modifier and Type	Method and Description
static void	authenticate() Does authenticate against a secured hadoop cluster In case of any bug fix make sure to fix the code at HdfsStorageAuthentication#authenticate as well.
static long	copyFileToZipStream(File file, ZipOutputStream zipOutputStream, org.apache.hadoop.util.Progressable progressable)
static org.apache.hadoop.fs.Path	distributedClassPath(org.apache.hadoop.fs.Path base)
static org.apache.hadoop.fs.Path	distributedClassPath(String path)
static void	ensurePaths(HadoopDruidIndexerConfig config)
static String	getJobTrackerAddress(org.apache.hadoop.conf.Configuration config)
static URI	getURIFromSegment(DataSegment dataSegment)
static void	injectDruidProperties(org.apache.hadoop.conf.Configuration configuration, HadoopDruidIndexerConfig hadoopDruidIndexerConfig)
static org.apache.hadoop.conf.Configuration	injectSystemProperties(org.apache.hadoop.conf.Configuration conf, HadoopDruidIndexerConfig hadoopDruidIndexerConfig)
static org.apache.hadoop.fs.Path	makeFileNamePath(org.apache.hadoop.fs.Path basePath, org.apache.hadoop.fs.FileSystem fs, DataSegment segmentTemplate, String baseFileName, DataSegmentPusher dataSegmentPusher)
static org.apache.hadoop.fs.Path	makeTmpPath(org.apache.hadoop.fs.Path basePath, org.apache.hadoop.fs.FileSystem fs, DataSegment segmentTemplate, org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID, DataSegmentPusher dataSegmentPusher)
static void	maybeDeleteIntermediatePath(boolean jobSucceeded, HadoopIngestionSpec indexerSchema)
static org.apache.hadoop.fs.Path	prependFSIfNullScheme(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static void	renameIndexFilesForSegments(HadoopIngestionSpec indexerSchema, List<DataSegmentAndIndexZipFilePath> segmentAndIndexZipFilePaths) Renames the index files for the segments.
static boolean	runJobs(List<Jobby> jobs)
static boolean	runSingleJob(Jobby job)
static DataSegmentAndIndexZipFilePath	serializeOutIndex(DataSegment segmentTemplate, org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.util.Progressable progressable, File mergedBase, org.apache.hadoop.fs.Path finalIndexZipFilePath, org.apache.hadoop.fs.Path tmpPath, DataSegmentPusher dataSegmentPusher)
static void	setupClasspath(org.apache.hadoop.fs.Path distributedClassPath, org.apache.hadoop.fs.Path intermediateClassPath, org.apache.hadoop.mapreduce.Job job) Uploads jar files to hdfs and configures the classpath.
static com.google.common.base.Predicate<Throwable>	shouldRetryPredicate()
static long	unzipNoGuava(org.apache.hadoop.fs.Path zip, org.apache.hadoop.conf.Configuration configuration, File outDir, org.apache.hadoop.util.Progressable progressable, org.apache.hadoop.io.retry.RetryPolicy retryPolicy)
static void	writeJobIdToFile(String hadoopJobIdFileName, String hadoopJobId)
static void	writeSegmentDescriptor(org.apache.hadoop.fs.FileSystem outputFS, DataSegmentAndIndexZipFilePath segmentAndPath, org.apache.hadoop.fs.Path descriptorPath, org.apache.hadoop.util.Progressable progressable)
static long	zipAndCopyDir(File baseDir, OutputStream baseOutputStream, org.apache.hadoop.util.Progressable progressable)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INDEX_ZIP

public static final String INDEX_ZIP

See Also:

Constant Field Values

Constructor Detail

JobHelper

public JobHelper()

Method Detail

distributedClassPath

public static org.apache.hadoop.fs.Path distributedClassPath(String path)

distributedClassPath

public static org.apache.hadoop.fs.Path distributedClassPath(org.apache.hadoop.fs.Path base)

authenticate

public static void authenticate()

Does authenticate against a secured hadoop cluster In case of any bug fix make sure to fix the code at HdfsStorageAuthentication#authenticate as well.

setupClasspath

public static void setupClasspath(org.apache.hadoop.fs.Path distributedClassPath,
                                  org.apache.hadoop.fs.Path intermediateClassPath,
                                  org.apache.hadoop.mapreduce.Job job)
                           throws IOException

Uploads jar files to hdfs and configures the classpath. Snapshot jar files are uploaded to intermediateClasspath and not shared across multiple jobs. Non-Snapshot jar files are uploaded to a distributedClasspath and shared across multiple jobs.

Parameters:

distributedClassPath - classpath shared across multiple jobs

intermediateClassPath - classpath exclusive for this job. used to upload SNAPSHOT jar files.

job - job to run

Throws:

IOException

shouldRetryPredicate

public static com.google.common.base.Predicate<Throwable> shouldRetryPredicate()

injectDruidProperties

public static void injectDruidProperties(org.apache.hadoop.conf.Configuration configuration,
                                         HadoopDruidIndexerConfig hadoopDruidIndexerConfig)

injectSystemProperties

public static org.apache.hadoop.conf.Configuration injectSystemProperties(org.apache.hadoop.conf.Configuration conf,
                                                                          HadoopDruidIndexerConfig hadoopDruidIndexerConfig)

ensurePaths

public static void ensurePaths(HadoopDruidIndexerConfig config)

writeJobIdToFile

public static void writeJobIdToFile(String hadoopJobIdFileName,
                                    String hadoopJobId)

runSingleJob

public static boolean runSingleJob(Jobby job)

runJobs

public static boolean runJobs(List<Jobby> jobs)

maybeDeleteIntermediatePath

public static void maybeDeleteIntermediatePath(boolean jobSucceeded,
                                               HadoopIngestionSpec indexerSchema)

serializeOutIndex

public static DataSegmentAndIndexZipFilePath serializeOutIndex(DataSegment segmentTemplate,
                                                               org.apache.hadoop.conf.Configuration configuration,
                                                               org.apache.hadoop.util.Progressable progressable,
                                                               File mergedBase,
                                                               org.apache.hadoop.fs.Path finalIndexZipFilePath,
                                                               org.apache.hadoop.fs.Path tmpPath,
                                                               DataSegmentPusher dataSegmentPusher)
                                                        throws IOException

Throws:

IOException

writeSegmentDescriptor

public static void writeSegmentDescriptor(org.apache.hadoop.fs.FileSystem outputFS,
                                          DataSegmentAndIndexZipFilePath segmentAndPath,
                                          org.apache.hadoop.fs.Path descriptorPath,
                                          org.apache.hadoop.util.Progressable progressable)
                                   throws IOException

Throws:

IOException

zipAndCopyDir

public static long zipAndCopyDir(File baseDir,
                                 OutputStream baseOutputStream,
                                 org.apache.hadoop.util.Progressable progressable)
                          throws IOException

Throws:

IOException

copyFileToZipStream

public static long copyFileToZipStream(File file,
                                       ZipOutputStream zipOutputStream,
                                       org.apache.hadoop.util.Progressable progressable)
                                throws IOException

Throws:

IOException

makeFileNamePath

public static org.apache.hadoop.fs.Path makeFileNamePath(org.apache.hadoop.fs.Path basePath,
                                                         org.apache.hadoop.fs.FileSystem fs,
                                                         DataSegment segmentTemplate,
                                                         String baseFileName,
                                                         DataSegmentPusher dataSegmentPusher)

makeTmpPath

public static org.apache.hadoop.fs.Path makeTmpPath(org.apache.hadoop.fs.Path basePath,
                                                    org.apache.hadoop.fs.FileSystem fs,
                                                    DataSegment segmentTemplate,
                                                    org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptID,
                                                    DataSegmentPusher dataSegmentPusher)

renameIndexFilesForSegments

public static void renameIndexFilesForSegments(HadoopIngestionSpec indexerSchema,
                                               List<DataSegmentAndIndexZipFilePath> segmentAndIndexZipFilePaths)
                                        throws IOException

Renames the index files for the segments. This works around some limitations of both FileContext (no s3n support) and NativeS3FileSystem.rename which will not overwrite. Note: segments should be renamed in the index task, not in a hadoop job, as race conditions between job retries can cause the final segment index file path to get clobbered.

Parameters:

indexerSchema - the hadoop ingestion spec

segmentAndIndexZipFilePaths - the list of segments with their currently stored tmp path and the final path that they should be renamed to.

Throws:

IOException

prependFSIfNullScheme

public static org.apache.hadoop.fs.Path prependFSIfNullScheme(org.apache.hadoop.fs.FileSystem fs,
                                                              org.apache.hadoop.fs.Path path)

unzipNoGuava

public static long unzipNoGuava(org.apache.hadoop.fs.Path zip,
                                org.apache.hadoop.conf.Configuration configuration,
                                File outDir,
                                org.apache.hadoop.util.Progressable progressable,
                                @Nullable
                                org.apache.hadoop.io.retry.RetryPolicy retryPolicy)
                         throws IOException

Throws:

IOException

getURIFromSegment

public static URI getURIFromSegment(DataSegment dataSegment)

getJobTrackerAddress

public static String getJobTrackerAddress(org.apache.hadoop.conf.Configuration config)