org.apache.carbondata.hadoop.api

Class CarbonInputFormat<T>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

org.apache.carbondata.hadoop.api.CarbonInputFormat<T>

Type Parameters:

T -

Direct Known Subclasses:

CarbonFileInputFormat, CarbonTableInputFormat

public abstract class CarbonInputFormat<T>
extends org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

Base class for carbondata input format, there are two input format implementations: 1. CarbonFileInputFormat: for reading carbondata files without table level metadata support. 2. CarbonTableInputFormat: for reading carbondata files with table level metadata support, such as segment and explicit schema metadata.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATABASE_NAME
protected int	hitedStreamFiles
static String	INPUT_SEGMENT_NUMBERS
protected int	numBlocks
protected int	numSegments
protected int	numStreamFiles
protected int	numStreamSegments
static String	TABLE_NAME

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

Constructor Summary

Constructors

Constructor and Description
CarbonInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.carbondata.core.scan.model.QueryModel	createQueryModel(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
org.apache.hadoop.mapreduce.RecordReader<Void,T>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
org.apache.carbondata.core.metadata.AbsoluteTableIdentifier	getAbsoluteTableIdentifier(org.apache.hadoop.conf.Configuration configuration)
static boolean	getAccessStreamingSegments(org.apache.hadoop.conf.Configuration configuration)
static String	getColumnProjection(org.apache.hadoop.conf.Configuration configuration)
static String	getDatabaseName(org.apache.hadoop.conf.Configuration configuration)
protected List<CarbonInputSplit>	getDataBlocksOfSegment(org.apache.hadoop.mapreduce.JobContext job, org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable, org.apache.carbondata.core.scan.expression.Expression expression, BitSet matchedPartitions, List<org.apache.carbondata.core.datamap.Segment> segmentIds, org.apache.carbondata.core.metadata.schema.PartitionInfo partitionInfo, List<Integer> oldPartitionIdList) get data blocks of given segment
static org.apache.carbondata.core.util.DataTypeConverter	getDataTypeConverter(org.apache.hadoop.conf.Configuration configuration)
protected org.apache.carbondata.core.scan.expression.Expression	getFilterPredicates(org.apache.hadoop.conf.Configuration configuration)
int	getHitedStreamFiles()
int	getNumBlocks()
int	getNumSegments()
int	getNumStreamFiles()
int	getNumStreamSegments()
abstract org.apache.carbondata.core.metadata.schema.table.CarbonTable	getOrCreateCarbonTable(org.apache.hadoop.conf.Configuration configuration) Get the cached CarbonTable or create it by TableInfo in `configuration`
static List<org.apache.carbondata.core.indexstore.PartitionSpec>	getPartitionsToPrune(org.apache.hadoop.conf.Configuration configuration) get list of partitions to prune
static org.apache.carbondata.core.readcommitter.ReadCommittedScope	getReadCommittedScope(org.apache.hadoop.conf.Configuration configuration)
CarbonReadSupport<T>	getReadSupportClass(org.apache.hadoop.conf.Configuration configuration)
abstract List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext job) Configurations FileInputFormat.INPUT_DIR are used to get table path to read.
protected static org.apache.carbondata.core.metadata.schema.table.TableInfo	getTableInfo(org.apache.hadoop.conf.Configuration configuration) Get TableInfo object from `configuration`
static String	getTableName(org.apache.hadoop.conf.Configuration configuration)
static boolean	getValidateSegmentsToAccess(org.apache.hadoop.conf.Configuration configuration) get list of segment to access
static boolean	isFgDataMapPruningEnable(org.apache.hadoop.conf.Configuration configuration)
protected boolean	isSplitable(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path filename)
String[]	projectAllColumns(org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable) Project all Columns for carbon reader
static void	setAccessStreamingSegments(org.apache.hadoop.conf.Configuration configuration, Boolean validate)
static void	setCarbonReadSupport(org.apache.hadoop.conf.Configuration configuration, Class<? extends CarbonReadSupport> readSupportClass)
static void	setColumnProjection(org.apache.hadoop.conf.Configuration configuration, CarbonProjection projection) Set the column projection column names from CarbonProjection
static void	setColumnProjection(org.apache.hadoop.conf.Configuration configuration, String[] projectionColumns) Set the column projection column names
static void	setDatabaseName(org.apache.hadoop.conf.Configuration configuration, String databaseName)
static void	setDataTypeConverter(org.apache.hadoop.conf.Configuration configuration, Class<? extends org.apache.carbondata.core.util.DataTypeConverter> converterClass) It is optional, if user does not set then it reads from store
static void	setFgDataMapPruning(org.apache.hadoop.conf.Configuration configuration, boolean enable)
static void	setFilterPredicates(org.apache.hadoop.conf.Configuration configuration, org.apache.carbondata.core.scan.expression.Expression filterExpression) It sets unresolved filter expression.
static void	setPartitionIdList(org.apache.hadoop.conf.Configuration configuration, List<String> partitionIds)
static void	setPartitionsToPrune(org.apache.hadoop.conf.Configuration configuration, List<org.apache.carbondata.core.indexstore.PartitionSpec> partitions) set list of partitions to prune
static void	setQuerySegment(org.apache.hadoop.conf.Configuration conf, org.apache.carbondata.core.metadata.AbsoluteTableIdentifier identifier) Set `CARBON_INPUT_SEGMENTS` from property to configuration
static void	setQuerySegment(org.apache.hadoop.conf.Configuration conf, String segmentList) Set `CARBON_INPUT_SEGMENTS` from property to configuration
static void	setReadCommittedScope(org.apache.hadoop.conf.Configuration configuration, org.apache.carbondata.core.readcommitter.ReadCommittedScope committedScope)
static void	setSegmentsToAccess(org.apache.hadoop.conf.Configuration configuration, List<org.apache.carbondata.core.datamap.Segment> validSegments) Set list of segments to access
static void	setTableInfo(org.apache.hadoop.conf.Configuration configuration, org.apache.carbondata.core.metadata.schema.table.TableInfo tableInfo) Set the `tableInfo` in `configuration`
static void	setTableName(org.apache.hadoop.conf.Configuration configuration, String tableName)
static void	setTablePath(org.apache.hadoop.conf.Configuration configuration, String tablePath)
static void	setTransactionalTable(org.apache.hadoop.conf.Configuration configuration, boolean isTransactionalTable)
static void	setValidateSegmentsToAccess(org.apache.hadoop.conf.Configuration configuration, Boolean validate) set list of segment to access

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, listStatus, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INPUT_SEGMENT_NUMBERS

public static final String INPUT_SEGMENT_NUMBERS

See Also:

Constant Field Values

DATABASE_NAME

public static final String DATABASE_NAME

See Also:

Constant Field Values

TABLE_NAME

public static final String TABLE_NAME

See Also:

Constant Field Values

numSegments

protected int numSegments

numStreamSegments

protected int numStreamSegments

numStreamFiles

protected int numStreamFiles

hitedStreamFiles

protected int hitedStreamFiles

numBlocks

protected int numBlocks

Constructor Detail

CarbonInputFormat

public CarbonInputFormat()

Method Detail

getNumSegments

public int getNumSegments()

getNumStreamSegments

public int getNumStreamSegments()

getNumStreamFiles

public int getNumStreamFiles()

getHitedStreamFiles

public int getHitedStreamFiles()

getNumBlocks

public int getNumBlocks()

setTableInfo

public static void setTableInfo(org.apache.hadoop.conf.Configuration configuration,
                                org.apache.carbondata.core.metadata.schema.table.TableInfo tableInfo)
                         throws IOException

Set the `tableInfo` in `configuration`

Throws:

IOException

getTableInfo

protected static org.apache.carbondata.core.metadata.schema.table.TableInfo getTableInfo(org.apache.hadoop.conf.Configuration configuration)
                                                                                  throws IOException

Get TableInfo object from `configuration`

Throws:

IOException

getOrCreateCarbonTable

public abstract org.apache.carbondata.core.metadata.schema.table.CarbonTable getOrCreateCarbonTable(org.apache.hadoop.conf.Configuration configuration)
                                                                                             throws IOException

Get the cached CarbonTable or create it by TableInfo in `configuration`

Throws:

IOException

setTablePath

public static void setTablePath(org.apache.hadoop.conf.Configuration configuration,
                                String tablePath)

setTransactionalTable

public static void setTransactionalTable(org.apache.hadoop.conf.Configuration configuration,
                                         boolean isTransactionalTable)

setPartitionIdList

public static void setPartitionIdList(org.apache.hadoop.conf.Configuration configuration,
                                      List<String> partitionIds)

setFilterPredicates

public static void setFilterPredicates(org.apache.hadoop.conf.Configuration configuration,
                                       org.apache.carbondata.core.scan.expression.Expression filterExpression)

It sets unresolved filter expression.

Parameters:

configuration -

setColumnProjection

public static void setColumnProjection(org.apache.hadoop.conf.Configuration configuration,
                                       String[] projectionColumns)

Set the column projection column names

Parameters:

configuration - Configuration info

projectionColumns - projection columns name

setColumnProjection

public static void setColumnProjection(org.apache.hadoop.conf.Configuration configuration,
                                       CarbonProjection projection)

Set the column projection column names from CarbonProjection

Parameters:

configuration - Configuration info

projection - CarbonProjection object that includes unique projection column name

getColumnProjection

public static String getColumnProjection(org.apache.hadoop.conf.Configuration configuration)

setFgDataMapPruning

public static void setFgDataMapPruning(org.apache.hadoop.conf.Configuration configuration,
                                       boolean enable)

isFgDataMapPruningEnable

public static boolean isFgDataMapPruningEnable(org.apache.hadoop.conf.Configuration configuration)

setSegmentsToAccess

public static void setSegmentsToAccess(org.apache.hadoop.conf.Configuration configuration,
                                       List<org.apache.carbondata.core.datamap.Segment> validSegments)

Set list of segments to access

setQuerySegment

public static void setQuerySegment(org.apache.hadoop.conf.Configuration conf,
                                   org.apache.carbondata.core.metadata.AbsoluteTableIdentifier identifier)

Set `CARBON_INPUT_SEGMENTS` from property to configuration

setQuerySegment

public static void setQuerySegment(org.apache.hadoop.conf.Configuration conf,
                                   String segmentList)

Set `CARBON_INPUT_SEGMENTS` from property to configuration

setValidateSegmentsToAccess

public static void setValidateSegmentsToAccess(org.apache.hadoop.conf.Configuration configuration,
                                               Boolean validate)

set list of segment to access

getValidateSegmentsToAccess

public static boolean getValidateSegmentsToAccess(org.apache.hadoop.conf.Configuration configuration)

get list of segment to access

setPartitionsToPrune

public static void setPartitionsToPrune(org.apache.hadoop.conf.Configuration configuration,
                                        List<org.apache.carbondata.core.indexstore.PartitionSpec> partitions)

set list of partitions to prune

getPartitionsToPrune

public static List<org.apache.carbondata.core.indexstore.PartitionSpec> getPartitionsToPrune(org.apache.hadoop.conf.Configuration configuration)
                                                                                      throws IOException

get list of partitions to prune

Throws:

IOException

getAbsoluteTableIdentifier

public org.apache.carbondata.core.metadata.AbsoluteTableIdentifier getAbsoluteTableIdentifier(org.apache.hadoop.conf.Configuration configuration)
                                                                                       throws IOException

Throws:

IOException

setReadCommittedScope

public static void setReadCommittedScope(org.apache.hadoop.conf.Configuration configuration,
                                         org.apache.carbondata.core.readcommitter.ReadCommittedScope committedScope)

getReadCommittedScope

public static org.apache.carbondata.core.readcommitter.ReadCommittedScope getReadCommittedScope(org.apache.hadoop.conf.Configuration configuration)
                                                                                         throws IOException

Throws:

IOException

getSplits

public abstract List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext job)
                                                                throws IOException

Configurations FileInputFormat.INPUT_DIR are used to get table path to read.

Overrides:

getSplits in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

Parameters:

job -

Returns:

List list of CarbonInputSplit

Throws:

IOException

getFilterPredicates

protected org.apache.carbondata.core.scan.expression.Expression getFilterPredicates(org.apache.hadoop.conf.Configuration configuration)

getDataBlocksOfSegment

protected List<CarbonInputSplit> getDataBlocksOfSegment(org.apache.hadoop.mapreduce.JobContext job,
                                                        org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable,
                                                        org.apache.carbondata.core.scan.expression.Expression expression,
                                                        BitSet matchedPartitions,
                                                        List<org.apache.carbondata.core.datamap.Segment> segmentIds,
                                                        org.apache.carbondata.core.metadata.schema.PartitionInfo partitionInfo,
                                                        List<Integer> oldPartitionIdList)
                                                 throws IOException

get data blocks of given segment

Throws:

IOException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<Void,T> createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit,
                                                                           org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
                                                                    throws IOException,
                                                                           InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<Void,T>

Throws:

IOException

InterruptedException

createQueryModel

public org.apache.carbondata.core.scan.model.QueryModel createQueryModel(org.apache.hadoop.mapreduce.InputSplit inputSplit,
                                                                         org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
                                                                  throws IOException

Throws:

IOException

getReadSupportClass

public CarbonReadSupport<T> getReadSupportClass(org.apache.hadoop.conf.Configuration configuration)

isSplitable

protected boolean isSplitable(org.apache.hadoop.mapreduce.JobContext context,
                              org.apache.hadoop.fs.Path filename)

Overrides:

isSplitable in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

setCarbonReadSupport

public static void setCarbonReadSupport(org.apache.hadoop.conf.Configuration configuration,
                                        Class<? extends CarbonReadSupport> readSupportClass)

setDataTypeConverter

public static void setDataTypeConverter(org.apache.hadoop.conf.Configuration configuration,
                                        Class<? extends org.apache.carbondata.core.util.DataTypeConverter> converterClass)

It is optional, if user does not set then it reads from store

Parameters:

configuration -

converterClass - is the Data type converter for different computing engine

getDataTypeConverter

public static org.apache.carbondata.core.util.DataTypeConverter getDataTypeConverter(org.apache.hadoop.conf.Configuration configuration)
                                                                              throws IOException

Throws:

IOException

setDatabaseName

public static void setDatabaseName(org.apache.hadoop.conf.Configuration configuration,
                                   String databaseName)

getDatabaseName

public static String getDatabaseName(org.apache.hadoop.conf.Configuration configuration)
                              throws org.apache.carbondata.core.exception.InvalidConfigurationException

Throws:

org.apache.carbondata.core.exception.InvalidConfigurationException

setTableName

public static void setTableName(org.apache.hadoop.conf.Configuration configuration,
                                String tableName)

getTableName

public static String getTableName(org.apache.hadoop.conf.Configuration configuration)
                           throws org.apache.carbondata.core.exception.InvalidConfigurationException

Throws:

org.apache.carbondata.core.exception.InvalidConfigurationException

setAccessStreamingSegments

public static void setAccessStreamingSegments(org.apache.hadoop.conf.Configuration configuration,
                                              Boolean validate)
                                       throws org.apache.carbondata.core.exception.InvalidConfigurationException

Throws:

org.apache.carbondata.core.exception.InvalidConfigurationException

getAccessStreamingSegments

public static boolean getAccessStreamingSegments(org.apache.hadoop.conf.Configuration configuration)

projectAllColumns

public String[] projectAllColumns(org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable)

Project all Columns for carbon reader

Parameters:

carbonTable -

Returns:

String araay of columnNames