org.apache.carbondata.hadoop.api

Class CarbonTableInputFormat<T>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

org.apache.carbondata.hadoop.api.CarbonInputFormat<T>

org.apache.carbondata.hadoop.api.CarbonTableInputFormat<T>

Type Parameters:

T -

public class CarbonTableInputFormat<T>
extends CarbonInputFormat<T>

InputFormat for reading carbondata files with table level metadata support, such as segment and explicit schema metadata.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATABASE_NAME
static String	INPUT_FILES
static String	INPUT_SEGMENT_NUMBERS
static String	TABLE_NAME

Fields inherited from class org.apache.carbondata.hadoop.api.CarbonInputFormat

hitedStreamFiles, numBlocks, numSegments, numStreamFiles, numStreamSegments

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

Constructor Summary

Constructors

Constructor and Description
CarbonTableInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.carbondata.core.mutate.data.BlockMappingVO	getBlockRowCount(org.apache.hadoop.mapreduce.Job job, org.apache.carbondata.core.metadata.schema.table.CarbonTable table, List<org.apache.carbondata.core.indexstore.PartitionSpec> partitions) Get the row count of the Block and mapping of segment and Block count.
org.apache.carbondata.core.metadata.schema.table.CarbonTable	getOrCreateCarbonTable(org.apache.hadoop.conf.Configuration configuration) Get the cached CarbonTable or create it by TableInfo in `configuration`
org.apache.carbondata.core.readcommitter.ReadCommittedScope	getReadCommitted(org.apache.hadoop.mapreduce.JobContext job, org.apache.carbondata.core.metadata.AbsoluteTableIdentifier identifier)
org.apache.carbondata.core.datamap.Segment[]	getSegmentsToAccess(org.apache.hadoop.mapreduce.JobContext job, org.apache.carbondata.core.readcommitter.ReadCommittedScope readCommittedScope) return valid segment to access
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext job) Configurations FileInputFormat.INPUT_DIR are used to get table path to read.
List<org.apache.hadoop.mapreduce.InputSplit>	getSplitsOfOneSegment(org.apache.hadoop.mapreduce.JobContext job, String targetSegment, List<Integer> oldPartitionIdList, org.apache.carbondata.core.metadata.schema.PartitionInfo partitionInfo) Read data in one segment.
List<org.apache.hadoop.mapreduce.InputSplit>	getSplitsOfStreaming(org.apache.hadoop.mapreduce.JobContext job, List<org.apache.carbondata.core.datamap.Segment> streamSegments, org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable)
List<org.apache.hadoop.mapreduce.InputSplit>	getSplitsOfStreaming(org.apache.hadoop.mapreduce.JobContext job, List<org.apache.carbondata.core.datamap.Segment> streamSegments, org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable, org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf filterResolverIntf) use file list in .carbonindex file to get the split of streaming.
protected org.apache.hadoop.mapreduce.lib.input.FileSplit	makeSplit(String segmentId, org.apache.hadoop.fs.Path file, long start, long length, String[] hosts, org.apache.carbondata.core.statusmanager.FileFormat fileFormat)
protected org.apache.hadoop.mapreduce.lib.input.FileSplit	makeSplit(String segmentId, org.apache.hadoop.fs.Path file, long start, long length, String[] hosts, String[] inMemoryHosts, org.apache.carbondata.core.statusmanager.FileFormat fileFormat)
void	refreshSegmentCacheIfRequired(org.apache.hadoop.mapreduce.JobContext job, org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable, org.apache.carbondata.core.statusmanager.SegmentUpdateStatusManager updateStatusManager, List<org.apache.carbondata.core.datamap.Segment> filteredSegmentToAccess) Method to check and refresh segment cache

Methods inherited from class org.apache.carbondata.hadoop.api.CarbonInputFormat

createQueryModel, createRecordReader, getAbsoluteTableIdentifier, getAccessStreamingSegments, getColumnProjection, getDatabaseName, getDataBlocksOfSegment, getDataTypeConverter, getFilterPredicates, getHitedStreamFiles, getNumBlocks, getNumSegments, getNumStreamFiles, getNumStreamSegments, getPartitionsToPrune, getReadCommittedScope, getReadSupportClass, getTableInfo, getTableName, getValidateSegmentsToAccess, isFgDataMapPruningEnable, isSplitable, projectAllColumns, setAccessStreamingSegments, setCarbonReadSupport, setColumnProjection, setColumnProjection, setDatabaseName, setDataTypeConverter, setFgDataMapPruning, setFilterPredicates, setPartitionIdList, setPartitionsToPrune, setQuerySegment, setQuerySegment, setReadCommittedScope, setSegmentsToAccess, setTableInfo, setTableName, setTablePath, setTransactionalTable, setValidateSegmentsToAccess

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, listStatus, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INPUT_SEGMENT_NUMBERS

public static final String INPUT_SEGMENT_NUMBERS

See Also:

Constant Field Values

INPUT_FILES

public static final String INPUT_FILES

See Also:

Constant Field Values

DATABASE_NAME

public static final String DATABASE_NAME

See Also:

Constant Field Values

TABLE_NAME

public static final String TABLE_NAME

See Also:

Constant Field Values

Constructor Detail

CarbonTableInputFormat

public CarbonTableInputFormat()

Method Detail

getOrCreateCarbonTable

public org.apache.carbondata.core.metadata.schema.table.CarbonTable getOrCreateCarbonTable(org.apache.hadoop.conf.Configuration configuration)
                                                                                    throws IOException

Get the cached CarbonTable or create it by TableInfo in `configuration`

Specified by:

getOrCreateCarbonTable in class CarbonInputFormat<T>

Throws:

IOException

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext job)
                                                       throws IOException

Configurations FileInputFormat.INPUT_DIR are used to get table path to read. Configurations FileInputFormat.INPUT_DIR are used to get table path to read.

Specified by:

getSplits in class CarbonInputFormat<T>

Parameters:

job -

Returns:

List list of CarbonInputSplit

Throws:

IOException

refreshSegmentCacheIfRequired

public void refreshSegmentCacheIfRequired(org.apache.hadoop.mapreduce.JobContext job,
                                          org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable,
                                          org.apache.carbondata.core.statusmanager.SegmentUpdateStatusManager updateStatusManager,
                                          List<org.apache.carbondata.core.datamap.Segment> filteredSegmentToAccess)
                                   throws IOException

Method to check and refresh segment cache

Parameters:

job -

carbonTable -

updateStatusManager -

filteredSegmentToAccess -

Throws:

IOException

getSplitsOfStreaming

public List<org.apache.hadoop.mapreduce.InputSplit> getSplitsOfStreaming(org.apache.hadoop.mapreduce.JobContext job,
                                                                         List<org.apache.carbondata.core.datamap.Segment> streamSegments,
                                                                         org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable)
                                                                  throws IOException

Throws:

IOException

getSplitsOfStreaming

public List<org.apache.hadoop.mapreduce.InputSplit> getSplitsOfStreaming(org.apache.hadoop.mapreduce.JobContext job,
                                                                         List<org.apache.carbondata.core.datamap.Segment> streamSegments,
                                                                         org.apache.carbondata.core.metadata.schema.table.CarbonTable carbonTable,
                                                                         org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf filterResolverIntf)
                                                                  throws IOException

use file list in .carbonindex file to get the split of streaming.

Throws:

IOException

makeSplit

protected org.apache.hadoop.mapreduce.lib.input.FileSplit makeSplit(String segmentId,
                                                                    org.apache.hadoop.fs.Path file,
                                                                    long start,
                                                                    long length,
                                                                    String[] hosts,
                                                                    org.apache.carbondata.core.statusmanager.FileFormat fileFormat)

makeSplit

protected org.apache.hadoop.mapreduce.lib.input.FileSplit makeSplit(String segmentId,
                                                                    org.apache.hadoop.fs.Path file,
                                                                    long start,
                                                                    long length,
                                                                    String[] hosts,
                                                                    String[] inMemoryHosts,
                                                                    org.apache.carbondata.core.statusmanager.FileFormat fileFormat)

getSplitsOfOneSegment

public List<org.apache.hadoop.mapreduce.InputSplit> getSplitsOfOneSegment(org.apache.hadoop.mapreduce.JobContext job,
                                                                          String targetSegment,
                                                                          List<Integer> oldPartitionIdList,
                                                                          org.apache.carbondata.core.metadata.schema.PartitionInfo partitionInfo)

Read data in one segment. For alter table partition statement

Parameters:

job -

targetSegment -

oldPartitionIdList - get old partitionId before partitionInfo was changed

Returns:

getSegmentsToAccess

public org.apache.carbondata.core.datamap.Segment[] getSegmentsToAccess(org.apache.hadoop.mapreduce.JobContext job,
                                                                        org.apache.carbondata.core.readcommitter.ReadCommittedScope readCommittedScope)

return valid segment to access

getBlockRowCount

public org.apache.carbondata.core.mutate.data.BlockMappingVO getBlockRowCount(org.apache.hadoop.mapreduce.Job job,
                                                                              org.apache.carbondata.core.metadata.schema.table.CarbonTable table,
                                                                              List<org.apache.carbondata.core.indexstore.PartitionSpec> partitions)
                                                                       throws IOException

Get the row count of the Block and mapping of segment and Block count.

Throws:

IOException

getReadCommitted

public org.apache.carbondata.core.readcommitter.ReadCommittedScope getReadCommitted(org.apache.hadoop.mapreduce.JobContext job,
                                                                                    org.apache.carbondata.core.metadata.AbsoluteTableIdentifier identifier)
                                                                             throws IOException

Throws:

IOException