org.apache.parquet.hadoop.rewrite

Class RewriteOptions.Builder

java.lang.Object

org.apache.parquet.hadoop.rewrite.RewriteOptions.Builder

Enclosing class:

RewriteOptions

public static class RewriteOptions.Builder
extends Object

Builder for RewriteOptions which is used for constructing ParquetRewriter.

Constructor Summary

Constructors

Constructor and Description
Builder(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.fs.Path> inputFiles, List<org.apache.hadoop.fs.Path> inputFilesToJoin, org.apache.hadoop.fs.Path outputFile) Create a builder to create a RewriterOptions.
Builder(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.fs.Path> inputFiles, org.apache.hadoop.fs.Path outputFile) Create a builder to create a RewriterOptions.
Builder(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path inputFile, org.apache.hadoop.fs.Path outputFile) Create a builder to create a RewriterOptions.
Builder(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path inputFile, org.apache.hadoop.fs.Path inputFileToJoin, org.apache.hadoop.fs.Path outputFile) Create a builder to create a RewriterOptions.
Builder(ParquetConfiguration conf, InputFile inputFile, InputFile inputFileToJoin, OutputFile outputFile) Create a builder to create a RewriterOptions.
Builder(ParquetConfiguration conf, InputFile inputFile, OutputFile outputFile) Create a builder to create a RewriterOptions.
Builder(ParquetConfiguration conf, List<InputFile> inputFiles, List<InputFile> inputFilesToJoin, OutputFile outputFile) Create a builder to create a RewriterOptions.
Builder(ParquetConfiguration conf, List<InputFile> inputFiles, OutputFile outputFile) Create a builder to create a RewriterOptions.

Method Summary

Modifier and Type	Method and Description
RewriteOptions.Builder	addInputFile(InputFile inputFile) Add an input file to read from.
RewriteOptions.Builder	addInputFile(org.apache.hadoop.fs.Path path) Add an input file to read from.
RewriteOptions.Builder	addInputFilesToJoin(InputFile fileToJoin) Add an input file to join.
RewriteOptions.Builder	addInputFileToJoinColumns(org.apache.hadoop.fs.Path path) Add an input join file to read from.
RewriteOptions	build() Build the RewriterOptions.
RewriteOptions.Builder	encrypt(List<String> encryptColumns) Set the columns to encrypt.
RewriteOptions.Builder	encryptionProperties(FileEncryptionProperties fileEncryptionProperties) Set the encryption properties to use for the output file.
RewriteOptions.Builder	ignoreJoinFilesMetadata(boolean ignoreJoinFilesMetadata) Set a flag whether metadata from join files should be ignored.
RewriteOptions.Builder	indexCacheStrategy(IndexCache.CacheStrategy cacheStrategy) Set the index(ColumnIndex, Offset and BloomFilter) cache strategy.
RewriteOptions.Builder	mask(Map<String,MaskMode> maskColumns) Set the columns to mask.
RewriteOptions.Builder	overwriteInputWithJoinColumns(boolean overwriteInputWithJoinColumns) Set a flag whether columns from join files need to overwrite columns from the main input files.
RewriteOptions.Builder	prune(List<String> columns) Set the columns to prune.
RewriteOptions.Builder	renameColumns(Map<String,String> renameColumns) Set the columns to be renamed.
RewriteOptions.Builder	transform(CompressionCodecName newCodecName) Set the compression codec to use for the output file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Builder

public Builder(org.apache.hadoop.conf.Configuration conf,
               org.apache.hadoop.fs.Path inputFile,
               org.apache.hadoop.fs.Path inputFileToJoin,
               org.apache.hadoop.fs.Path outputFile)

Create a builder to create a RewriterOptions.

Parameters:

conf - configuration for reading from input files and writing to output file

inputFile - input file path to read from

inputFileToJoin - input join file path to read from

outputFile - output file path to rewrite to

Builder

public Builder(org.apache.hadoop.conf.Configuration conf,
               org.apache.hadoop.fs.Path inputFile,
               org.apache.hadoop.fs.Path outputFile)

Create a builder to create a RewriterOptions.

Parameters:

conf - configuration for reading from input files and writing to output file

inputFile - input file path to read from

outputFile - output file path to rewrite to

Builder

public Builder(ParquetConfiguration conf,
               InputFile inputFile,
               OutputFile outputFile)

Create a builder to create a RewriterOptions.

Parameters:

conf - configuration for reading from input files and writing to output file

inputFile - input file to read from

outputFile - output file to rewrite to

Builder

public Builder(ParquetConfiguration conf,
               InputFile inputFile,
               InputFile inputFileToJoin,
               OutputFile outputFile)

Create a builder to create a RewriterOptions.

Parameters:

conf - configuration for reading from input files and writing to output file

inputFile - input file to read from

inputFileToJoin - input join file to read from

outputFile - output file to rewrite to

Builder

public Builder(org.apache.hadoop.conf.Configuration conf,
               List<org.apache.hadoop.fs.Path> inputFiles,
               org.apache.hadoop.fs.Path outputFile)

Create a builder to create a RewriterOptions.

Please note that if merging more than one file, the schema of all files must be the same. Otherwise, the rewrite will fail.

The rewrite will keep original row groups from all input files. This may not be optimal if row groups are very small and will not solve small file problems. Instead, it will make it worse to have a large file footer in the output file. TODO: support rewrite by record to break the original row groups into reasonable ones.

See ParquetRewriter for more details.

Parameters:

conf - configuration for reading from input files and writing to output file

inputFiles - list of input file paths to read from

outputFile - output file path to rewrite to

Builder

public Builder(ParquetConfiguration conf,
               List<InputFile> inputFiles,
               OutputFile outputFile)

Create a builder to create a RewriterOptions.

Please note that if merging more than one file, the schema of all files must be the same. Otherwise, the rewrite will fail.

The rewrite will keep original row groups from all input files. This may not be optimal if row groups are very small and will not solve small file problems. Instead, it will make it worse to have a large file footer in the output file. TODO: support rewrite by record to break the original row groups into reasonable ones.

See ParquetRewriter for more details.

Parameters:

conf - configuration for reading from input files and writing to output file

inputFiles - list of input file paths to read from

outputFile - output file path to rewrite to

Builder

public Builder(org.apache.hadoop.conf.Configuration conf,
               List<org.apache.hadoop.fs.Path> inputFiles,
               List<org.apache.hadoop.fs.Path> inputFilesToJoin,
               org.apache.hadoop.fs.Path outputFile)

Create a builder to create a RewriterOptions.

Please note the schema of all files in each file group inputFiles and inputFilesToJoin must be the same while those two schemas can be different in comparison with each other. Otherwise, the rewrite will fail.

The rewrite will keep original row groups from all input files. This may not be optimal if row groups are very small and will not solve small file problems. Instead, it will make it worse to have a large file footer in the output file. TODO: support rewrite by record to break the original row groups into reasonable ones.

See ParquetRewriter for more details.

Parameters:

conf - configuration for reading from input files and writing to output file

inputFiles - list of input file paths to read from

inputFilesToJoin - list of input join file paths to read from

outputFile - output file path to rewrite to

Builder

public Builder(ParquetConfiguration conf,
               List<InputFile> inputFiles,
               List<InputFile> inputFilesToJoin,
               OutputFile outputFile)

Create a builder to create a RewriterOptions.

Please note the schema of all files in each file group inputFiles and inputFilesToJoin must be the same while those two schemas can be different in comparison with each other. Otherwise, the rewrite will fail.

The rewrite will keep original row groups from all input files. This may not be optimal if row groups are very small and will not solve small file problems. Instead, it will make it worse to have a large file footer in the output file.

See ParquetRewriter for more details.

Parameters:

conf - configuration for reading from input files and writing to output file

inputFiles - list of input file paths to read from

inputFilesToJoin - list of input join file paths to read from

outputFile - output file path to rewrite to

Method Detail

prune

public RewriteOptions.Builder prune(List<String> columns)

Set the columns to prune.

By default, all columns are kept.

Parameters:

columns - list of columns to prune

Returns:

self

transform

public RewriteOptions.Builder transform(CompressionCodecName newCodecName)

Set the compression codec to use for the output file.

By default, the codec is the same as the input file.

Parameters:

newCodecName - compression codec to use

Returns:

self

mask

public RewriteOptions.Builder mask(Map<String,MaskMode> maskColumns)

Set the columns to mask.

By default, no columns are masked.

Parameters:

maskColumns - map of columns to mask to the masking mode

Returns:

self

renameColumns

public RewriteOptions.Builder renameColumns(Map<String,String> renameColumns)

Set the columns to be renamed.

Note that nested columns can't be renamed, in case of GroupType column only top level column can be renamed.

Parameters:

renameColumns - map where keys are original names and values are new names

Returns:

self

encrypt

public RewriteOptions.Builder encrypt(List<String> encryptColumns)

Set the columns to encrypt.

By default, no columns are encrypted.

Parameters:

encryptColumns - list of columns to encrypt

Returns:

self

encryptionProperties

public RewriteOptions.Builder encryptionProperties(FileEncryptionProperties fileEncryptionProperties)

Set the encryption properties to use for the output file.

This is required if encrypting columns are not empty.

Parameters:

fileEncryptionProperties - encryption properties to use

Returns:

self

addInputFile

public RewriteOptions.Builder addInputFile(org.apache.hadoop.fs.Path path)

Add an input file to read from.

Parameters:

path - input file path to read from

Returns:

self

addInputFileToJoinColumns

public RewriteOptions.Builder addInputFileToJoinColumns(org.apache.hadoop.fs.Path path)

Add an input join file to read from.

Parameters:

path - input file path to read from

Returns:

self

addInputFile

public RewriteOptions.Builder addInputFile(InputFile inputFile)

Add an input file to read from.

Parameters:

inputFile - input file to read from

Returns:

self

addInputFilesToJoin

public RewriteOptions.Builder addInputFilesToJoin(InputFile fileToJoin)

Add an input file to join.

Parameters:

fileToJoin - input file to join

Returns:

self

indexCacheStrategy

public RewriteOptions.Builder indexCacheStrategy(IndexCache.CacheStrategy cacheStrategy)

Set the index(ColumnIndex, Offset and BloomFilter) cache strategy.

This could reduce the random seek while rewriting with PREFETCH_BLOCK strategy, NONE by default.

Parameters:

cacheStrategy - the index cache strategy, supports: IndexCache.CacheStrategy#NONE or IndexCache.CacheStrategy#PREFETCH_BLOCK

Returns:

self

overwriteInputWithJoinColumns

public RewriteOptions.Builder overwriteInputWithJoinColumns(boolean overwriteInputWithJoinColumns)

Set a flag whether columns from join files need to overwrite columns from the main input files.

By default, join files columns do not overwrite the main input file columns.

Parameters:

overwriteInputWithJoinColumns - a flag if columns from join files should overwrite columns from the main input files

Returns:

self

ignoreJoinFilesMetadata

public RewriteOptions.Builder ignoreJoinFilesMetadata(boolean ignoreJoinFilesMetadata)

Set a flag whether metadata from join files should be ignored.

By default, metadata is not ignored.

Parameters:

ignoreJoinFilesMetadata - a flag if metadata from join files should be ignored

Returns:

self

build

public RewriteOptions build()

Build the RewriterOptions.

Returns:

a RewriterOptions