Package org.apache.pinot.common.segment.generation

Class SegmentGenerationUtils

java.lang.Object

org.apache.pinot.common.segment.generation.SegmentGenerationUtils

public class SegmentGenerationUtils
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
static String	PINOT_PLUGINS_DIR
static String	PINOT_PLUGINS_TAR_GZ

Method Summary

Modifier and Type	Method	Description
static String	generateSchemaURI(String controllerUri, String table)
static String	generateTableConfigURI(String controllerUri, String table)
static URI	getDirectoryURI(String uriStr)	Convert Directory URI String to URI Object, default to local file system scheme.
static String	getFileName(URI inputFileURI)	Extract file name from a given URI.
static URI	getFileURI(String uriStr, URI fullUriForPathOnlyUriStr)	Convert a File URI String to URI Object, use parent URI scheme/userInfo/host/port if sheme is not specified.
static URI	getRelativeOutputPath(URI baseInputDir, URI inputFile, URI outputDir)	Generate a relative output directory path when `useRelativePath` flag is on.
static Schema	getSchema(String schemaURIString)
static Schema	getSchema(String schemaURIString, String authToken)
static TableConfig	getTableConfig(String tableConfigURIStr)	Deprecated.
static TableConfig	getTableConfig(String tableConfigURIStr, String authToken)
static List<String>	listMatchedFilesWithRecursiveOption(PinotFS pinotFs, URI fileUri, String includePattern, String excludePattern, boolean searchRecursively)
static String	sanitizeURIString(String path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PINOT_PLUGINS_TAR_GZ

public static final String PINOT_PLUGINS_TAR_GZ

See Also:

Constant Field Values

PINOT_PLUGINS_DIR

public static final String PINOT_PLUGINS_DIR

See Also:

Constant Field Values

Method Detail

generateSchemaURI

public static String generateSchemaURI(String controllerUri,
                                       String table)

generateTableConfigURI

public static String generateTableConfigURI(String controllerUri,
                                            String table)

getSchema

public static Schema getSchema(String schemaURIString)

getSchema

public static Schema getSchema(String schemaURIString,
                               String authToken)

getTableConfig

@Deprecated
public static TableConfig getTableConfig(String tableConfigURIStr)

Deprecated.

getTableConfig

public static TableConfig getTableConfig(String tableConfigURIStr,
                                         String authToken)

getRelativeOutputPath

public static URI getRelativeOutputPath(URI baseInputDir,
                                        URI inputFile,
                                        URI outputDir)

Generate a relative output directory path when `useRelativePath` flag is on. This method will compute the relative path based on `inputFile` and `baseInputDir`, then apply only the directory part of relative path to `outputDir`. E.g. baseInputDir = "/path/to/input" inputFile = "/path/to/input/a/b/c/d.avro" outputDir = "/path/to/output" getRelativeOutputPath(baseInputDir, inputFile, outputDir) = /path/to/output/a/b/c

getFileName

public static String getFileName(URI inputFileURI)

Extract file name from a given URI.

Parameters:

inputFileURI -

Returns:

getFileURI

public static URI getFileURI(String uriStr,
                             URI fullUriForPathOnlyUriStr)
                      throws URISyntaxException

Convert a File URI String to URI Object, use parent URI scheme/userInfo/host/port if sheme is not specified.

Parameters:

uriStr -

fullUriForPathOnlyUriStr -

Returns:

Throws:

URISyntaxException

getDirectoryURI

public static URI getDirectoryURI(String uriStr)
                           throws URISyntaxException

Convert Directory URI String to URI Object, default to local file system scheme.

Parameters:

uriStr -

Returns:

Throws:

URISyntaxException

listMatchedFilesWithRecursiveOption

public static List<String> listMatchedFilesWithRecursiveOption(PinotFS pinotFs,
                                                               URI fileUri,
                                                               @Nullable
                                                               String includePattern,
                                                               @Nullable
                                                               String excludePattern,
                                                               boolean searchRecursively)
                                                        throws Exception

Parameters:

pinotFs - root directory fs

fileUri - root directory uri

includePattern - optional glob patterns for files to include

excludePattern - optional glob patterns for files to exclude

searchRecursively - if ture, search files recursively from directory specified in fileUri

Returns:

list of matching files.

Throws:

IOException - on IO failure for list files in root directory.

URISyntaxException - for matching file URIs

RuntimeException - if there is no matching file.

Exception

sanitizeURIString

public static String sanitizeURIString(String path)