io.delta.storage.internal

Class S3LogStoreUtil

Object

io.delta.storage.internal.S3LogStoreUtil

public final class S3LogStoreUtil
extends Object

Static utility methods for the S3SingleDriverLogStore. Used to trick the class loader so we can use methods of org.apache.hadoop:hadoop-aws without needing to load this as a dependency for tests in core.

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.fs.FileStatus[]	s3ListFromArray(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path resolvedPath, org.apache.hadoop.fs.Path parentPath) Uses the S3ListRequest.v2 interface with the startAfter parameter to only list files which are lexicographically greater than resolvedPath.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

s3ListFromArray

public static org.apache.hadoop.fs.FileStatus[] s3ListFromArray(org.apache.hadoop.fs.FileSystem fs,
                                                                org.apache.hadoop.fs.Path resolvedPath,
                                                                org.apache.hadoop.fs.Path parentPath)
                                                         throws java.io.IOException

Uses the S3ListRequest.v2 interface with the startAfter parameter to only list files which are lexicographically greater than resolvedPath. Wraps s3ListFrom in an array. Contained in this class to avoid contaminating other classes with dependencies on recent Hadoop versions. TODO: Remove this method when iterators are used everywhere.

Throws:

java.io.IOException