water.persist

Class PersistS3

java.lang.Object

water.persist.Persist

water.persist.PersistS3

public final class PersistS3
extends water.persist.Persist

Persistence backend for S3

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PersistS3.H2OAWSCredentialsProviderChain Modified version of default credentials provider which includes H2O-specific credentials provider.
static class	PersistS3.H2SO3InputStream

Nested classes/interfaces inherited from class water.persist.Persist

water.persist.Persist.PersistEntry

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	S3_CONNECTION_TIMEOUT_PROP S3 connection timeout property name
static java.lang.String	S3_ENABLE_PATH_STYLE Enable S3 path style access via setting the property to true.
static java.lang.String	S3_END_POINT S3 end-point, for example: "https://localhost:9000
static java.lang.String	S3_FORCE_HTTP S3 force HTTP traffic
static java.lang.String	S3_MAX_ERROR_RETRY_PROP S3 maximal error retry number
static java.lang.String	S3_MAX_HTTP_CONNECTIONS_PROP S3 maximal http connections
static java.lang.String	S3_REGION S3 region, for example "us-east-1", see Region.getRegion(com.amazonaws.regions.Regions) for region list
static java.lang.String	S3_SOCKET_TIMEOUT_PROP S3 socket timeout property name

Constructor Summary

Constructors

Constructor and Description
PersistS3()

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.String>	calcTypeaheadMatches(java.lang.String filter, int limit)
void	cleanUp()
static java.lang.String[]	decodeKey(water.Key k) Decodes the given H2O key to the S3 bucket and key name.
void	delete(water.Value v)
static water.Key	encodeKey(java.lang.String bucket, java.lang.String key) Creates the key for given S3 bucket and key.
static com.amazonaws.services.s3.AmazonS3	getClient()
void	importFiles(java.lang.String path, java.lang.String pattern, java.util.ArrayList<java.lang.String> files, java.util.ArrayList<java.lang.String> keys, java.util.ArrayList<java.lang.String> fails, java.util.ArrayList<java.lang.String> dels)
byte[]	load(water.Value v)
static water.Key	loadKey(com.amazonaws.services.s3.model.ObjectListing listing, com.amazonaws.services.s3.model.S3ObjectSummary obj)
java.io.InputStream	open(java.lang.String path)
static java.io.InputStream	openStream(water.Key k, water.util.RIStream.ProgressMonitor pmon)
void	store(water.Value v)
water.Key	uriToKey(java.net.URI uri)

Methods inherited from class water.persist.Persist

canHandle, create, delete, exists, getHomeDirectory, getParent, getTotalSpace, getUsableSpace, isDirectory, isSeekableOpenSupported, key2Str, length, list, load, mkdirs, openSeekable, rename, wrapSeekable

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

S3_SOCKET_TIMEOUT_PROP

public static final java.lang.String S3_SOCKET_TIMEOUT_PROP

S3 socket timeout property name

See Also:

Constant Field Values

S3_CONNECTION_TIMEOUT_PROP

public static final java.lang.String S3_CONNECTION_TIMEOUT_PROP

S3 connection timeout property name

See Also:

Constant Field Values

S3_MAX_ERROR_RETRY_PROP

public static final java.lang.String S3_MAX_ERROR_RETRY_PROP

S3 maximal error retry number

See Also:

Constant Field Values

S3_MAX_HTTP_CONNECTIONS_PROP

public static final java.lang.String S3_MAX_HTTP_CONNECTIONS_PROP

S3 maximal http connections

See Also:

Constant Field Values

S3_FORCE_HTTP

public static final java.lang.String S3_FORCE_HTTP

S3 force HTTP traffic

See Also:

Constant Field Values

S3_END_POINT

public static final java.lang.String S3_END_POINT

S3 end-point, for example: "https://localhost:9000

See Also:

Constant Field Values

S3_REGION

public static final java.lang.String S3_REGION

S3 region, for example "us-east-1", see Region.getRegion(com.amazonaws.regions.Regions) for region list

See Also:

Constant Field Values

S3_ENABLE_PATH_STYLE

public static final java.lang.String S3_ENABLE_PATH_STYLE

Enable S3 path style access via setting the property to true. See: S3ClientOptions.setPathStyleAccess(boolean)

See Also:

Constant Field Values

Constructor Detail

PersistS3

public PersistS3()

Method Detail

getClient

public static com.amazonaws.services.s3.AmazonS3 getClient()

open

public java.io.InputStream open(java.lang.String path)

Overrides:

open in class water.persist.Persist

openStream

public static java.io.InputStream openStream(water.Key k,
                                             water.util.RIStream.ProgressMonitor pmon)
                                      throws java.io.IOException

Throws:

java.io.IOException

loadKey

public static water.Key loadKey(com.amazonaws.services.s3.model.ObjectListing listing,
                                com.amazonaws.services.s3.model.S3ObjectSummary obj)
                         throws java.io.IOException

Throws:

java.io.IOException

importFiles

public void importFiles(java.lang.String path,
                        java.lang.String pattern,
                        java.util.ArrayList<java.lang.String> files,
                        java.util.ArrayList<java.lang.String> keys,
                        java.util.ArrayList<java.lang.String> fails,
                        java.util.ArrayList<java.lang.String> dels)

Specified by:

importFiles in class water.persist.Persist

load

public byte[] load(water.Value v)

Specified by:

load in class water.persist.Persist

store

public void store(water.Value v)

Specified by:

store in class water.persist.Persist

encodeKey

public static water.Key encodeKey(java.lang.String bucket,
                                  java.lang.String key)

Creates the key for given S3 bucket and key. Returns the H2O key, or null if the key cannot be created.

Parameters:

bucket - Bucket name

key - Key name (S3)

Returns:

H2O key pointing to the given bucket and key.

decodeKey

public static java.lang.String[] decodeKey(water.Key k)

Decodes the given H2O key to the S3 bucket and key name. Returns the array of two strings, first one is the bucket name and second one is the key name.

Parameters:

k - Key to be decoded.

Returns:

Pair (array) of bucket name and key name.

delete

public void delete(water.Value v)

Specified by:

delete in class water.persist.Persist

uriToKey

public water.Key uriToKey(java.net.URI uri)
                   throws java.io.IOException

Specified by:

uriToKey in class water.persist.Persist

Throws:

java.io.IOException

cleanUp

public void cleanUp()

Specified by:

cleanUp in class water.persist.Persist

calcTypeaheadMatches

public java.util.List<java.lang.String> calcTypeaheadMatches(java.lang.String filter,
                                                             int limit)

Specified by:

calcTypeaheadMatches in class water.persist.Persist