Package com.microsoft.azure.kusto.ingest

Class ManagedStreamingIngestClient

java.lang.Object

com.microsoft.azure.kusto.ingest.IngestClientBase

com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient

All Implemented Interfaces:

IngestClient, QueuedIngestClient, Closeable, AutoCloseable

public class ManagedStreamingIngestClient
extends IngestClientBase
implements QueuedIngestClient

ManagedStreamingIngestClient

This class combines a managed streaming client with a queued streaming client, to create an optimized experience. Since the streaming client communicates directly with the engine, it's more prone to failure, so this class holds both a streaming client and a queued client. It tries 3 times using the streaming client, after which it falls back to the queued streaming client in case of failure. If the size of the stream is bigger than 4194304, it will fall back to the queued streaming client.

Field Summary

Fields

Modifier and Type	Field	Description
static int	ATTEMPT_COUNT
static String	CLASS_NAME
static int	MAX_STREAMING_SIZE_BYTES

Constructor Summary

Constructors

Constructor	Description
ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder, @Nullable HttpClientProperties properties)
ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder, @Nullable HttpClientProperties properties, boolean autoCorrectEndpoint)
ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder, @Nullable org.apache.http.impl.client.CloseableHttpClient httpClient)
ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder, @Nullable org.apache.http.impl.client.CloseableHttpClient httpClient, boolean autoCorrectEndpoint)
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder)	Deprecated. - This method is slated to be private.
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder, boolean autoCorrectEndpoint)
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder, @Nullable HttpClientProperties properties)
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder, @Nullable HttpClientProperties properties, boolean autoCorrectEndpoint)	Deprecated. - This method is slated to be private.
ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager, AzureStorageClient storageClient, StreamingClient streamingClient)	Deprecated. - This method is slated to be private.
ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager, AzureStorageClient storageClient, StreamingClient streamingClient, ExponentialRetry retryTemplate)

Method Summary

Modifier and Type	Method	Description
void	close()
static ManagedStreamingIngestClient	fromDmConnectionString(ConnectionStringBuilder dmConnectionString)	Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead.
static ManagedStreamingIngestClient	fromDmConnectionString(ConnectionStringBuilder dmConnectionString, @Nullable HttpClientProperties properties)	Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead.
static ManagedStreamingIngestClient	fromEngineConnectionString(ConnectionStringBuilder engineConnectionString)	Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead.
static ManagedStreamingIngestClient	fromEngineConnectionString(ConnectionStringBuilder engineConnectionString, @Nullable HttpClientProperties properties)	Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead.
protected String	getClientType()
IngestionResourceManager	getResourceManager()
protected IngestionResult	ingestFromBlobImpl(BlobSourceInfo blobSourceInfo, IngestionProperties ingestionProperties)	Ingest data from a blob storage into Kusto database.
protected IngestionResult	ingestFromFileImpl(FileSourceInfo fileSourceInfo, IngestionProperties ingestionProperties)
protected IngestionResult	ingestFromResultSetImpl(ResultSetSourceInfo resultSetSourceInfo, IngestionProperties ingestionProperties)	Ingest data from a Result Set into Kusto database.
protected IngestionResult	ingestFromStreamImpl(StreamSourceInfo streamSourceInfo, IngestionProperties ingestionProperties)	Ingest data from an input stream, into Kusto database.
void	setQueueRequestOptions(com.azure.storage.common.policy.RequestRetryOptions queueRequestOptions)	Setter for QueueRequestOptions used by the client on adding ingest message to the Azure queue, read here https://docs.microsoft.com/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#ingest-client-flavors about Kusto queued ingestion

Methods inherited from class com.microsoft.azure.kusto.ingest.IngestClientBase

getIngestionTraceAttributes, ingestFromBlob, ingestFromFile, ingestFromResultSet, ingestFromStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.microsoft.azure.kusto.ingest.IngestClient

ingestFromBlob, ingestFromFile, ingestFromResultSet, ingestFromStream

Field Detail

ATTEMPT_COUNT

public static final int ATTEMPT_COUNT

See Also:

Constant Field Values

MAX_STREAMING_SIZE_BYTES

public static final int MAX_STREAMING_SIZE_BYTES

See Also:

Constant Field Values

CLASS_NAME

public static final String CLASS_NAME

Constructor Detail

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder)
                             throws URISyntaxException

Deprecated.

- This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder) instead.

Parameters:

ingestionEndpointConnectionStringBuilder - - Endpoint for ingesting data, usually starts with "https://ingest-"

queryEndpointConnectionStringBuilder - - Endpoint for querying data, does not include "ingest-"

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder,
                                    boolean autoCorrectEndpoint)
                             throws URISyntaxException

Parameters:

ingestionEndpointConnectionStringBuilder - - Endpoint for ingesting data, usually starts with "https://ingest-"

queryEndpointConnectionStringBuilder - - Endpoint for querying data, does not include "ingest-"

autoCorrectEndpoint - - Flag to indicate whether to correct the endpoint URI or not

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder,
                                    @Nullable
                                    @Nullable HttpClientProperties properties,
                                    boolean autoCorrectEndpoint)
                             throws URISyntaxException

Deprecated.

- This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder, HttpClientProperties) instead. This constructor should only be used for advanced cases. If your endpoints are standard, or you do not know, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties))} instead.

Parameters:

ingestionEndpointConnectionStringBuilder - - Endpoint for ingesting data, usually starts with "https://ingest-"

queryEndpointConnectionStringBuilder - - Endpoint for querying data, does not include "ingest-"

properties - - Additional properties to configure the http client

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder,
                                    @Nullable
                                    @Nullable HttpClientProperties properties,
                                    boolean autoCorrectEndpoint)
                             throws URISyntaxException

Parameters:

connectionStringBuilder - - Client connection string

properties - - Additional properties to configure the http client

autoCorrectEndpoint - - Flag to indicate whether to correct the endpoint URI or not

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder,
                                    @Nullable
                                    @Nullable org.apache.http.impl.client.CloseableHttpClient httpClient,
                                    boolean autoCorrectEndpoint)
                             throws URISyntaxException

Parameters:

connectionStringBuilder - - Client connection string

httpClient - - HTTP client

autoCorrectEndpoint - - Flag to indicate whether to correct the endpoint URI or not

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder,
                                    @Nullable
                                    @Nullable HttpClientProperties properties)
                             throws URISyntaxException

Parameters:

ingestionEndpointConnectionStringBuilder - - Endpoint for ingesting data, usually starts with "https://ingest-"

queryEndpointConnectionStringBuilder - - Endpoint for querying data, does not include "ingest-"

properties - - Additional properties to configure the http client

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder,
                                    @Nullable
                                    @Nullable HttpClientProperties properties)
                             throws URISyntaxException

Parameters:

connectionStringBuilder - - Client connection string

properties - - Additional properties to configure the http client

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder,
                                    @Nullable
                                    @Nullable org.apache.http.impl.client.CloseableHttpClient httpClient)
                             throws URISyntaxException

Parameters:

connectionStringBuilder - - Client connection string

httpClient - - HTTP client

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager,
                                    AzureStorageClient storageClient,
                                    StreamingClient streamingClient)

Deprecated.

- This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder) instead.

Parameters:

resourceManager - ingestion resources manager

storageClient - - storage utilities

streamingClient - - the streaming client

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager,
                                    AzureStorageClient storageClient,
                                    StreamingClient streamingClient,
                                    ExponentialRetry retryTemplate)

Parameters:

resourceManager - ingestion resources manager

storageClient - - storage utilities

streamingClient - - the streaming client

retryTemplate - - retry template

Method Detail

fromDmConnectionString

public static ManagedStreamingIngestClient fromDmConnectionString(ConnectionStringBuilder dmConnectionString)
                                                           throws URISyntaxException

Deprecated.

- Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from a DM connection string, with default http client properties. This method infers the engine connection string from the DM connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

dmConnectionString - dm connection string

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

fromDmConnectionString

public static ManagedStreamingIngestClient fromDmConnectionString(ConnectionStringBuilder dmConnectionString,
                                                                  @Nullable
                                                                  @Nullable HttpClientProperties properties)
                                                           throws URISyntaxException

Deprecated.

- Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from a DM connection string. This method infers the engine connection string from the DM connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

dmConnectionString - dm connection string

properties - additional properties to configure the http client

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

fromEngineConnectionString

public static ManagedStreamingIngestClient fromEngineConnectionString(ConnectionStringBuilder engineConnectionString)
                                                               throws URISyntaxException

Deprecated.

- Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from an engine connection string, with default http client properties. This method infers the DM connection string from the engine connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

engineConnectionString - engine connection string

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

fromEngineConnectionString

public static ManagedStreamingIngestClient fromEngineConnectionString(ConnectionStringBuilder engineConnectionString,
                                                                      @Nullable
                                                                      @Nullable HttpClientProperties properties)
                                                               throws URISyntaxException

Deprecated.

- Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from an engine connection string. This method infers the DM connection string from the engine connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

engineConnectionString - engine connection string

properties - additional properties to configure the http client

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

ingestFromFileImpl

protected IngestionResult ingestFromFileImpl(FileSourceInfo fileSourceInfo,
                                             IngestionProperties ingestionProperties)
                                      throws IngestionClientException,
                                             IngestionServiceException

Specified by:

ingestFromFileImpl in class IngestClientBase

Throws:

IngestionClientException

IngestionServiceException

ingestFromBlobImpl

protected IngestionResult ingestFromBlobImpl(BlobSourceInfo blobSourceInfo,
                                             IngestionProperties ingestionProperties)
                                      throws IngestionClientException,
                                             IngestionServiceException

Ingest data from a blob storage into Kusto database.

This method ingests the data from a given blob, described in blobSourceInfo, into Kusto database, according to the properties mentioned in ingestionProperties

Specified by:

ingestFromBlobImpl in class IngestClientBase

Parameters:

blobSourceInfo - The specific SourceInfo to be ingested

ingestionProperties - Settings used to customize the ingestion operation

Returns:

IngestionResult object including the ingestion result

Throws:

IngestionClientException - An exception originating from a client activity

IngestionServiceException - An exception returned from the service

See Also:

BlobSourceInfo, IngestionProperties

ingestFromResultSetImpl

protected IngestionResult ingestFromResultSetImpl(ResultSetSourceInfo resultSetSourceInfo,
                                                  IngestionProperties ingestionProperties)
                                           throws IngestionClientException,
                                                  IngestionServiceException

Description copied from class: IngestClientBase

Ingest data from a Result Set into Kusto database.

This method ingests the data from a given Result Set, described in resultSetSourceInfo, into Kusto database, according to the properties mentioned in ingestionProperties

Ingesting from ResultSet is equivalent to ingesting from a csv stream. The DataFormat should be empty or set to "csv", and the mapping, should it be provided, should be csv mapping.

Specified by:

ingestFromResultSetImpl in class IngestClientBase

Parameters:

resultSetSourceInfo - The specific SourceInfo to be ingested

ingestionProperties - Settings used to customize the ingestion operation

Returns:

IngestionResult object including the ingestion result

Throws:

IngestionClientException - An exception originating from a client activity

IngestionServiceException - An exception returned from the service

See Also:

ResultSetSourceInfo, IngestionProperties

ingestFromStreamImpl

protected IngestionResult ingestFromStreamImpl(StreamSourceInfo streamSourceInfo,
                                               IngestionProperties ingestionProperties)
                                        throws IngestionClientException,
                                               IngestionServiceException

Description copied from class: IngestClientBase

Ingest data from an input stream, into Kusto database.

This method ingests the data from a given input stream, described in streamSourceInfo, into Kusto database, according to the properties mentioned in ingestionProperties

Specified by:

ingestFromStreamImpl in class IngestClientBase

Parameters:

streamSourceInfo - The specific SourceInfo to be ingested

ingestionProperties - Settings used to customize the ingestion operation

Returns:

IngestionResult object including the ingestion result

Throws:

IngestionClientException - An exception originating from a client activity

IngestionServiceException - An exception returned from the service

See Also:

StreamSourceInfo, IngestionProperties

getClientType

protected String getClientType()

Specified by:

getClientType in class IngestClientBase

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException

setQueueRequestOptions

public void setQueueRequestOptions(com.azure.storage.common.policy.RequestRetryOptions queueRequestOptions)

Description copied from interface: QueuedIngestClient

Setter for QueueRequestOptions used by the client on adding ingest message to the Azure queue, read here https://docs.microsoft.com/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#ingest-client-flavors about Kusto queued ingestion

Specified by:

setQueueRequestOptions in interface QueuedIngestClient

Parameters:

queueRequestOptions - - Options to use when creating QueueClient

getResourceManager

public IngestionResourceManager getResourceManager()

Specified by:

getResourceManager in interface QueuedIngestClient