org.apache.hadoop.fs.azurebfs.services

Class AbfsClientThrottlingIntercept

java.lang.Object

org.apache.hadoop.fs.azurebfs.services.AbfsClientThrottlingIntercept

All Implemented Interfaces:

AbfsThrottlingIntercept

public final class AbfsClientThrottlingIntercept
extends Object
implements AbfsThrottlingIntercept

Throttles Azure Blob File System read and write operations to achieve maximum throughput by minimizing errors. The errors occur when the account ingress or egress limits are exceeded and the server-side throttles requests. Server-side throttling causes the retry policy to be used, but the retry policy sleeps for long periods of time causing the total ingress or egress throughput to be as much as 35% lower than optimal. The retry policy is also after the fact, in that it applies after a request fails. On the other hand, the client-side throttling implemented here happens before requests are made and sleeps just enough to minimize errors, allowing optimal ingress and/or egress throughput.

Constructor Summary

Constructors

Constructor and Description
AbfsClientThrottlingIntercept(String accountName, AbfsConfiguration abfsConfiguration)

Method Summary

Modifier and Type	Method and Description
void	sendingRequest(AbfsRestOperationType operationType, AbfsCounters abfsCounters) Called before the request is sent.
void	updateMetrics(AbfsRestOperationType operationType, AbfsHttpOperation abfsHttpOperation) Updates the metrics for successful and failed read and write operations.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbfsClientThrottlingIntercept

public AbfsClientThrottlingIntercept(String accountName,
                                     AbfsConfiguration abfsConfiguration)

Method Detail

updateMetrics

public void updateMetrics(AbfsRestOperationType operationType,
                          AbfsHttpOperation abfsHttpOperation)

Updates the metrics for successful and failed read and write operations.

Specified by:

updateMetrics in interface AbfsThrottlingIntercept

Parameters:

operationType - Only applicable for read and write operations.

abfsHttpOperation - Used for status code and data transferred.

sendingRequest

public void sendingRequest(AbfsRestOperationType operationType,
                           AbfsCounters abfsCounters)

Called before the request is sent. Client-side throttling uses this to suspend the request, if necessary, to minimize errors and maximize throughput.

Specified by:

sendingRequest in interface AbfsThrottlingIntercept

Parameters:

operationType - Only applicable for read and write operations.

abfsCounters - Used for counters.