Package org.apache.pinot.common.http

Class MultiHttpRequest

  • public class MultiHttpRequest
extends Object
    Class to support multiple http operations in parallel by using the executor that is passed in. This is a wrapper around Apache common HTTP client. The execute method is re-usable but there is no real benefit to it. All the connection management is handled by the input HttpConnectionManager. Note that we cannot use SimpleHttpConnectionManager as it is not thread safe. Use MultiThreadedHttpConnectionManager as shown in the example below. As GET is commonly used, there is a dedicated execute method for it. Other http methods like DELETE can use the generic version of execute method. Usage: 
     
    List<String> urls = Arrays.asList("http://www.linkedin.com", "http://www.google.com");
    MultiHttpRequest mhr = new MultiHttpRequest(Executors.newCachedThreadPool(),
           new MultiThreadedHttpConnectionManager());
    CompletionService<GetMethod> completionService = mhr.execute(urls, headers, timeoutMs);
    for (int i = 0; i < urls.size(); i++) {
      GetMethod getMethod = null;
      try {
        getMethod = completionService.take().get();
        if (getMethod.getStatusCode() >= 300) {
          System.out.println("error");
          continue;
        }
        System.out.println("Got data: " +  getMethod.getResponseBodyAsString());
      } catch (ExecutionException e) {
         if (Throwables.getRootcause(e) instanceof SocketTimeoutException) {
           System.out.println("Timeout");
         }
      } finally {
        if (getMethod != null) {
          getMethod.releaseConnection();
        }
      }
    }

    • Constructor Detail

      • MultiHttpRequest

        public MultiHttpRequest​(Executor executor,
                        org.apache.commons.httpclient.HttpConnectionManager connectionManager)
        Parameters:
        executor - executor service to use for making parallel requests
        connectionManager - http connection manager to use.

    • Method Detail

      • execute

        public CompletionService<org.apache.commons.httpclient.methods.GetMethod> execute​(List<String> urls,
                                                                                  @Nullable
                                                                                  Map<String,​String> requestHeaders,
                                                                                  int timeoutMs)
        GET urls in parallel using the executor service.
        Parameters:
        urls - absolute URLs to GET
        requestHeaders - headers to set when making the request
        timeoutMs - timeout in milliseconds for each GET request
        Returns:
        instance of CompletionService. Completion service will provide results as they arrive. The order is NOT same as the order of URLs

      • execute

        public <T extends org.apache.commons.httpclient.HttpMethodBase> CompletionService<T> execute​(List<String> urls,
                                                                                             @Nullable
                                                                                             Map<String,​String> requestHeaders,
                                                                                             int timeoutMs,
                                                                                             String httpMethodName,
                                                                                             Function<String,​T> httpMethodSupplier)
        Execute certain http method on the urls in parallel using the executor service.
        Parameters:
        urls - absolute URLs to execute the http method
        requestHeaders - headers to set when making the request
        timeoutMs - timeout in milliseconds for each http request
        httpMethodName - the name of the http method like GET, DELETE etc.
        httpMethodSupplier - a function to create a new http method object.
        Returns:
        instance of CompletionService. Completion service will provide results as they arrive. The order is NOT same as the order of URLs