Speed up REST with HTTP Pipelining

Lots of apps download data over HTTP. It's what we use for the Issues Browser demo, and our up-and-coming Amazon Browser demo. In the simple web service paradigm REST, HTTP is the transport layer. HTTP is not just for web browsers!

If you're writing an app that uses HTTP, you may have some code where you make multiple requests to the same server. Some examples:

grabbing 10 images from Flickr's web service

posting 2 events to Google Calendar

updating a dozen business objects in your Amazon S3 shared storage

Multiple requests can be made in multiple ways:

One connection for each request, maing a new connection when the previous connection completes

One connection for each request, making all connections in parallel

One connection for all requests, sending a new request when the previous response completes

One connection for all requests, sending all requests upfront

Option 4 is called "pipelining" and is quite possibly the fastest way to do batch HTTP requests. Unfortunately, support for HTTP pipelining is desperately lacking among Java HTTP client libraries! According to the Oakland Software comparison:

J2SE 1.6 No.

Jakarta Commons Nope

Oakland Software Nopers!

innovation.ch Yes!

Although the innovation.ch HTTPClient is a bit stale (no update since 2001), it implements HTTP pipelining properly. It's also LGPL, so if you find yourself wanting to freshen things up a bit, you can. Here's the basics to pipeline three files:

  HTTPConnection httpConnection = new HTTPConnection("swank.ca");
  HTTPResponse indexResponse = httpConnection.Get("/index.html");
  HTTPResponse lensesResponse = httpConnection.Get("/slacker/notopaque/lenses.png");
  HTTPResponse nintendodsResponse = httpConnection.Get("/slacker/notopaque/nintendods.png");

  InputStream indexStream = indexResponse.getInputStream();
  // code that reads indexStream ...
  InputStream lensesStream = lensesResponse.getInputStream();
  // code that reads lensesStream ...
  InputStream nintendoDsStream = nintendoDsResponse.getInputStream();
  // code that reads nintendoDsStream ...

I strongly recommend you try adding HTTP pipelining to your app - you just might get the speed boost you're looking for!