HttpWebRequest, avoiding the pitfalls

It seems that there are MANY ways to perform http web request poorly. This is a huge problem in today’s world where web-services are more common than bankrupt banks. Here is a quick pattern of how to do it right:

public string Fetch(Uri requestUri)
{

HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(requestUri);

webRequest.Timeout = requestConnectTimeoutInMs; // take timeout from config
webRequest.ReadWriteTimeout = requestReadWriteTimeoutInMs; // take timeout from config

using (WebResponse webResponse = webRequest.GetResponse())
using (StreamReader streamReader = new StreamReader(new TimeoutStream(webResponse.GetResponseStream(), fetchTimeoutInMs)) // take timeout from config
     return streamReader.ReadToEnd();

}

Details:

1. Setting Timeout property: to make sure we don’t wait the default 100 seconds for “ACK” from the server. WAY too much.
2. Setting ReadWriteTimeout: This is crucial to understand. StreamReader under the hood read data in chunks, this timeout determine how much time you should wait for reading a single chunk. 100 seconds, the default value, is again WAY too much.
3. Using TimeoutStream (you need to implement your own or let me know if you’re interested and I’ll send it to you): Alright, let’s say you’re willing to wait for 500ms for ACK (Timeout), up to 500ms for reading every chunk (ReadWriteTimeout) but not more than 5 seconds for the entire read to complete. There is no way to achieve it without TimeoutStream. It will start a timer internally and override Seek/Read/Write (etc) method by checking the timer before calling the internal stream method. TimeoutStream is a very simple wrapper around Stream. For example:

   public override int Read(byte[] buffer, int offset, int count)
      {
          CheckTimeout(); // throw TimeoutException if timeout was reached
          return _stream.Read(buffer, offset, count);
      }

Multiple HttpWebRequest limitation:
By default, you can’t perform more than 2-3 async HttpWebRequest (depends on the OS). In order to override it (the easiest way, IMHO) don’t forget to add this under <configuration> section in the application’s config file:

<system.net>
  <connectionManagement>
     <add address=”*” maxconnection=”65000″ />
  </connectionManagement>
</system.net>

Why should you follow these guidelines:

1. Never trust 3rd party components: avoid excuses like “my site is not responsive because 1000 threads are waiting for web-service-X to respond”. By setting those parameters you’re safe to make your own choices of how much time to wait. Log and monitor these things to adjust your application and alert your suppliers.
2. Be able to determine your own SLA for the world: again, if internally you need to call a web-service, make sure you’re able to control the time you’re willing to spend. You’ve got clients to serve and they want you to meet the SLA as you promised!

Important note about recycling HttpWebRequest.GetResposne()
Simply put, it’s not working by design. That means that if you fail to get a response on time (due to 1,2 or 3), don’t call the webRequest.GetResponse() again as it is cached internally (you’ll get the same HttpWebResposne). What you should do is to re-create the HttpWebRequest and try again. I don’t agree with the selected design by Microsoft for this method, but at least it’s good to be aware of it.

   from MSDN:

   ” Multiple calls to GetResponse return the same response object; the request is not reissued. ”

Final note:
You should obviously consider writing a HttpWebRequestHelper class (or extension method) and use it instead of copy&paste this code all over your codebase.