Tackling timeout issues when uploading large files with HttpWebRequest

If you ever had to upload large volumes of data over HTTP, you probably ran into timeout issues. The default Timeout value for HttpWebRequest is 100 seconds, which means that if it takes more than that from the time you send the request headers to the time you receive the response headers, your request will fail. Obviously, if you’re uploading a large file, you need to increase that timeout… but to which value?

If you know the available bandwidth, you could calculate a rough estimate of how long it should take to upload the file, but it’s not very reliable, because if there is some network congestion, it will take longer, and your request will fail even though it could have succeeded given enough time. So, should you set the timeout to a very large value, like several hours, or even Timeout.Infinite? Probably not. The most compelling reason is that even though the transfer itself could take hours, some phases of the exchange shouldn’t take that long. Let’s decompose the phases of an HTTP upload:

timeout1

Obtaining the request stream or getting the response (orange parts) isn’t supposed to take very long, so obviously we need a rather short timeout there (the default value of 100 seconds seems reasonable). But sending the request body (blue part) could take much longer, and there is no reliable way  to decide how long that should be; as long as we keep sending data and the server is receiving it, there is no reason not to continue, even if it’s taking hours. So we actually don’t want a timeout at all there! Unfortunately, the behavior of the Timeout property is to consider everything from the call to GetRequestStream to the return of GetResponse

In my opinion, it’s a design flaw of the HttpWebRequest class, and one that has bothered me for a very long time. So I eventually came up with a solution. It relies on the fact that the asynchronous versions of GetRequestStream and GetResponse don’t have a timeout mechanism. Here’s what the documentation says:

The Timeout property has no effect on asynchronous requests made with the BeginGetResponse or BeginGetRequestStream method.

In the case of asynchronous requests, the client application implements its own time-out mechanism. Refer to the example in the BeginGetResponse method.

So, a solution could be to to use these methods directly (or the new Task-based versions: GetRequestStreamAsync and GetResponseAsync); but more often than not, you already have an existing code base that uses the synchronous methods, and changing the code to make it fully asynchronous is usually not trivial. So, the easy approach is to create synchronous wrappers around BeginGetRequestStream and BeginGetResponse, with a way to specify a timeout for these operations:

    public static class WebRequestExtensions
    {
        public static Stream GetRequestStreamWithTimeout(
            this WebRequest request,
            int? millisecondsTimeout = null)
        {
            return AsyncToSyncWithTimeout(
                request.BeginGetRequestStream,
                request.EndGetRequestStream,
                millisecondsTimeout ?? request.Timeout);
        }

        public static WebResponse GetResponseWithTimeout(
            this HttpWebRequest request,
            int? millisecondsTimeout = null)
        {
            return AsyncToSyncWithTimeout(
                request.BeginGetResponse,
                request.EndGetResponse,
                millisecondsTimeout ?? request.Timeout);
        }

        private static T AsyncToSyncWithTimeout<T>(
            Func<AsyncCallback, object, IAsyncResult> begin,
            Func<IAsyncResult, T> end,
            int millisecondsTimeout)
        {
            var iar = begin(null, null);
            if (!iar.AsyncWaitHandle.WaitOne(millisecondsTimeout))
            {
                var ex = new TimeoutException();
                throw new WebException(ex.Message, ex, WebExceptionStatus.Timeout, null);
            }
            return end(iar);
        }
    }

(note that I used the Begin/End methods rather than the Async methods, in order to keep compatibility with older versions of .NET)

These extension methods can be used instead of GetRequestStream and GetResponse; each of them will timeout if they take too long, but once you have the request stream, you can take as long as you want to upload the data. Note that the stream itself has its own read and write timeout (5 minutes by default), so if 5 minutes go by without any data being uploaded, the Write method will cause an exception. Here is the new upload scenario using these methods:

timeout2

As you can see, the only difference is that the timeout doesn’t apply anymore to the transfer of the request body, but only to obtaining the request stream and getting the response. Here’s a full example that corresponds to the scenario above:

long UploadFile(string path, string url, string contentType)
{
    // Build request
    var request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = WebRequestMethods.Http.Post;
    request.AllowWriteStreamBuffering = false;
    request.ContentType = contentType;
    string fileName = Path.GetFileName(path);
    request.Headers["Content-Disposition"] = string.Format("attachment; filename=\"{0}\"", fileName);
    
    try
    {
        // Open source file
        using (var fileStream = File.OpenRead(path))
        {
            // Set content length based on source file length
            request.ContentLength = fileStream.Length;
            
            // Get the request stream with the default timeout
            using (var requestStream = request.GetRequestStreamWithTimeout())
            {
                // Upload the file with no timeout
                fileStream.CopyTo(requestStream);
            }
        }
        
        // Get response with the default timeout, and parse the response body
        using (var response = request.GetResponseWithTimeout())
        using (var responseStream = response.GetResponseStream())
        using (var reader = new StreamReader(responseStream))
        {
            string json = reader.ReadToEnd();
            var j = JObject.Parse(json);
            return j.Value<long>("Id");
        }
    }
    catch (WebException ex)
    {
        if (ex.Status == WebExceptionStatus.Timeout)
        {
            LogError(ex, "Timeout while uploading '{0}'", fileName);
        }
        else
        {
            LogError(ex, "Error while uploading '{0}'", fileName);
        }
        throw;
    }
}

I hope you will find this helpful!

Comments

  1. I want to POST large files to a server using HttpClient. Httpclient has a default timeout of 100 seconds. I do not want to set timeout to infinite. My httpclient should not timeout under normal circumstances and if the httpclient is not able to write data to the request stream it should timeout after say 4 minutes. I want to accomplish exactly the same thing as what you have done with HttpClient. Any ideas on how accomplish this ?
    Also any idea on why HttpClient timesout even if I do an asynchronous request like HttpClient.SendAsync() ?

    1. Thomas Levesque

      Hi PR,

      I don’t know how to solve the issue for HttpClient. IIRC, HttpClient.SendAsync uses the BeginGetRequestStream/BeginGetResponse methods internally, so it probably implements its own timeout mechanism, and I don’t think there’s an easy way to disable it…

      1. Hi Thomas,
        Thanks for the response. I’m trying to implement a timeout mechanism for my HTTPWebrequest and HttpClient. Towards that I have another question.

        In your article you state :
        “you can take as long as you want to upload the data. Note that the stream itself has its own read and write timeout (5 minutes by default), so if 5 minutes go by without any data being uploaded, the Write method will cause an exception”

        MSDN documentation for read/write timeout states
        “Gets or sets a time-out in milliseconds when writing to or reading from a stream. The number of milliseconds before the writing or reading times out.”

        If I understand MSDN documentation correctly, your ENTIRE read/write operation must be completed within 5 minutes or else you will time out.
        In your article you state “so if 5 minutes go by WITHOUT any data being uploaded” you will time out.
        Is it not contradictory or have I misunderstood something ?

        Thanks

        1. Thomas Levesque

          Actually the description on MSDN is a little ambiguous… From what I have observed, it works the way I described it. Or perhaps it’s the time before a call to Write times out… (when uploading a large volume of data, you typically do many calls to Write, not just one)

          1. Yeah, Microsofts documentation is misleading I think.
            The ReadWriteTimeout actually only times out if there is no data written/read from the stream for that time.
            So I think it is also a good Idea to decrease that timeout since 5 minutes seems too large.

  2. Steven Wilber

    Just to say thank you. I’ve been struggling with a GET request provided by a supplier that takes more than 100 seconds to return. I’ve not been able to set the WebRequest timeout to more than 100 seconds, but finally this technique worked. Thank you.

  3. When I try to download a file, GetResponseStream() times out after the default timeout for httpwebrequest. Any idea on how to fix it ?

    1. Thomas Levesque

      Sorry for the late reply, for some reason I didn’t get the email notification…

      Is it the call to GetResponseStream that fails, or reading the response body once you have the stream? From what I observed, the timeout doesn’t occur while downloading.

  4. Great Article Indeed!
    When I tried uploading 6GB file I got error
    “Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.”

    Detailed error:
    Exception Type: “SocketException”
    Message: “An existing connection was forcibly closed by the remote host”

    Exception Type: “IOException”
    Message: “Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.”

    As you mentioned stream has timeout, I’ve set to
    I’ve following settings,
    request.WriteTimeout=30000000
    request.ReadTimeout=30000000
    request.GetRequestStreamWithTimeout(30000000)

    Error occured exactly at
    fileStream.CopyTo(requestStream);

    1. Thomas Levesque

      Hi Enrique,

      The solution I explain in this post only addresses timeout issues, not other kinds of issues that can occur during a long transfer. In your case, it looks like the server actively closed the connection, so it’s not a timeout.

  5. Cont. to above comment.

    To be precise the method name that error occured was MultipleWrite()
    Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.

    And at
    MultipleSend()
    An existing connection was forcibly closed by the remote host

    Looks like these MultipleWrite/send methods are CopyTo Internal methods.

    1. Thomas Levesque

      My previous comment still stands, your error has nothing to do with a timeout (or perhaps the server itself has a timeout for completing the request, but nothing you do from the client side can change that)

  6. First I tried it as described here, without timeout during writing to the stream. That didn’t work, i.e. it timed out.

    So I read the MSDN docs and understood it differently as what is described above. Most especially: one cannot write to the stream without a timeout because there is a default of 5 minutes (ReadWriteTimeout) which, contrary to above, is the timeout for actual reading/writing, and not the waiting of the start of transmission.

    This understanding is reinforced by the accepted answer here: http://stackoverflow.com/questions/7250983/httpwebrequests-timeout-and-readwritetimeout-what-do-these-mean-for-the-unde

    Given this, I modified my code so that I set ReadWriteTimeout property. That still did not work, because the long ReadWriteTimeout I set for writing to the stream was not honored because GetResponseWithTimeout() timed out first, somehow aborting the writing to the stream.

    I described this in better detail here: http://stackoverflow.com/questions/35264680/httpwebrequest-timeout-for-large-files for which I have to solution to yet 🙁

    Any thoughts on this?

    1. Thomas Levesque

      Hi Pao,

      I didn’t mention the ReadWriteTimeout, because it’s related to a different issue. Basically, if you have ReadWriteTimeout of 5 minutes, it means that it will timeout if you’re unable to read or write anything for 5 minutes. If that happens, it probably means the connection is lost. GetResponseWithTimeout does have a timeout (as the name implies), but at this point ReadWriteTimeout doesn’t come into play. ReadWriteTimeout is used to set the ReadTimeout/WriteTimeout on the request and response streams, so it only makes sense once you have a stream, i.e. when you’re uploading the request body or downloading the response body.

  7. Thanks for the great article. Really helped me understand some timeout issues. However I got one problem with your approach.

    In order to get it working, the HttpWebRequest.AllowWriteStreamBuffering = false is mandatory. Otherwise the upload will take place together with the GetResponce-call.
    That’s not an issue, but in order to set AllowWriteStreamBuffering = false, I also need to set HttpWebRequest.SendChunked = true. Thats required, otherwise I get an exception.But when I do that, uploads are much slower then before. So far I have no Idea why that is. Any ideas?

    1. Thomas Levesque

      Hi Daniel,

      Good point about AllowWriteStreamBuffering. Although if you’re sending a very large file without disabling buffering, you’re going to run into memory issues anyway…

      Regarding SendChunked, it’s only required if you don’t specify the ContentLength.
      (scratch that, I was confusing with something else.)

    2. Thomas Levesque

      What is the exception you’re getting if you don’t send SendChunked to true? I have used AllowWriteStreamBuffering=false without setting SendChunked to true for a long time, and it worked fine. Perhaps it’s a server-side problem?

  8. I’m getting a System.Net.ProtocolViolationException:
    When performing a write operation with AllowWriteStreamBuffering set to false, you must either set ContentLength to a non-negative number or set SendChunked to true.

    However, I just figured out that the default Buffersize of Streamwriter (which I’m using) is 4kb. That was killing me. Increasing it to 40kb solved the problem 🙂

    But just found out another strange thing. When setting keepAlive = false together with AllowWriteStreamBuffering = false I don’t get the system.Net.ProtocolViolationException. In that case I get System.NotSupportedException: The stream does not support concurrent IO read or write operations.

    When I just also add SendChunked= true, everything works. I’m really confused now 🙂

    1. Thomas Levesque

      Why use a StreamWriter at all? It’s only useful if you’re sending text, and typically if you send text directly it’s not going to be very large. If you’re sending a file, use Stream directly.

      1. For some reason I need to serialize a .net Object to a Json and send it to the server. In order to do so, I’m using a Json Serializer which writes to the stream by the streamwriter.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>