Tag Archives: upload

Handling multipart requests with JSON and file uploads in ASP.NET Core

Suppose we’re writing an API for a blog. Our "create post" endpoint should receive the title, body, tags and an image to display at the top of the post. This raises a question: how do we send the image? There are at least 3 options:

  • Embed the image bytes as base64 in the JSON payload, e.g.

    {
        "title": "My first blog post",
        "body": "This is going to be the best blog EVER!!!!",
        "tags": [ "first post", "hello" ],
        "image": "iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg=="
    }
    

    This works fine, but it’s probably not a very good idea to embed an arbitrarily long blob in JSON, because it could use a lot of memory if the image is very large.

  • Send the JSON and image as separate requests. Easy, but what if we want the image to be mandatory? There’s no guarantee that the client will send the image in a second request, so our post object will be in an invalid state.

  • Send the JSON and image as a multipart request.

The last approach seems the most appropriate; unfortunately it’s also the most difficult to support… There is no built-in support for this scenario in ASP.NET Core. There is some support for the multipart/form-data content type, though; for instance, we can bind a model to a multipart request body, like this:

public class MyRequestModel
{
    [Required]
    public string Title { get; set; }
    [Required]
    public string Body { get; set; }
    [Required]
    public IFormFile Image { get; set; }
}

public IActionResult Post([FromForm] MyRequestModel request)
{
    ...
}

But if we do this, it means that each property maps to a different part of the request; we’re completely giving up on JSON.

There’s also a MultipartReader class that we can use to manually decode the request, but it means we have to give up model binding and automatic model validation entirely.

Custom model binder

Ideally, we’d like to have a request model like this:

public class CreatePostRequestModel
{
    [Required]
    public string Title { get; set; }
    [Required]
    public string Body { get; set; }
    public string[] Tags { get; set; }
    [Required]
    public IFormFile Image { get; set; }
}

Where the Title, Body and Tags properties come from JSON and the Image property comes from the uploaded file.

Fortunately, ASP.NET Core is very flexible, and we can actually make this work, by writing a custom model binder.

Here it is:

using System;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.ModelBinding;
using Microsoft.AspNetCore.Mvc.ModelBinding.Binders;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Newtonsoft.Json;

namespace TestMultipart.ModelBinding
{
    public class JsonWithFilesFormDataModelBinder : IModelBinder
    {
        private readonly IOptions<MvcJsonOptions> _jsonOptions;
        private readonly FormFileModelBinder _formFileModelBinder;

        public JsonWithFilesFormDataModelBinder(IOptions<MvcJsonOptions> jsonOptions, ILoggerFactory loggerFactory)
        {
            _jsonOptions = jsonOptions;
            _formFileModelBinder = new FormFileModelBinder(loggerFactory);
        }

        public async Task BindModelAsync(ModelBindingContext bindingContext)
        {
            if (bindingContext == null)
                throw new ArgumentNullException(nameof(bindingContext));

            // Retrieve the form part containing the JSON
            var valueResult = bindingContext.ValueProvider.GetValue(bindingContext.FieldName);
            if (valueResult == ValueProviderResult.None)
            {
                // The JSON was not found
                var message = bindingContext.ModelMetadata.ModelBindingMessageProvider.MissingBindRequiredValueAccessor(bindingContext.FieldName);
                bindingContext.ModelState.TryAddModelError(bindingContext.ModelName, message);
                return;
            }

            var rawValue = valueResult.FirstValue;

            // Deserialize the JSON
            var model = JsonConvert.DeserializeObject(rawValue, bindingContext.ModelType, _jsonOptions.Value.SerializerSettings);

            // Now, bind each of the IFormFile properties from the other form parts
            foreach (var property in bindingContext.ModelMetadata.Properties)
            {
                if (property.ModelType != typeof(IFormFile))
                    continue;

                var fieldName = property.BinderModelName ?? property.PropertyName;
                var modelName = fieldName;
                var propertyModel = property.PropertyGetter(bindingContext.Model);
                ModelBindingResult propertyResult;
                using (bindingContext.EnterNestedScope(property, fieldName, modelName, propertyModel))
                {
                    await _formFileModelBinder.BindModelAsync(bindingContext);
                    propertyResult = bindingContext.Result;
                }

                if (propertyResult.IsModelSet)
                {
                    // The IFormFile was sucessfully bound, assign it to the corresponding property of the model
                    property.PropertySetter(model, propertyResult.Model);
                }
                else if (property.IsBindingRequired)
                {
                    var message = property.ModelBindingMessageProvider.MissingBindRequiredValueAccessor(fieldName);
                    bindingContext.ModelState.TryAddModelError(modelName, message);
                }
            }

            // Set the successfully constructed model as the result of the model binding
            bindingContext.Result = ModelBindingResult.Success(model);
        }
    }
}

To use it, just apply this attribute to the CreatePostRequestModel class above:

[ModelBinder(typeof(JsonWithFilesFormDataModelBinder), Name = "json")]
public class CreatePostRequestModel

This tells ASP.NET Core to use our custom model binder to bind this class. The Name = "json" part tells our binder in which part of the request it should read the JSON (this is the bindingContext.FieldName in the binder code).

Now we just need to pass a CreatePostRequestModel to our controller action, and we’re done:

[HttpPost]
public ActionResult<Post> CreatePost(CreatePostRequestModel post)
{
    ...
}

This approach enables us to have a clean controller code and keep the benefits of model binding and validation. It messes up the Swagger/OpenAPI model though, but hey, you can’t have everything!

Tackling timeout issues when uploading large files with HttpWebRequest

If you ever had to upload large volumes of data over HTTP, you probably ran into timeout issues. The default Timeout value for HttpWebRequest is 100 seconds, which means that if it takes more than that from the time you send the request headers to the time you receive the response headers, your request will fail. Obviously, if you’re uploading a large file, you need to increase that timeout… but to which value?

If you know the available bandwidth, you could calculate a rough estimate of how long it should take to upload the file, but it’s not very reliable, because if there is some network congestion, it will take longer, and your request will fail even though it could have succeeded given enough time. So, should you set the timeout to a very large value, like several hours, or even Timeout.Infinite? Probably not. The most compelling reason is that even though the transfer itself could take hours, some phases of the exchange shouldn’t take that long. Let’s decompose the phases of an HTTP upload:

timeout1

Obtaining the request stream or getting the response (orange parts) isn’t supposed to take very long, so obviously we need a rather short timeout there (the default value of 100 seconds seems reasonable). But sending the request body (blue part) could take much longer, and there is no reliable way  to decide how long that should be; as long as we keep sending data and the server is receiving it, there is no reason not to continue, even if it’s taking hours. So we actually don’t want a timeout at all there! Unfortunately, the behavior of the Timeout property is to consider everything from the call to GetRequestStream to the return of GetResponse

In my opinion, it’s a design flaw of the HttpWebRequest class, and one that has bothered me for a very long time. So I eventually came up with a solution. It relies on the fact that the asynchronous versions of GetRequestStream and GetResponse don’t have a timeout mechanism. Here’s what the documentation says:

The Timeout property has no effect on asynchronous requests made with the BeginGetResponse or BeginGetRequestStream method.

In the case of asynchronous requests, the client application implements its own time-out mechanism. Refer to the example in the BeginGetResponse method.

So, a solution could be to to use these methods directly (or the new Task-based versions: GetRequestStreamAsync and GetResponseAsync); but more often than not, you already have an existing code base that uses the synchronous methods, and changing the code to make it fully asynchronous is usually not trivial. So, the easy approach is to create synchronous wrappers around BeginGetRequestStream and BeginGetResponse, with a way to specify a timeout for these operations:

    public static class WebRequestExtensions
    {
        public static Stream GetRequestStreamWithTimeout(
            this WebRequest request,
            int? millisecondsTimeout = null)
        {
            return AsyncToSyncWithTimeout(
                request.BeginGetRequestStream,
                request.EndGetRequestStream,
                millisecondsTimeout ?? request.Timeout);
        }

        public static WebResponse GetResponseWithTimeout(
            this HttpWebRequest request,
            int? millisecondsTimeout = null)
        {
            return AsyncToSyncWithTimeout(
                request.BeginGetResponse,
                request.EndGetResponse,
                millisecondsTimeout ?? request.Timeout);
        }

        private static T AsyncToSyncWithTimeout<T>(
            Func<AsyncCallback, object, IAsyncResult> begin,
            Func<IAsyncResult, T> end,
            int millisecondsTimeout)
        {
            var iar = begin(null, null);
            if (!iar.AsyncWaitHandle.WaitOne(millisecondsTimeout))
            {
                var ex = new TimeoutException();
                throw new WebException(ex.Message, ex, WebExceptionStatus.Timeout, null);
            }
            return end(iar);
        }
    }

(note that I used the Begin/End methods rather than the Async methods, in order to keep compatibility with older versions of .NET)

These extension methods can be used instead of GetRequestStream and GetResponse; each of them will timeout if they take too long, but once you have the request stream, you can take as long as you want to upload the data. Note that the stream itself has its own read and write timeout (5 minutes by default), so if 5 minutes go by without any data being uploaded, the Write method will cause an exception. Here is the new upload scenario using these methods:

timeout2

As you can see, the only difference is that the timeout doesn’t apply anymore to the transfer of the request body, but only to obtaining the request stream and getting the response. Here’s a full example that corresponds to the scenario above:

long UploadFile(string path, string url, string contentType)
{
    // Build request
    var request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = WebRequestMethods.Http.Post;
    request.AllowWriteStreamBuffering = false;
    request.ContentType = contentType;
    string fileName = Path.GetFileName(path);
    request.Headers["Content-Disposition"] = string.Format("attachment; filename=\"{0}\"", fileName);
    
    try
    {
        // Open source file
        using (var fileStream = File.OpenRead(path))
        {
            // Set content length based on source file length
            request.ContentLength = fileStream.Length;
            
            // Get the request stream with the default timeout
            using (var requestStream = request.GetRequestStreamWithTimeout())
            {
                // Upload the file with no timeout
                fileStream.CopyTo(requestStream);
            }
        }
        
        // Get response with the default timeout, and parse the response body
        using (var response = request.GetResponseWithTimeout())
        using (var responseStream = response.GetResponseStream())
        using (var reader = new StreamReader(responseStream))
        {
            string json = reader.ReadToEnd();
            var j = JObject.Parse(json);
            return j.Value<long>("Id");
        }
    }
    catch (WebException ex)
    {
        if (ex.Status == WebExceptionStatus.Timeout)
        {
            LogError(ex, "Timeout while uploading '{0}'", fileName);
        }
        else
        {
            LogError(ex, "Error while uploading '{0}'", fileName);
        }
        throw;
    }
}

I hope you will find this helpful!