Issue

HttpRange's public ResourceRegion toResourceRegion(Resource resource) method incorrectly validates Content-length when the remaining amount of ranged-data is less than the starting range value, throwing 'position' exceeds the resource length errors and returning 416 errors in my server's response.

Version

Spring Boot version 2.3.0.RELEASE.

Details

When returning a ResponseEntity<Resource> with a byte range of 199819264-201121204/201121205, the EntityResponseBuilder's call to HttpRange.toResourceRegion() throws an error. Upon inspection, this seems to be caused by the Content-length: 1301941 being 1 byte greater than the getRangeStart() and getRangeEnd() functions. However, the content length is actually correct due to the range start and range end values being inclusive.

For example, using the same Content-range from above:

  • long contentLength = getLengthFor(resource) -> 1301941
  • long start = getRangeStart(contentLength) -> 199819264
  • long end = getRangeEnd(contentLength) -> contentLength - 1 == 1301940
  • Assert.isTrue(start < contentLength) -> error

If rangeEnd - rangeStart = contentLength - 1, then 201121204 - 199819264 = 1301941; 1301941 - 1 = 1301940 due to the rangeEnd being inclusive (i.e. we add 1 to rangeEnd - rangeStart. This means the response Content-range should be correct. However, it's failing just because the getRangeStart() is greater than the contentLength.

Furthermore, start < contentLength should not be the deciding factor to determining if the response is valid; rather start + contentLength-1 <= end should be the deciding factor.

Here is an image of my debugger showing the issue in action:

Spring content-range bug

Comment From: poutsma

Unless I'm missing something, you are getting an exception (and the resulting 416) because the start of the range (199819264) is higher than the actual file length (1301941). The fact that the range requested happens to have the same length as the content-length is not relevant here.

I am not entirely sure what you expect Spring to do in this case, except return a 416. RFC 7233 section 4.4 is quite clear on this (https://tools.ietf.org/html/rfc7233#section-4.4)

Comment From: D-Pow

To be more transparent about this, the core cause of the issue occurs when proxying a Resource from an external service to the client. Essentially, the client is requesting chunks of the resource that is hosted elsewhere, the server is forwarding that request to the external service, and then relaying that back to the client. So the steps go:

  • Client: Requests Range: 199819264-.
  • Server: Requests Range: 199819264- to external service.
  • External service: Returns the resource's bytes with Content-Range: 199819264-201121204/201121205.
  • Server: Forwards the service's bytes in a Resource with Content-Range: 199819264-201121204/201121205 (which is accurate of what data is being returned).
  • Spring's internals: Throws error/returns 416 due to basing its valid/invalid logic on start < contentLength even though this is technically a valid response.

The main point of this bug request is that Content-Length is not the important part of the validation step. The important part is to ensure that start + contentLength-1 <= end. Otherwise, it's impossible to do Range requests if resources aren't hosted locally on the server.

Comment From: poutsma

Otherwise, it's impossible to do Range requests if resources aren't hosted locally on the server.

And that is exactly the problem at hand: Spring MVC's Resource support is not designed to expose remote services. It is designed to represent local files on the file system, where a range request represents a slice of that file. ~~Even our own UrlResource only supports file:// URLs and not http:// for a reason~~ (edit: turns out this is not true, and http is supported, but I would not recommended it). Even if we removed the assertion that causes the exception, you would run into other problems, for instance because of the precautions we added against DoS attacks (see #21851).

You could—theoretically—fix this by creating your own custom Resource implementation that returns the content-length of the remote resource (the length of the entire resource, not just the range). But be very careful that you're not copying all the headers of the the original response, because you will run into security and routing problems.

Personally, I would build up the HTTP response "by hand", by using a HttpServletResponse as parameter, explicitly adding all HTTP headers required, and writing the response to the ServletOutputStream. This way, you could even stream the remote response to the output stream as it comes in, instead of buffering it all in memory and returning as ByteArrayResource.

So I am afraid this scenarios does not have an out-of-the-box solution in Spring yet. We are considering solutions, but I am pretty sure that we will not end up using Resource. Resource does not have the contract to express the asynchrony that is inherent in remote operations.