Spring Can not stream Files that are larger than 250 MB to the Azure Blob storage with the WebClient and Flux<DataBuffer>

Affects: \SpringBoot 3.0.6 -> Spring 6.0.8

---

Hi :-)

Describe the bug

I have a Rest API that streams file to a client. That client (WebClient) streams (Flux<DataBuffer> and DataBufferUtils.write) the files directly to the Azure Blobstorage. The Problem is that I can not stream Files that are larger than 250MB large. If i try too, the client stuck for ever. With the same code i can write to the filesystem any size of files. I created an example Project with a README.md to reproduce it. https://github.com/git9999999/azure-blob-large-file-upload-problem

Please note that i had opened a ticket for Microsoft here https://github.com/Azure/azure-sdk-for-java/issues/35477 As we have Premium support from Microsoft, I am in direct contact with the People in charge. Currently they are investigating but have a hard time to locate the Problem.

Please can you have a look that his code and check if there is an issue with it? Maybe I use the DataBufferUtils.write the wrong way.

here: https://github.com/git9999999/azure-blob-large-file-upload-problem/blob/main/downloader/src/main/java/com/azureproblem/blob/controller/AzureBlobBugDownloaderController.java#L132

or here:

    @GetMapping(path = "/trigger-download-to-blob/{fileSizeInMb}")
    public void triggerDownloadToBlob(@PathVariable int fileSizeInMb) {
        log.info("triggerDownload");

        var flux = this.webClient
            .get()
            .uri("/serve-file/" + fileSizeInMb)
            .accept(MediaType.APPLICATION_OCTET_STREAM)
            .exchangeToFlux(clientResponse -> clientResponse.body(BodyExtractors.toDataBuffers()));

        var destination = "TestDownloadToAzureBlobStorage" + System.currentTimeMillis() + ".pdf";

        var blobClientTarget = this.containerClient.getBlobClient(destination);

        try (var outputStream = blobClientTarget.getBlockBlobClient().getBlobOutputStream(this.parallelTransferOptions, null, null, null, null)) {
            DataBufferUtils.write(flux, outputStream)
                .map(DataBufferUtils::release)
                .blockLast(Duration.ofHours(22));
            outputStream.flush();
        } catch (IOException e) {
            throw new IllegalStateException(e);
        }

        log.info("!!!!!!!!!!!!!!!!!!!!!!!!!!!!   end download of {}", destination);
    }

We are really unsure where the Problem is, either in the BlobStorage Client or in the WebFlux/DataBufferUtils.

Exception or Stack Trace

No stracktrace Avalible.

To Reproduce

https://github.com/Azure/azure-sdk-for-java/issues/35477 + README.md in the Project

Comment From: dinisgarcia

Any help here would be more than welcome

Comment From: alidandach

Hello @git9999999 ,

Can you try to change the configuration of WebClient ?

I believe that the reactor Netty has default settings that may limit the maximum payload size and connection timeout.

Also, make sure if the tomcat server can process this large file.

Comment From: sdeleuze

Based on this comment, It looks like switching to the asynchronous Azure client solved that issue. The root issue was not identified, but from my understanding there is no strong evidence there is something to change on Spring. Please comment if your analysis is different.

Comment From: git9999999

@sdeleuze I agree on your analysis. Please close the case.