Affects: springframework at 6.0.13 is okay but update to 6.1.1 then error
springframework 6.0.13(boot 3.1.5) is okay but update to 6.1.1(boot 3.2.0) then error
I have been performing file uploads to s3 using the content of FilePart. It was working well, but after updating the Spring version, I started getting NoSuchFileException. It occurs correctly about 1 in 20 attempts. Therefore, the majority of the requests fail to upload the file.
The error message is as follows:
java.nio.file.NoSuchFileException: /var/folders/d_/d03cqx_x01n1jy_pgqdyklsw0000gn/T/spring-multipart-7499578255272102614/11133527665414509817.multipart
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
at org.springframework.http.codec.multipart.DefaultParts$FileContent.lambda$content$0(DefaultParts.java:295)
at reactor.core.publisher.FluxUsing.subscribe(FluxUsing.java:75)
and this is my code(kotlin)
private class UploadStates(var bucket: String, var fileKey: String) {
var uploadId: String? = null
var partCounter = 0
var completedParts: MutableMap<Int, CompletedPart> = HashMap()
var buffered = 0
}
fun saveFile(fileKey: String, s3AsyncClient: S3AsyncClient, part: FilePart): Mono<String> {
val uploadState = UploadStates("aa", fileKey)
return Mono
.fromFuture(
s3AsyncClient
.createMultipartUpload(
CreateMultipartUploadRequest.builder()
.contentType((part.headers().contentType ?: MediaType.APPLICATION_OCTET_STREAM).toString())
.key(fileKey)
.bucket("aa")
.build()
)
)
.flatMapMany { response: CreateMultipartUploadResponse ->
checkResult(response)
uploadState.uploadId = response.uploadId()
part.content()
}
.bufferUntil { buffer: DataBuffer ->
uploadState.buffered += buffer.readableByteCount()
return@bufferUntil if (uploadState.buffered >= S3_BUFFER_SIZE) {
uploadState.buffered = 0
true
} else {
false
}
}
.map { buffers: List<DataBuffer> ->
concatBuffers(buffers)
}
.flatMap { buffer: ByteBuffer ->
uploadPart(s3AsyncClient, uploadState, buffer)
}
.reduce(
uploadState
) { state: UploadStates, completedPart: CompletedPart ->
state.completedParts[completedPart.partNumber()] = completedPart
state
}
.flatMap { state: UploadStates ->
completeUpload(s3AsyncClient, state)
}
.map { response: SdkResponse ->
checkResults(response)
uploadState.fileKey
}
.onErrorMap { e ->
throw RuntimeException(e.localizedMessage)
}
}
private fun uploadPart(s3AsyncClient: S3AsyncClient, uploadState: UploadStates, buffer: ByteBuffer): Mono<CompletedPart> {
val partNumber: Int = ++uploadState.partCounter
val request: CompletableFuture<UploadPartResponse> = s3AsyncClient.uploadPart(
UploadPartRequest.builder()
.bucket(uploadState.bucket)
.key(uploadState.fileKey)
.partNumber(partNumber)
.uploadId(uploadState.uploadId)
.contentLength(buffer.capacity().toLong())
.build(),
AsyncRequestBody.fromPublisher(Mono.just(buffer))
)
return Mono
.fromFuture(request)
.map { uploadPartResult ->
checkResults(uploadPartResult)
CompletedPart.builder()
.eTag(uploadPartResult.eTag())
.partNumber(partNumber)
.build()
}
}
private fun checkResults(result: SdkResponse) {
if (result.sdkHttpResponse() == null || !result.sdkHttpResponse().isSuccessful) {
throw UploadFailedException(response = result)
}
}
private fun completeUpload(s3AsyncClient: S3AsyncClient, state: UploadStates): Mono<CompleteMultipartUploadResponse> {
val multipartUpload: CompletedMultipartUpload = CompletedMultipartUpload.builder()
.parts(state.completedParts.values)
.build()
return Mono.fromFuture(
s3AsyncClient.completeMultipartUpload(
CompleteMultipartUploadRequest.builder()
.bucket(state.bucket)
.uploadId(state.uploadId)
.multipartUpload(multipartUpload)
.key(state.fileKey)
.build()
)
)
}
Firstly, I downgraded the version, so the problem is currently resolved, but I'm reaching out because I'm unable to upgrade the version.
Comment From: poutsma
It's hard to determine what's going on without having a complete picture. If you'd like us to spend some time investigating, please take the time to provide a complete minimal sample (something that we can unzip or git clone, build, and deploy) that reproduces the problem.
Comment From: fclemonschool
okay, this is sample but need aws error.zip
and postman json New Collection.postman_collection.json @poutsma
Comment From: poutsma
Due to https://github.com/spring-projects/spring-framework/issues/31567, we now clean up temp files after the HTTP exchange has been completed. What you are seeing is the result of that, meaning that you are attempting to use a FilePart when the temp file backing it has already been deleted.
Looking at the sample, it is not surprising that the files are missing. In FileUtil.kt, line 24, the Mono<String> result from the saveFile function is subscribed to. In a reactive environment, such as a WebFlux application, you never want to subscribe to a stream yourself, you typically propagate the stream to the higher layer. In the sample, this would mean returning Mono<String> in FilePart.upload and FileService, and Mono<ResponseEntity<String>> in TestController.
There could be other issues in the sample, I only took a brief glance. At any rate, this is expected behavior given the sample code.