Bug description Exception is thrown when attempting to send a request to OpenAI transcription service. Please note that the same logic was working approximately 1-2 months ago.
Environment Kotlin + Java 21 Spring Boot 3.3.1 Spring AI 1.0.0-M1 (Also tried on 1.0.0-SNAPSHOT)
Steps to reproduce
Simply execute .call method from OpenAiAudioTranscriptionModel with a ByteArrayResource set in the Prompt.
Here's my method:
fun transcribeAudio(audio: ByteArray): String {
val resource = ByteArrayResource(audio)
val options = OpenAiAudioTranscriptionOptions.builder()
.withResponseFormat(OpenAiAudioApi.TranscriptResponseFormat.TEXT)
.build()
val prompt = AudioTranscriptionPrompt(resource, options)
val response = client.call(prompt)
val output = response.result.output
log.info { "[Transcription] Result - ${response.result}" }
return output
}
Expected behavior No exception. Executed method returns transcribed audio as text.
Some of the things that I tried to fix the issue
- Removed all custom beans that could affect the serialization - ObjectMapper and MappingJackson2HttpMessageConverter
- Custom ObjectMapper bean with disabled SerializationFeature.FAIL_ON_EMPTY_BEANS
@Bean
fun objectMapper(): ObjectMapper {
return ObjectMapper().apply {
disable(SerializationFeature.FAIL_ON_EMPTY_BEANS);
}
}
- Custom MappingJackson2HttpMessageConverter bean
@Bean
fun mappingJackson2HttpMessageConverter(objectMapper: ObjectMapper): MappingJackson2HttpMessageConverter {
objectMapper.configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false)
val converter = MappingJackson2HttpMessageConverter(objectMapper)
return converter
}
Stack trace
org.springframework.http.converter.HttpMessageConversionException: Type definition error: [simple type, class java.io.ByteArrayInputStream]
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:489)
at org.springframework.http.converter.AbstractGenericHttpMessageConverter$1.writeTo(AbstractGenericHttpMessageConverter.java:94)
at org.springframework.http.client.HttpComponentsClientHttpRequest$BodyEntity.writeTo(HttpComponentsClientHttpRequest.java:155)
at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:253)
at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:141)
at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:717)
at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
at org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:116)
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:188)
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116)
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183)
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:99)
at org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchangeInternal(DefaultRestClient.java:492)
at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.retrieve(DefaultRestClient.java:460)
at org.springframework.ai.openai.api.OpenAiAudioApi.createTranscription(OpenAiAudioApi.java:670)
at org.springframework.ai.openai.OpenAiAudioTranscriptionModel.lambda$call$0(OpenAiAudioTranscriptionModel.java:153)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:344)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:217)
at org.springframework.ai.openai.OpenAiAudioTranscriptionModel.call(OpenAiAudioTranscriptionModel.java:124)
at io.github.clivelewis.assistant.chat.impl.openai.OpenAiTranscriptionService.transcribeAudio(OpenAiTranscriptionService.kt:24)
at io.github.clivelewis.assistant.telegram.commands.ConversationCommand.transcribeVoiceMessage(ConversationCommand.kt:187)
at io.github.clivelewis.assistant.telegram.commands.ConversationCommand.access$transcribeVoiceMessage(ConversationCommand.kt:28)
at io.github.clivelewis.assistant.telegram.commands.ConversationCommand$transcribeVoiceMessage$1.invokeSuspend(ConversationCommand.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:811)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:715)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:702)
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class java.io.ByteArrayInputStream and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: org.springframework.util.LinkedMultiValueMap["file"]->java.util.ArrayList[0]->org.springframework.ai.openai.api.OpenAiAudioApi$1["inputStream"])
at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)
at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1330)
at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:414)
at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.failForEmpty(UnknownSerializer.java:53)
at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.serialize(UnknownSerializer.java:30)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:732)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:770)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:183)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:502)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:422)
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1570)
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1061)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:483)
... 45 common frames omitted
Comment From: clivelewis
I cloned the project, checked out to tags/v1.0.0-M1 and ran tests from OpenAiTranscriptionModelIT.java
All tests succeeded, so it's issue on my side. Either because of Kotlin, or because of my code...
Comment From: clivelewis
I played around with different custom Bean definitions and somehow defining OpenAiAudioTranscriptionModel solved the issue for me...
@Bean
fun openAiAudioApi(@Value("\${spring.ai.openai.api-key}") apiKey: String) = OpenAiAudioApi(apiKey)
@Bean
fun openAiAudioTranscriptionModel(openAiAudioApi: OpenAiAudioApi) = OpenAiAudioTranscriptionModel(openAiAudioApi)
I'm closing the issue even though I have no idea why this helped. Will appreciate if somehow could explain.