Bug description When making a basic synchronous openAiChatClient.call( prompt) with a single UserMessage, an Exception is thrown (com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1]))
I have inspected the traffic using Charles, and the OpenAI API is returning a correct JSON response.
Environment Please provide as many details as possible: Spring AI version, Java version, which vector store you use if any, etc OpenJDK 18.0.2 Spring Framework 6.1.4 Spring Boot 3.2.3 Spring Open AI 0.8.0-SNAPSHOT
Steps to reproduce Basic synchronous call
String message = "Tell me about OpenAI";
UserMessage userMessage = new UserMessage( message );
List<Message> openAiMessages = List.of( userMessage );
Prompt prompt = new Prompt( openAiMessages );
ChatResponse call = openAiChatClient.call( prompt );
Generation generation = call.getResults().get( 0 );
System.out.println( generation.getOutput().getContent() );
Expected behavior Chat response is printed.
Stack Trace
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406) ~[spring-web-6.1.4.jar:6.1.4]
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:354) ~[spring-web-6.1.4.jar:6.1.4]
at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:213) ~[spring-web-6.1.4.jar:6.1.4]
... 82 common frames omitted
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 2]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:697) ~[jackson-core-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:512) ~[jackson-core-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:529) ~[jackson-core-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:3103) ~[jackson-core-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:757) ~[jackson-core-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:181) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105) ~[jackson-databind-2.15.4.jar:2.15.4]
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481) ~[jackson-databind-2.15.4.jar:2.15.4]
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:395) ~[spring-web-6.1.4.jar:6.1.4]
... 84 common frames omitted
Comment From: dolukhanov
I narrowed down the problem to the project having a dependencies on:
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5</artifactId>
<version>5.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5-fluent</artifactId>
<version>5.3.1</version>
</dependency>
Once these are removed, the code functions as expected.
Leaving this issue open, should it require further investigation.
Comment From: markpollack
I am not able to reproduce. Here is a working sample of your code. myai.zip
Are you using any ChatOptions configuration in your code? That is an area in the code path where we use Jackson, but if no special options are set, then jackson isn't involved in the code path. Maybe provide a longer stacktrace to localize it better.
Comment From: carlspring
I am experiencing the same issue.
org.springframework.ai:spring-ai-openai-spring-boot-starter:0.8.0- OpenJDK:
21.0.2 - Spring Boot:
3,2,3 - Spring Framework:
6.1.4 - Jackson:
2.17.1
package com.carlspring.ai.openai;
import com.carlspring.ai.openai.config.OpenAITestConfiguration;
import org.junit.jupiter.api.Test;
import org.springframework.ai.autoconfigure.openai.OpenAiAutoConfiguration;
import org.springframework.ai.chat.ChatResponse;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.openai.OpenAiChatClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
/**
* @author carlspring
*/
@SpringBootTest(classes = { OpenAITest.class,
OpenAiAutoConfiguration.class,
OpenAITestConfiguration.class })
public class OpenAITest
{
@Autowired
private OpenAiChatClient chatClient;
@Test
public void testBasic()
{
// Sync request
ChatResponse response = chatClient.call(new Prompt("Is Linux the same as Unix?"));
System.out.println(String.join(" ", response.getResults().stream().map((g) -> g.getOutput().toString()).toList()));
}
}
package com.carlspring.ai.openai.config;
import org.springframework.context.annotation.Bean;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
@Component
public class OpenAITestConfiguration
{
@Bean
public RestClient.Builder restClientBuilder()
{
return RestClient.builder();
}
}
04:50:23.924 13-05-2024 | DEBUG | main | org.springframework.web.client.DefaultRestClient | Writing [ChatCompletionRequest[messages=[ChatCompletionMessage[content=Is Linux the same as Unix?, role=USER, name=null, toolCallId=null, toolCalls=null]], model=gpt-3.5-turbo, frequencyPenalty=0.0, logitBias=null, maxTokens=null, n=1, presencePenalty=null, responseFormat=null, seed=null, stop=null, stream=false, temperature=0.7, topP=null, tools=null, toolChoice=null, user=null]] as "application/json" with org.springframework.http.converter.json.MappingJackson2HttpMessageConverter
04:50:26.018 13-05-2024 | DEBUG | main | org.springframework.web.client.DefaultRestClient | Reading to [org.springframework.ai.openai.api.OpenAiApi$ChatCompletion]
org.springframework.web.client.RestClientException: Error while extracting response for type [org.springframework.ai.openai.api.OpenAiApi$ChatCompletion] and content type [application/json]
at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:236)
at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.readBody(DefaultRestClient.java:667)
at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.toEntityInternal(DefaultRestClient.java:637)
at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.toEntity(DefaultRestClient.java:626)
at org.springframework.ai.openai.api.OpenAiApi.chatCompletionEntity(OpenAiApi.java:656)
at org.springframework.ai.openai.OpenAiChatClient.chatCompletionWithTools(OpenAiChatClient.java:337)
at org.springframework.ai.openai.OpenAiChatClient.lambda$call$1(OpenAiChatClient.java:151)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:335)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:211)
at org.springframework.ai.openai.OpenAiChatClient.call(OpenAiChatClient.java:147)
at com.carlspring.ai.openai.OpenAITest.testBasic(OpenAITest.java:30)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:354)
at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:213)
... 13 more
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 2]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:697)
at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:512)
at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:529)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:3103)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:757)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:181)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105)
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:395)
... 15 more
It seems that the answers here might be related and that the JSON reponse needs to be read fully before trying to read it as an object.
Comment From: vwiencek
I have the same problem and can't figure out how to solve it
Comment From: jocax
I had the same issue and found a workaround by setting the required http header "Accept-Encoding". I applied the header in the Rest.Builder. One the header is set the response can be read without the above error.
@Bean
RestClient.Builder restClientBuilder() {
return RestClient.builder()
.defaultHeaders(new Consumer<HttpHeaders>() {
@Override
public void accept(HttpHeaders httpHeaders) {
// https://github.com/spring-projects/spring-ai/issues/372
httpHeaders.set("Accept-Encoding", "gzip, deflate");
}
});
}
Hope this helps.
Comment From: ThomasVitale
Spring Boot provides an autoconfigured RestClient.Builder object, including JSON serialization configuration. Have you tried using that one instead of defining one yourself?
When you create a RestClient via RestClient.builder(), you don't get the Spring Boot autoconfiguration, so it's up to you to configure encoding and serialization. If you create one from an autowired RestClient.Builder bean, that is all configured for you and should work out-of-the-box.
Comment From: Alien2150
So my workaround that is not that invasive as overwritting the default RestClient builder:
@Bean
fun chatModel(
commonProperties: OpenAiConnectionProperties,
chatProperties: OpenAiChatProperties,
webClientBuilder: WebClient.Builder,
retryTemplate: RetryTemplate,
functionCallbackContext: FunctionCallbackContext,
responseErrorHandler: ResponseErrorHandler,
): OpenAiChatModel {
val restClientBuilder = RestClient.builder().defaultHeaders {
it.set(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate")
}
val openAiApi = OpenAiApi(
chatProperties.baseUrl ?: commonProperties.baseUrl,
chatProperties.apiKey ?: commonProperties.apiKey,
restClientBuilder,
webClientBuilder,
responseErrorHandler,
)
return OpenAiChatModel(
openAiApi,
chatProperties.options,
functionCallbackContext,
retryTemplate,
)
}
As a suggestion: Could the bean for RestClient builder not use a "Qualifier" annotation so it would not be overwritten by the default one? (OpenAiAutoConfiguration seems to use the default one)
Comment From: markpollack
I was thining the same thing @Alien2150 in relation to setting timeouts per model, using a qualifier annotation that will pick up a specific restClientBuilder. It would seem that this problem would have been solved already with apps calling many microservices, each with different time outs... https://github.com/spring-projects/spring-ai/issues/512 and related issues discuss it more.
Comment From: coderphonui
So my workaround that is not that invasive as overwritting the default RestClient builder:
``` @Bean fun chatModel( commonProperties: OpenAiConnectionProperties, chatProperties: OpenAiChatProperties, webClientBuilder: WebClient.Builder, retryTemplate: RetryTemplate, functionCallbackContext: FunctionCallbackContext, responseErrorHandler: ResponseErrorHandler, ): OpenAiChatModel { val restClientBuilder = RestClient.builder().defaultHeaders { it.set(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate") }
val openAiApi = OpenAiApi( chatProperties.baseUrl ?: commonProperties.baseUrl, chatProperties.apiKey ?: commonProperties.apiKey, restClientBuilder, webClientBuilder, responseErrorHandler, ) return OpenAiChatModel( openAiApi, chatProperties.options, functionCallbackContext, retryTemplate, ) }```
As a suggestion: Could the bean for RestClient builder not use a "Qualifier" annotation so it would not be overwritten by the default one? (OpenAiAutoConfiguration seems to use the default one)
Adding the httpHeaders.set("Accept-Encoding", "gzip, deflate"); fixed the issue. Thanks!
Comment From: asw12
I encountered and researched this defect a bit further out of curiosity, and found the culprit to be brotli decompression support, hence why using an Accept-Encoding that excludes "br" avoids the problem.
The dependency tree bringing in brotli looks something like:
[INFO] | +- org.springframework.ai:spring-ai-tika-document-reader:jar:1.0.0-M6:compile
...
[INFO] | | +- org.apache.tika:tika-parser-pkg-module:jar:3.0.0:compile
[INFO] | | | +- org.brotli:dec:jar:0.1.2:compile
where that brotli dec-0.1.2.jar was a version that was last published to Maven Central by Google 7 years ago.
There is an issue with BrotliInputStream in that library when they read from an internal buffer without depleting it, that the read method incorrectly reports a -1 value signifying an EOF. There appears to be a fix in later versions, but even though this fix was committed 3 years ago, they never published it.
With the HttpComponents ClientHttpRequestFactory, note that the presence of the BrotliInputStream class is enough to have the underlying HTTP Client 5 library to add the accept header.
So the problem looks like * Jackson does a preliminary check to see if the body has content by calling read() for one byte. * BrotliInputStream populates the buffer with ~8kb worth of data. '{' is read from it. * That character is "unread" back to a buffer held by a wrapping PushbackInputStream * Jackson tries to parse JSON from the InputStream with its parser. * The '{' is read back, followed by some hundreds of bytes, but not enough to cause BrotliInputStream's buffer to be depleted. * BrotliInputStream reports -1 for the read call, so Jackson only sees a START_OBJECT before an unexpected JsonEOFException is thrown.