Spring-ai asking for longer prompts triggers - Request processing failed: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:11434/api/chat": timeout

Hello, I see that there was a possible duplicate of this but it was closed as not having provided enough info

Bug description i have tried to use phi3, llama3.2:1b, llama3.2:latest... i have asked questions with wordy answers inside the terminal and i get a response every time. However, with the spring ai project, every time i ask a question that should generate a somewhat longer answer, i get the above error; if i ask plain things like 'count to 10' - i get a response; If i ask 'count to 100' - error above followed by 'Socket closed '

Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:11434/api/chat": timeout] with root cause

java.net.SocketException: Socket closed
    at java.base/sun.nio.ch.NioSocketImpl.endRead(NioSocketImpl.java:253) ~[na:na]
    at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:332) ~[na:na]
    at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355) ~[na:na]
    at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808) ~[na:na]
    at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966) ~[na:na]
    at okio.InputStreamSource.read(JvmOkio.kt:93) ~[okio-jvm-3.6.0.jar:na]
    at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:128) ~[okio-jvm-3.6.0.jar:na]
    at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:430) ~[okio-jvm-3.6.0.jar:na]
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:323) ~[okio-jvm-3.6.0.jar:na]

Environment

<properties>
        <java.version>17</java.version>
        <spring-ai.version>1.0.0-M3</spring-ai.version>
   </properties>
<dependency>
                <groupId>org.springframework.ai</groupId>
                <artifactId>spring-ai-bom</artifactId>
                <version>${spring-ai.version}</version>
                <type>pom</type>
                <scope>import</scope>
</dependency>

i run an amazon Linux EC2 with processor: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16GB RAM

-- could these limited resources be problematic? Steps to reproduce created a plain spring boot project with spring-ai-bom; added a plain ChatController with contents

private final ChatModel chatModel; public ChatController(ChatModel chatModel) { this.chatModel = chatModel; }

@GetMapping("/ask")
public String askQuestion(@RequestParam(value = "message") String message) {
    return chatModel.call(message);
}

Expected behavior i expect a response from the model which is as large as its output context (1024 chars for ex for llama3.2:1b)

Minimal Complete Reproducible example setup as above and ask to provide a long description;

Comment From: ThomasVitale

You might need to increase the default timeout value.

If you're using Spring Boot 3.3, then you can use a RestClientCustomizer.

@Bean
RestClientCustomizer restClientCustomizer() {
      return restClientBuilder -> {
          restClientBuilder
                  .requestFactory(new BufferingClientHttpRequestFactory(
                          ClientHttpRequestFactories.get(ClientHttpRequestFactorySettings.DEFAULTS
                                          .withConnectTimeout(Duration.ofSeconds(60))
                                          .withReadTimeout(Duration.ofSeconds(120))
                          )));
      };
}

In Spring Boot 3.4, you can customise that via configuration properties:

spring:
  http:
    client:
      connect-timeout: 30s
      read-timeout: 120s

There is a feature request to make these timeouts more easily configurable for Spring AI: https://github.com/spring-projects/spring-ai/issues/512

Comment From: vzR

thanks a lot for the quick reply - indeed this works - amazing help; to me this issue can be closed.

Comment From: ThomasVitale

@vzR happy to help! I'm glad that fixed the problem. I think you have the option down here to close the issue.

Comment From: vzR

thanks for the quick help! closing the issue.

Comment From: MusicBoooox

You might need to increase the default timeout value.

If you're using Spring Boot 3.3, then you can use a RestClientCustomizer.

java @Bean RestClientCustomizer restClientCustomizer() { return restClientBuilder -> { restClientBuilder .requestFactory(new BufferingClientHttpRequestFactory( ClientHttpRequestFactories.get(ClientHttpRequestFactorySettings.DEFAULTS .withConnectTimeout(Duration.ofSeconds(60)) .withReadTimeout(Duration.ofSeconds(120)) ))); }; }

In Spring Boot 3.4, you can customise that via configuration properties:

spring: http: client: connect-timeout: 30s read-timeout: 120s

There is a feature request to make these timeouts more easily configurable for Spring AI: #512

If I use springboot3.2，what I should do?