Spring-ai Will Spring-AI support async processing?

Please do a quick search on GitHub issues first, the feature you are about to request might have already been requested.

Expected Behavior

I recommand that Spring AI provide SPI/API that supports asynchronous processing.

Current Behavior

The ChatClient is currently

public interface ChatClient {

    default String generate(String message) { // implementation omitted
    }

    AiResponse generate(Prompt prompt);

}

which only allows synchronous processing. In similar frameworks like langchain they have both generate and agenerate. It would be good and reasonable for SPring-AI framework to support async so that we can make use of many async process features & libs in Java for example netty, project reactor, RxJava, etc., perhaps with better performance.

Comment From: markpollack

Yes, but the documentation (and issues) needs updating.

The interface that supports this is StreamingChatClient

@FunctionalInterface
public interface StreamingChatClient {

    Flux<ChatResponse> generateStream(Prompt prompt);

}

There are implementations for

• OpenAI
• Azure OpenAI
• AWS Bedrock Antropic
• AWS Bedrock Cohere
• AWS Bedrock Llama2
• AWS Bedrock Titan
• Ollama

I've created an issue to add docs for these.

We need to look into the other model providers we support to see which support streaming and then create issues for each of them. Created this issue to track that.