Comment From: csterwa
@tzolov does the PR referenced here resolve the retry implementation enhancement?
Comment From: ThomasVitale
Adding some context to this issue: a RetryTemplate is used to add retry capabilities to the blocking model calls. At first, the same strategy was used for the streaming calls. After realising that it wasn't working as expected in a reactive stream, the usage of RetryTemplate streaming calls has been removed. In this issue, we would need to come up with a design for a retry feature to adopt in all streaming implementations across the different model providers.
Comment From: apappascs
Hi @ThomasVitale , @tzolov
I've been working on a potential solution for adding retry logic to the ChatModel streaming functionality. To illustrate the approach, I’ve created a branch with the necessary changes https://github.com/spring-projects/spring-ai/compare/main...apappascs:spring-ai:retry-implementation-openai-chatmodel-streaming
Would an implementation like this make sense for the project? I’d love to get feedback and refine the approach further if necessary.
Comment From: ilayaperumalg
@chemicL Could you check the above? Thanks
Comment From: chemicL
@ilayaperumalg sure, if you meant the proposal from @apappascs, my comment follows.
@apappascs thanks for taking the time to look into it. In my view it is not necessary to mimic the RetryTemplate. The particular Flux/Mono can have retry defined on top using the retryWhen spec directly without the builder in between. What is probably more important is whether the caller of the SpringAI API should take care of retries or whether that functionality is built in. If it's built-in then some specification in the API can be provided that maps to the retryWhen usage internally. Otherwise, the user should be free to define a retry logic on top of the streaming responses themselves. E.g. in case of WebClient use typically it's the user defining the retry on the returned Mono/Flux types from the API.
@tzolov can you offer some guidance what your expectations are?