Current Behavior

Currently, existing convert() implementations (BeanOutputConverter, ListOutputConverter, MapOutputConverter) throw exceptions when the response can't be parsed.

Proposed Behavior

I saw Spring AI has retry properties: https://docs.spring.io/spring-ai/reference/api/chat/openai-chat.html#_chat_properties But these are for the initial calls to the model if I understood correctly, not the converters.

Would it be a good idea to set the optional amount of retries for the converters too? If parsing fails, we can retry to call the AI model in hopes of getting a better response.

Context

I am getting the wrong output from the model from time to time, so I configured retries in my code. But what do you think about having this feature in Spring AI?

Comment From: ThomasVitale

I like the suggestion. LangChain provides a couple of OutputParsers for dealing with errors:

  • an OutputFixingParser which contacts a second model in case the default one fails in producing a valid output;
  • a RetryOutputParser which tries again with the same model with the hope to get a better response.

It would be nice to have something similar for the StructuredOutputConverter infrastructure in Spring AI:

Comment From: etlweather

That's how I coded my app so far. Also, I think the parsers need to be more lenient with the responses, particularly when using smaller open models, there is a fair amount of small errors (don't know if OpenAI or other larger models make the same amount of errors).

I am doing a simple List<String> and I get a fair amount of errors with Mistral 7B. Even a response like:

* Entry one
* Entry two
* Entry three

Didn't parse as a List<String> because it's expecting JSON.

Comment From: etlweather

... yes, I am going to try the ListOutputConverter - that might work better in my case... still would like the retry!