If the user question (or the available functions) force multiple roundtrips of the user flow, the existing Azure OpenAI implementation only calls functions in the first roundtrip.
Example Question: What's the weather like in San Francisco? If the weather is above 25 degrees, please check the weather in Tokyo and Paris
==> Will return:
The reason for this issue is, that the second call to Azure OpenAI (including the tool_id response) does not include the tools of the original request anymore. This of course makes the model unable to call another tool if required.
Unfortunatly there are additional refactorings required to enable streaming support - see first comment.
Should/Might also fix: - https://github.com/spring-projects/spring-ai/discussions/588 - https://github.com/spring-projects/spring-ai/issues/613 (at least partly)
Comment From: timostark
The first simple fix did not work in the streaming scenario. Also Azure OpenAI was not ready for multiple roundtrips. Reason seems to be some "hacks" required for the azureApi in the ChatModel.stream method. These hacks weren't called again on the second roundtrip as the AbstractFunctionCallSupport was directly calling the Azure Library instead of passing back to the stream method (like e.g. the OpenAI implementation is doing it..).
The suggested fix extracts the azure part of the streaming (the "hacks" :-) ). In case of tool calls - identical to openai - the azure model itself is handling the message adding and recursive calling of the streaming api.
PS: I think that my solution is not really good, but imo a "bigger" refactoring would be required at the streaming side.. as far as i see the flux mapping prevents the consumer to retrieve the tokens as soon as they are sent.. (making streaming "senseless" at the moment). I am not 100% sure where to start here, so i just tried to make it usable for now.
Comment From: timostark
One last remark.. The upgrade to azure-ai-openai 1.0.0.-beta.10 (from 8) causes a ton of JsonEOFException during streaming && tool-calls (reproducible also before my change).
<artifactId>azure-ai-openai</artifactId>
<version>1.0.0-beta.10</version>
These issues do not occur with beta 8.
Comment From: tzolov
Hi @timostark , I'm so grateful for your efforts in identifying and solving this issue. Great job!
Currently, I'm refactoring, the Azure OpenAI function calling support to raise the API to the Spring AI API level. Similar to what I did recently fro OpenAI, Anthropic and VertexAI Gemini. Have the feeling that this might solve the issue for both call and streaming. Will reuse your ITs to verify it.
Btw, what name should I use for the javadoc @author? Your timostark handle or something else?
Comment From: timostark
@tzolov i've quickly raised https://github.com/spring-projects/spring-ai/pull/1054 to avoid double efforts.. I was refactoring Azure AI to be compliant to OpenAI API and also switch to the Async API. By that the streaming is actually working...
I've also reverted to beta 8 and reported an azure bug ( https://github.com/Azure/azure-sdk-for-java/issues/41164 )
If you are fine with the direction that new PR is going (it includes the changes of this PR) you can close this one..
Comment From: timostark
ah and @autor timostark is fine :-)
Comment From: tzolov
@timostark , I'm not sure what causes the json reading errors, but it seems to be some Azure API bug.
Even if I iterate the direct openAIClient response without and processing:
IterableStream<ChatCompletions> chatCompletionsStream = this.openAIClient.getChatCompletionsStream(options.getModel(), options);
chatCompletionsStream.stream().forEach(cc -> System.out.println(cc));
I still see errors like this, showing that the underlining json have been cut in the middle
ERROR 82446 --- [oundedElastic-1] c.a.c.i.s.DefaultJsonSerializer : com.azure.json.implementation.jackson.core.io.JsonEOFException: Unexpected end-of-input within/between Object entries at [Source: (byte[])"{"choices":[{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"delta":{"content":" Story"},"finish_reason":null,"index""; line: 1, column: 285]
Comment From: tzolov
@timostark, please feel free to review this PR before merged: https://github.com/spring-projects/spring-ai/pull/1055
the https://github.com/spring-projects/spring-ai/pull/1054 come bit late for me. But if there is something in 1054 missing in 1055 feel free to rise another PR.
Again , thank you very much for the effort.
Comment From: tzolov
Most of it is addressed by the feb036d2f6534505ad0bdac12925f93a52737e73