Spring-ai OpenAi Stream miss some data

Please do a quick search on GitHub issues first, there might be already a duplicate issue for the one you are about to create. If the bug is trivial, just go ahead and create the issue. Otherwise, please take a few moments and fill in the following sections:

Bug description When I use stream output with OpenAI, I found some times there are some data missed.

Environment 1.0.0-SNAPSHOT

Steps to reproduce I debugged the code and found that the issue likely occurs in the following section of the code. I noticed that "ChatCompletion before handling" prints the data returned by OpenAI, but "ChatCompletion after handling" does not print anything, indicating that the data is lost at this step. However, I am not very familiar with Flux, so could you please assist in locating and fixing the issue? Spring-ai OpenAi Stream miss some data

Comment From: wangyulong-patsnap

I never use function call, so isToolFunctionCall in method "handleFunctionCallOrReturnStream" is always false

Comment From: mkleine

We observed the same. Returning the OpenAI result as a flux gives incomplete results.

Comment From: mfadili

Indeed, observed the same issue.

Comment From: didalgolab

It works fine for me. I use Spring AI with OpenAI default API endpoints in streaming mode and I didn't noticed any issues. You need to provide reproducible example.

I have also run a quick JUnit test for completeness, which passes successfully::

    @Test
    void streamCompletenessTest() throws InterruptedException {
        UserMessage userMessage = new UserMessage(
                "List ALL natural numbers in range [1, 1000]. Make sure to not omit any.");
        Prompt prompt = new Prompt(List.of(userMessage));
        Flux<ChatResponse> flux = streamingChatModel.stream(prompt);

        StringBuilder stitchedResponseContent = new StringBuilder();
        CountDownLatch latch = new CountDownLatch(1);

        flux.subscribe(
                resp -> stitchedResponseContent.append(resp.getResult().getOutput().getContent()),
                err -> {
                    System.err.println("Error occurred: " + err.getMessage());
                    latch.countDown();
                },
                () -> {
                    System.out.println("Flux completed");
                    latch.countDown();
                }
        );

        boolean completed = latch.await(120, TimeUnit.SECONDS);
        if (!completed) {
            throw new AssertionError("Flux did not complete within the timeout period");
        }

        System.out.println("LLM responded: " + stitchedResponseContent);
        IntStream.rangeClosed(1, 1000).forEach(n -> {
            assertThat(stitchedResponseContent).contains(String.valueOf(n));
        });
    }

Flux completed
LLM responded: Sure! Here is the complete list of natural numbers from 1 to 1000:

1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
...
991, 992, 993, 994, 995, 996, 997, 998, 999, 1000.

Tests passed: 1 of 1 test

Comment From: mfadili

Thank your for your answer. Here is my case and the evidence of the issue based on your example: My spring api code is:

StringBuilder answer = new StringBuilder();
        Flux<org.springframework.ai.chat.model.ChatResponse> flux = chatModel.stream(prompt);
        return flux.doOnNext(chatResponse -> {
            String responseContent = chatResponse.getResults().get(0).getOutput().getContent();
            answer.append(responseContent);
            log.info("ChatResponse: {}", responseContent);
        }).doOnComplete(() -> {
            gptMessageService.save(threadId, question, answer.toString());
            storeConversationHistory(threadId);
        });

This flux is consumed from a Next Js frontend with EventSourcePolyfill library.

You see that: 6, 9, 10, 11, 12, 48 to 56, etc... are missing in the log.

2024-06-29T14:53:58.142+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 2024-06-29T14:53:58.155+02:00 INFO 23428 --- [ajielms] [task-1] c.a.m.service.gpt.AjielAssistantService : ChatResponse: Sure 2024-06-29T14:53:58.165+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.166+02:00 INFO 23428 --- [ajielms] [task-3] c.a.m.service.gpt.AjielAssistantService : ChatResponse: here 2024-06-29T14:53:58.185+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: is 2024-06-29T14:53:58.186+02:00 INFO 23428 --- [ajielms] [task-5] c.a.m.service.gpt.AjielAssistantService : ChatResponse: the 2024-06-29T14:53:58.205+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: list 2024-06-29T14:53:58.207+02:00 INFO 23428 --- [ajielms] [task-7] c.a.m.service.gpt.AjielAssistantService : ChatResponse: of 2024-06-29T14:53:58.237+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: all 2024-06-29T14:53:58.239+02:00 INFO 23428 --- [ajielms] [task-9] c.a.m.service.gpt.AjielAssistantService : ChatResponse: natural 2024-06-29T14:53:58.254+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: numbers 2024-06-29T14:53:58.255+02:00 INFO 23428 --- [ajielms] [task-11] c.a.m.service.gpt.AjielAssistantService : ChatResponse: in 2024-06-29T14:53:58.279+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: the 2024-06-29T14:53:58.280+02:00 INFO 23428 --- [ajielms] [task-13] c.a.m.service.gpt.AjielAssistantService : ChatResponse: range 2024-06-29T14:53:58.304+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: from 2024-06-29T14:53:58.306+02:00 INFO 23428 --- [ajielms] [task-15] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.333+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 1 2024-06-29T14:53:58.334+02:00 INFO 23428 --- [ajielms] [task-17] c.a.m.service.gpt.AjielAssistantService : ChatResponse: to 2024-06-29T14:53:58.350+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.351+02:00 INFO 23428 --- [ajielms] [task-19] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 100 2024-06-29T14:53:58.373+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 0 2024-06-29T14:53:58.375+02:00 INFO 23428 --- [ajielms] [task-21] c.a.m.service.gpt.AjielAssistantService : ChatResponse: :

2024-06-29T14:53:58.396+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 1 2024-06-29T14:53:58.397+02:00 INFO 23428 --- [ajielms] [task-23] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.423+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.424+02:00 INFO 23428 --- [ajielms] [task-25] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 2 2024-06-29T14:53:58.442+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.444+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.464+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 3 2024-06-29T14:53:58.465+02:00 INFO 23428 --- [ajielms] [task-29] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.490+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.493+02:00 INFO 23428 --- [ajielms] [task-31] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 4 2024-06-29T14:53:58.726+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.730+02:00 INFO 23428 --- [ajielms] [task-33] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.752+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 5 2024-06-29T14:53:58.753+02:00 INFO 23428 --- [ajielms] [task-35] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.755+02:00 INFO 23428 --- [ajielms] [task-36] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.759+02:00 INFO 23428 --- [ajielms] [task-37] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.762+02:00 INFO 23428 --- [ajielms] [task-38] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 7 2024-06-29T14:53:58.763+02:00 INFO 23428 --- [ajielms] [task-39] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.764+02:00 INFO 23428 --- [ajielms] [task-40] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 8 2024-06-29T14:53:58.769+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.770+02:00 INFO 23428 --- [ajielms] [task-42] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.774+02:00 INFO 23428 --- [ajielms] [task-43] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.775+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.793+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.793+02:00 INFO 23428 --- [ajielms] [task-46] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 13 2024-06-29T14:53:58.952+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:58.953+02:00 INFO 23428 --- [ajielms] [task-48] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:58.987+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 14 2024-06-29T14:53:58.989+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.026+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 15 2024-06-29T14:53:59.027+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.028+02:00 INFO 23428 --- [ajielms] [task-53] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.046+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 16 2024-06-29T14:53:59.047+02:00 INFO 23428 --- [ajielms] [task-55] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.048+02:00 INFO 23428 --- [ajielms] [task-56] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.076+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 17 2024-06-29T14:53:59.077+02:00 INFO 23428 --- [ajielms] [task-58] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.106+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 18 2024-06-29T14:53:59.107+02:00 INFO 23428 --- [ajielms] [task-60] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.138+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 19 2024-06-29T14:53:59.139+02:00 INFO 23428 --- [ajielms] [task-62] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.139+02:00 INFO 23428 --- [ajielms] [task-63] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.167+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 20 2024-06-29T14:53:59.168+02:00 INFO 23428 --- [ajielms] [task-65] c.a.m.service.gpt.AjielAssistantService : ChatResponse: ,

2024-06-29T14:53:59.220+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 21 2024-06-29T14:53:59.221+02:00 INFO 23428 --- [ajielms] [task-67] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.251+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 22 2024-06-29T14:53:59.252+02:00 INFO 23428 --- [ajielms] [task-69] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.284+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 23 2024-06-29T14:53:59.285+02:00 INFO 23428 --- [ajielms] [task-71] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.324+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 24 2024-06-29T14:53:59.325+02:00 INFO 23428 --- [ajielms] [task-73] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.325+02:00 INFO 23428 --- [ajielms] [task-74] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.359+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 25 2024-06-29T14:53:59.359+02:00 INFO 23428 --- [ajielms] [task-76] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.384+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 26 2024-06-29T14:53:59.385+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.420+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 27 2024-06-29T14:53:59.421+02:00 INFO 23428 --- [ajielms] [task-80] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.421+02:00 INFO 23428 --- [ajielms] [task-81] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.459+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 28 2024-06-29T14:53:59.460+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.495+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 29 2024-06-29T14:53:59.495+02:00 INFO 23428 --- [ajielms] [task-85] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.496+02:00 INFO 23428 --- [ajielms] [task-86] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.682+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 30 2024-06-29T14:53:59.683+02:00 INFO 23428 --- [ajielms] [task-88] c.a.m.service.gpt.AjielAssistantService : ChatResponse: ,

2024-06-29T14:53:59.684+02:00 INFO 23428 --- [ajielms] [task-89] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 32 2024-06-29T14:53:59.685+02:00 INFO 23428 --- [ajielms] [task-90] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.686+02:00 INFO 23428 --- [ajielms] [task-91] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 33 2024-06-29T14:53:59.687+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.687+02:00 INFO 23428 --- [ajielms] [task-93] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.688+02:00 INFO 23428 --- [ajielms] [task-94] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 34 2024-06-29T14:53:59.688+02:00 INFO 23428 --- [ajielms] [task-95] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.721+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.722+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 35 2024-06-29T14:53:59.722+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.724+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.742+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 36 2024-06-29T14:53:59.743+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.745+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.780+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 37 2024-06-29T14:53:59.783+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.784+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.805+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 38 2024-06-29T14:53:59.807+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:53:59.807+02:00 INFO 23428 --- [ajielms] [task-108] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.844+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 39 2024-06-29T14:53:59.847+02:00 INFO 23428 --- [ajielms] [task-110] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.879+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 40 2024-06-29T14:53:59.881+02:00 INFO 23428 --- [ajielms] [task-112] c.a.m.service.gpt.AjielAssistantService : ChatResponse: ,

2024-06-29T14:53:59.907+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 41 2024-06-29T14:53:59.908+02:00 INFO 23428 --- [ajielms] [task-114] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:53:59.943+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 42 2024-06-29T14:53:59.944+02:00 INFO 23428 --- [ajielms] [task-116] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.013+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 43 2024-06-29T14:54:00.014+02:00 INFO 23428 --- [ajielms] [task-118] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.022+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 44 2024-06-29T14:54:00.023+02:00 INFO 23428 --- [ajielms] [task-120] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.052+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 45 2024-06-29T14:54:00.055+02:00 INFO 23428 --- [ajielms] [task-122] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.083+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 46 2024-06-29T14:54:00.083+02:00 INFO 23428 --- [ajielms] [task-124] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.144+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 47 2024-06-29T14:54:00.145+02:00 INFO 23428 --- [ajielms] [task-126] c.a.m.service.gpt.AjielAssistantService : ChatResponse: ,

2024-06-29T14:54:00.147+02:00 INFO 23428 --- [ajielms] [task-127] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:54:00.215+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.215+02:00 INFO 23428 --- [ajielms] [task-129] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:54:00.399+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.399+02:00 INFO 23428 --- [ajielms] [task-131] c.a.m.service.gpt.AjielAssistantService : ChatResponse: , 2024-06-29T14:54:00.419+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.420+02:00 INFO 23428 --- [ajielms] [task-133] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.463+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 57 2024-06-29T14:54:00.463+02:00 INFO 23428 --- [ajielms] [task-135] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.491+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 58 2024-06-29T14:54:00.492+02:00 INFO 23428 --- [ajielms] [task-137] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.524+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 59 2024-06-29T14:54:00.525+02:00 INFO 23428 --- [ajielms] [task-139] c.a.m.service.gpt.AjielAssistantService : ChatResponse:
2024-06-29T14:54:00.577+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 60 2024-06-29T14:54:00.577+02:00 INFO 23428 --- [ajielms] [task-141] c.a.m.service.gpt.AjielAssistantService : ChatResponse: ,

2024-06-29T14:54:00.825+02:00 INFO 23428 --- [ajielms] [reactor-http-nio-2] c.a.m.service.gpt.AjielAssistantService : ChatResponse: 61

Comment From: cy1665

Indeed, observed the same issue.

Comment From: mfadili

Could this issue be related to this one in openai ? https://community.openai.com/t/4o-and-4-api-output-has-typo-missing-words/869225

Comment From: markpollack

Is the issue the use of doOnNext vs subscribe?

Comment From: wangyulong-patsnap

Is the issue the use of doOnNext vs subscribe?问题是 doOnNext 使用 vs subscribe 吗？

I used OpenAiChatModel.stream

Comment From: cy1665

Is the issue the use of doOnNext vs subscribe?

Could this issue be related to this one in openai ? https://community.openai.com/t/4o-and-4-api-output-has-typo-missing-words/869225

I added the origin log, openai output is normal.

I rewrite OpenAiChatModel to remove the switchMap() , it's work normal now.

Comment From: cy1665

I also replace the flatMap to concatMap, because it's may causes output order problems.

Comment From: uwilken

Same issue here. Looks like the streaming API is working differently for different chat client implementations. With bedrock anthropic i get full responses. With open ai I receive incomplete answers.

The code is as simple as that:

  @PostMapping(value = "/flux", produces = MediaType.TEXT_PLAIN_VALUE)
  public Flux<String> streamData() {

    return chatClient.prompt().user("who is joe biden?")
            .stream()
            .content();
  }

Example open ai response:

Joe Biden is an American politician and attorney who serves as the46 president of the United He was born20,194, in, Pennsylvania. Biden began his political career in when he was the U from Delaware, serving from1973 to2009. He gained national prominence as a of the Foreign Relations as a vice president Barack Obama.
...

He was born20,194,

Example anthropic response:

 Here is a summary of key information about Joe Biden:

- Full Name: Joseph Robinette Biden Jr.
- Born: November 20, 1942 (age 80) in Scranton, Pennsylvania
- Political Party: Democratic
...

Comment From: sobychacko

Based on the information provided above, here is a simple integration test against OpenAI. It works as expected every time I run it. Could someone please take a look at this test and see if we can reproduce the problems mentioned on this issue? That way, we can triage it further. I am trying to make it fail so we can see the issue in the test itself.

@Test
void streamCompletenessTest() throws InterruptedException {
    UserMessage userMessage = new UserMessage(
            "List ALL natural numbers in range [1, 1000]. Make sure to not omit any.");
    Prompt prompt = new Prompt(List.of(userMessage));

    StringBuilder answer = new StringBuilder();
    CountDownLatch latch = new CountDownLatch(1);

    Flux<ChatResponse> chatResponseFlux = streamingChatModel.stream(prompt)
            .doOnNext(chatResponse -> {
                String responseContent = chatResponse.getResults().get(0).getOutput().getContent();
                answer.append(responseContent);
            }).doOnComplete(() -> {
                System.out.println(answer);
                latch.countDown();
            });
    chatResponseFlux.subscribe();
    assertThat(latch.await(120, TimeUnit.SECONDS)).isTrue();
    IntStream.rangeClosed(1, 1000).forEach(n -> {
        assertThat(answer).contains(String.valueOf(n));
    });
}

Comment From: sobychacko

@wangyulong-patsnap Could you try this test and see if it doesn't work for you? Please advise on a way to make this test fail, so that we can try to reproduce your scenario. Thanks!

Comment From: sobychacko

We added this test above as part of the test suite. See the commit: https://github.com/spring-projects/spring-ai/commit/4d41080d455b3584aa10ec5c7046144890de852c

It passes on our CI system. As I indicated above, if you encounter this issue, please review that test and see if you can reproduce it. Thanks!

Comment From: wangyulong-patsnap

@wangyulong-patsnap Could you try this test and see if it doesn't work for you? Please advise on a way to make this test fail, so that we can try to reproduce your scenario. Thanks!

I tried this test, and it's working normally now. I'm unable to reproduce the issue at this moment, though it was consistently reproducible before. When it was consistently occurring, I attempted using the same model with the langchain4j framework for streaming output, and it worked fine.

Comment From: uwilken

I tried to reproduce the unit test, I modified it and found out that there seem to be a problem with org.springframework.ai.chat.client.ChatClient.ChatClientRequest.StreamPromptResponseSpec#content.

Here is my test code: working:

  @Test
  void streamCompletenessTestWithChatClientWorking() throws InterruptedException {
    UserMessage userMessage = new UserMessage(
            "Who is joe biden?");
    Prompt prompt = new Prompt(List.of(userMessage));

    StringBuilder answer = new StringBuilder();
    CountDownLatch latch = new CountDownLatch(1);

    ChatClient chatClient = ChatClient.builder(openAiChatModel).build();

    Flux<ChatResponse> chatResponseFlux = chatClient.prompt(prompt).stream()
            .chatResponse()
            .doOnNext(chatResponse -> {
              String responseContent = chatResponse.getResults().get(0).getOutput().getContent();
              answer.append(responseContent);
            }).doOnComplete(() -> {
              latch.countDown();
            });
    chatResponseFlux.subscribe();
    assertThat(latch.await(120, TimeUnit.SECONDS)).isTrue();
    System.out.println(answer);
    assertThat(answer).contains("the 46th ");
  }

Broken test:

@Test
  void streamCompletenessTestWithChatClientBidenBroken() throws InterruptedException {
    UserMessage userMessage = new UserMessage(
            "Who is joe biden?");
    Prompt prompt = new Prompt(List.of(userMessage));

    StringBuilder answer = new StringBuilder();
    CountDownLatch latch = new CountDownLatch(1);

    ChatClient chatClient = ChatClient.builder(openAiChatModel).build();

    //The #content() method seems to swallow the blanks
    Flux<String> chatResponseFlux = chatClient.prompt(prompt).stream().content()
            .doOnNext(chatResponse -> {
              answer.append(chatResponse);
            }).doOnComplete(() -> {
              latch.countDown();
            });
    chatResponseFlux.subscribe();
    assertThat(latch.await(120, TimeUnit.SECONDS)).isTrue();
    System.out.println(answer);
    assertThat(answer).contains("the 46th ");
  }

At least chatClient.prompt(prompt).stream().content() seems to swallow leading blanks.

Comment From: mkleine

``` @PostMapping(value = "/flux", produces = MediaType.TEXT_PLAIN_VALUE) public Flux streamData() {
return chatClient.prompt().user("who is joe biden?")
        .stream()
        .content();
} ```

I've extended this example and it turns out that the doOnNext approach gives complete responses, while webflux returns incomplete responses if we're just passing the flux to the framework.

The following example writes complete answers to the log:

  @PostMapping(value = "/complete", produces = MediaType.TEXT_PLAIN_VALUE)
  public Flux<String> complete() {
      StringBuilder answer = new StringBuilder();

      Flux<ChatResponse> flux = chatClient.prompt()
              .user("Who is Joe Biden?").stream()
              .chatResponse()
              .doOnNext(chatResponse -> {
                String responseContent = ofNullable(chatResponse.getResults())
                        .filter(results -> !results.isEmpty())
                        .map(results -> results.get(0))
                        .map(Generation::getOutput)
                        .map(AssistantMessage::getContent)
                        .orElse("");
                answer.append(responseContent);
              }).doOnComplete(() -> {
                System.out.println(answer);
              });

    flux.subscribe();
    return Mono.just("complete result in log").flux();
  }

while this example returns incomplete responses to the client

``` @PostMapping(value = "/incomplete", produces = MediaType.TEXT_PLAIN_VALUE) public Flux incomplete() { return chatClient.prompt() .user("Who is Joe Biden?").stream() .chatResponse() .map(f -> ofNullable(f).map(ChatResponse::getResult) .map(Generation::getOutput) .map(AssistantMessage::getContent) .orElse("") ); }

Comment From: mkleine

At least chatClient.prompt(prompt).stream().content() seems to swallow leading blanks.

This sounds very much like https://github.com/spring-projects/spring-ai/issues/1089, doesn't it?

Comment From: sobychacko

Fixed via this issue: https://github.com/spring-projects/spring-ai/issues/1089