Bug description
I ran LM Studio Local LLM API Server (OpenAI compatibility mode). ChatClient.call()
API works perfectly, but the ChatClient.stream()
does not work!
I enabled debug logs and also used the SimpleLoggerAdvisor
and saw this log for streaming:
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet : POST "/stream", parameters={}
DEBUG 54629 --- [streaming] [nio-8080-exec-6] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to dev.danvega.streaming.ChatController#chatWithStream(String)
DEBUG 54629 --- [streaming] [nio-8080-exec-6] m.m.a.RequestResponseBodyMethodProcessor : Read "application/json;charset=UTF-8" to ["{"message": "What is the largest country on Earth?"}"]
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.a.c.c.advisor.SimpleLoggerAdvisor : request: AdvisedRequest[chatModel=OpenAiChatModel [defaultOptions=OpenAiChatOptions: {"streamUsage":false,"model":"meta-llama-3.1-8b-instruct","temperature":0.7}], userText={"message": "What is the largest country on Earth?"}, systemText=, chatOptions=OpenAiChatOptions: {"streamUsage":false,"model":"meta-llama-3.1-8b-instruct","temperature":0.7}, media=[], functionNames=[], functionCallbacks=[], messages=[], userParams={}, systemParams={}, advisors=[org.springframework.ai.chat.client.advisor.observation.ObservableRequestResponseAdvisor@6e5f4d0], advisorParams={}]
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.w.r.f.client.ExchangeFunctions : [8220a7c] HTTP POST http://localhost:1234/v1/chat/completions
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.http.codec.json.Jackson2JsonEncoder : [8220a7c] Encoding [ChatCompletionRequest[messages=[ChatCompletionMessage[rawContent={"message": "What is the largest co (truncated)...]
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.w.c.request.async.WebAsyncManager : Started async request for "/stream"
DEBUG 54629 --- [streaming] [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet : Exiting but response remains open for further handling
But I could not see anything in the LM Studio server log!
Environment
Spring AI version: 1.0.0-M6
Java version: 21
OS: macOS and Windows 11
Steps to reproduce
I used the spring-ai-openai-spring-boot-starter
lib with this configuration:
ai:
openai:
api-key: lm-studio
base-url: http://localhost:1234
chat:
options:
model: meta-llama-3.1-8b-instruct
embedding:
options:
model: text-embedding-nomic-embed-text-v1.5-embedding
Expected behavior
When I call LM Studio Local LLM API Server (OpenAI compatibility mode) using the curl command (with stream enabled), the ChatClient.call()
API also works with the above configuration.
Minimal Complete Reproducible example
You can run an LM Studio Local LLM API Server (OpenAI compatibility mode) and clone this example from @danvega and apply my my configs. You will see that the ChatClient.call()
works, but the stream()
does not!
Comment From: dev-jonghoonpark
LM Studio Local LLM API Server (OpenAI compatibility mode)
The link seems to be invalid, returning a 404 error.