Spring AI MessageChatMemoryAdvisor results in an error if prompt is initialized with List<Message> instead of userText

Bug description Create a prompt with

Prompt prompt = new Prompt(messages); // list of messages including UserMessage

And set MessageChatMemoryAdvisor as an advisor.

chatClient
  .prompt(prompt)
  .system(systemText)
  .advisors(new MessageChatMemoryAdvisor(chatMemory))

Results in an error that Content cannot be null because MessageChatMemoryAdvisor is looking for userText() which is not present but the prompt already has the UserMessage in list of messages.

Environment Spring AI 1.0.0-SNAPSHOT SpringBoot 3.4.2 Java 21

Comment From: dev-jonghoonpark

What chat model and chat memory are you using? Can you attach an error messages?

Comment From: rwankar

I'm using Azure OpenAI

Here is the stack trace..

java.lang.IllegalArgumentException: Content must not be null for SYSTEM or USER messages
    at org.springframework.util.Assert.notNull(Assert.java:181) ~[spring-core-6.2.2.jar!/:6.2.2]
    at org.springframework.ai.chat.messages.AbstractMessage.<init>(AbstractMessage.java:69) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.messages.UserMessage.<init>(UserMessage.java:62) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.messages.UserMessage.<init>(UserMessage.java:49) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor.before(MessageChatMemoryAdvisor.java:97) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor.aroundCall(MessageChatMemoryAdvisor.java:62) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundCall$1(DefaultAroundAdvisorChain.java:98) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at io.micrometer.observation.Observation.observe(Observation.java:564) ~[micrometer-observation-1.14.3.jar!/:1.14.3]
    at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundCall(DefaultAroundAdvisorChain.java:98) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at com.celoxis.psa.base.ai.HintChatMemoryAdvisor.aroundCall(HintChatMemoryAdvisor.java:47) ~[!/:14.5.0] 
    at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundCall$1(DefaultAroundAdvisorChain.java:98) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at io.micrometer.observation.Observation.observe(Observation.java:564) ~[micrometer-observation-1.14.3.jar!/:1.14.3] 
    at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundCall(DefaultAroundAdvisorChain.java:98) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.doGetChatResponse(DefaultChatClient.java:493) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT] 
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.lambda$doGetObservableChatResponse$1(DefaultChatClient.java:482) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at io.micrometer.observation.Observation.observe(Observation.java:564) ~[micrometer-observation-1.14.3.jar!/:1.14.3]
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.doGetObservableChatResponse(DefaultChatClient.java:482) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.doGetChatResponse(DefaultChatClient.java:466) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.chatResponse(DefaultChatClient.java:510) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]

Comment From: dev-jonghoonpark

ChatMemory chatMemory = new InMemoryChatMemory();
String content = ChatClient.builder(chatModel).build()
        .prompt("hi")
        .system("systemText")
        .advisors(new MessageChatMemoryAdvisor(chatMemory))
        .call()
        .content();
System.out.println(content);

I have tried testing in the same environment, but it is not reproduced.

Could you check if the prompt or systemText variable is null? The logs show the message: "Content must not be null for SYSTEM or USER messages."

Comment From: rwankar

If you create prompt from text it works. You need to create Prompt from a list of messages which contains a UserMessage. I've mentioned this in my first post.

The stack trace show an issue on line 97 in MessageChatMemoryAdvisor

UserMessage userMessage = new UserMessage(request.userText(), request.media());

It should probably create a UserMessage only if request.userText() is not null.

Comment From: dev-jonghoonpark

Prompt prompt = new Prompt(List.of(new UserMessage("hi")));

ChatMemory chatMemory = new InMemoryChatMemory();
String content = ChatClient.builder(chatModel).build()
        .prompt(prompt)
        .system("systemText")
        .advisors(new MessageChatMemoryAdvisor(chatMemory))
        .call()
        .content();
System.out.println(content);

This also works. It would be nice if you could provide a more specific example.

Comment From: rwankar

Here is a working example. It's a SpringBoot maven project. The example is a bit about getting the weather in a city. There are two tools provided. getZip() and getWeather(). To get the weather in a city, the LLM should first call the getZip tool to get the zip code and then use the zip code to call the getWeather() function.

I'm trying to get the ChatMemory updated with all the messages including the tool call and tool call result. So I'm following the example as suggested in the "User-controlled tool execution" in the Tools documentation.

There are 4 files. AiClient.java - responsible for building the ChatClient with the tools and the actual Chat. Application.java - The main SpringBoot app ChatController.java - The RestController with /chat GET endpoint. ApiKeys.java - reads the keys from Environment.

Set 2 environment variables azure.openai.key azure.openai.endpoint

Access localhost:8080/chat?q=what is the weather in Seattle?

On a side note, ToolCallingChatOptions.builder().toolCallbacks() expects List<FunctionCallback>. But FunctionCallback is deprecated. Should I be doing this a different way?

spring-ai-test.zip