If the message type used is UserMessage, it will not be built correctly
List<ChatCompletionMessage> chatCompletionMessages = prompt.getInstructions().stream().map(m -> {
// Add text content.
List<MediaContent> contents = new ArrayList<>(List.of(new MediaContent(m.getContent())));
if (m instanceof UserMessage userMessage) {
if (!CollectionUtils.isEmpty(userMessage.getMedia())) {
// Add media content.
contents.addAll(userMessage.getMedia()
.stream()
.map(media -> new MediaContent(
new MediaContent.ImageUrl(this.fromMediaData(media.getMimeType(), media.getData()))))
.toList());
}
}
return new ChatCompletionMessage(contents, ChatCompletionMessage.Role.valueOf(m.getMessageType().name()));
}).toList();
Here's my modified example
List<ChatCompletionMessage> chatCompletionMessages = prompt.getInstructions().stream().map(m -> {
if (m instanceof UserMessage userMessage) {
List<ChatCompletionMessage.MediaContent> contents = userMessage.getMedia().stream()
.flatMap(media -> Stream.of(
new ChatCompletionMessage.MediaContent(userMessage.getContent()),
new ChatCompletionMessage.MediaContent(
new MediaContent.ImageUrl(this.fromMediaData(media.getMimeType(), media.getData())))
))
.collect(Collectors.toList());
if (!contents.isEmpty()) {
return new ChatCompletionMessage(contents, ChatCompletionMessage.Role.valueOf(m.getMessageType().name()));
}
}
return new ChatCompletionMessage(m.getContent(), ChatCompletionMessage.Role.valueOf(m.getMessageType().name()));
}).collect(Collectors.toList());
Comment From: lwgCodePlus
@mxsl-gr
Comment From: magicgone-cn
I find that only model glm-4-alltools support the content structure below "content": [ { "type":"text", "text":"hello I'm Mike, 6 years old" }] .
for example, if I send a request like this, other models like glm-4-air will only say hello to me, not tell me correct age. if I use model glm-4-alltools, it will tell me "You will be 7 years old next year"
completion = client.chat.completions.create(
model="glm-4-alltools",
messages=[
{
"role": "user",
"content": [{
"type": "text",
"text": "I'm Mike, 6 years old"
}]
},
{
"role": "assistant",
"content": [{
"type": "text",
"text": "Hello Mike"
}]
},
{
"role": "user",
"content": [{
"type": "text",
"text": "Do you know how old am I next year"
}]
}
],
temperature=0.7,
stream=True
)
Comment From: mxsl-gr
@mxsl-gr
hi, @lwgCodePlus sorry, i went on a trip and just back now. but i didn't understand this problem. is there any example or error message?
Comment From: mxsl-gr
I find that only model glm-4-alltools support the content structure below我发现只有模型glm-4-alltools支持下面的内容结构 "content": [ { "type":"text", "text":"hello I'm Mike, 6 years old" }] .“content”： [ { “type”：“text”， “text”：“你好，我是Mike，6岁” }] .
for example, if I send a request like this, other models like glm-4-air will only say hello to me, not tell me correct age.例如，如果我发送这样的请求，其他型号（如GLM-4-Air）只会你好，不会告诉我正确的年龄。 if I use model glm-4-alltools, it will tell me "You will be 7 years old next year"如果我使用模型 glm-4-alltools，它会告诉我“明年你就 7 岁了”
python completion = client.chat.completions.create( model="glm-4-alltools", messages=[ { "role": "user", "content": [{ "type": "text", "text": "I'm Mike, 6 years old" }] }, { "role": "assistant", "content": [{ "type": "text", "text": "Hello Mike" }] }, { "role": "user", "content": [{ "type": "text", "text": "Do you know how old am I next year" }] } ], temperature=0.7, stream=True )
hi, @magicgone-cn , i tried and indeed there is a problem. use model glm-4-air and {content:{type:'', text:''}}, it seems that only the first user message is taking effect, and other messages are being ignored.
i checked the ZhiPu document, the messages parameter between glm-4 and glm-4-alltools is inconsistent, even though they are using seem the same api https://open.bigmodel.cn/api/paas/v4/chat/completions.
i think there is already a PR inadvertently fix it, using string content when user message is no media content.
Comment From: markpollack
Does the current PR fix the issues observed in this conversation?
https://github.com/spring-projects/spring-ai/pull/1188