Spring-ai Call method from OpenAiChatModel Class does not work for media generated from PNG images.

Hello!

When I use the "call" method from OpenAiChatModel Class (injected via the default constructor to @Autowired), passing as parameters a Prompt object instatied with a Usermessage with: - String textContent; and - Media media (generated from a PNG image)

don't get a result.

Like this:

@Autowired private OpenAiChatModel chatModel;

...

public String myFunction(String textContent, File file) { Media media = new Media(MimeTypeUtils.IMAGE_PNG, new FileSystemResource(file)); Prompt prompt = new Prompt(new UserMessage(textContent, media)); chatModel.call(prompt); .... }

When I make this call, I don't get a result with a PNG image, but with a JPG image, it works well.

I also tested with the "stream" method from the same Class and it works well with all types of images.

Additionally: If I manually instantiate the OpenAiChatModel by setting the RestClient.Builder of the OpenAiApi as: RestClient.builder(new RestTemplate()), then everything works well for the images.

Comment From: dev-jonghoonpark

@flplvr-magie Hi, in my case, it works well. Could you please provide a more specific example?

Comment From: flplvr-magie

@dev-jonghoonpark I think I found what is causing the problem. It is not directly related to the image type, JPG or PNG.

However, it is related to the size. I noticed that both PNG and JPG images larger than ~ 700KB have an issue with the call() method of the OpenAiChatModel class. I send the request and do not receive a response, but the stream() method apparently works well.

I imagine this might be some restriction intentionally or unintentionally included in the RestClient.Builder used in the OpenAiApi class, but not in the WebClient.Builder.

So when I set the RestClient.Builder as RestClient.builder(new RestTemplate()), perhaps I bypass this image size restriction.