This change introduces a new field for tracking cached tokens in the OpenAI API response. It extends the Usage record to include PromptTokensDetails, allowing for more granular token usage reporting. The OpenAiUsage class is updated to expose this new data, and corresponding unit tests are added to verify the behavior.
This enhancement provides more detailed insights into token usage, indicating how many of the prompt tokens were a cache hit.
Fixes https://github.com/spring-projects/spring-ai/issues/1506
Comment From: tzolov
Thank you for the update and the enhancement. would be reviewed shortly.
Comment From: tzolov
LGTM
Comment From: tzolov
rebased, added author docs, squashed and merged at 2c17577f2fb9dfd84c17da7801b0ed9d47c9939c