Bug description call BedrockAnthropic return Exception
Exception in thread "main" software.amazon.awssdk.services.bedrockruntime.model.ValidationException: prompt must start with "
Human:" turn after an optional system prompt (Service: BedrockRuntime, Status Code: 400, Request ID: 8518716b-eab3-4015-a108-ad374e3e5e16)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:224)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
at software.amazon.awssdk.services.bedrockruntime.DefaultBedrockRuntimeClient.invokeModel(DefaultBedrockRuntimeClient.java:150)
at org.springframework.ai.bedrock.api.AbstractBedrockApi.internalInvocation(AbstractBedrockApi.java:202)
at org.springframework.ai.bedrock.anthropic.api.AnthropicChatBedrockApi.chatCompletion(AnthropicChatBedrockApi.java:220)
at org.springframework.ai.bedrock.anthropic.BedrockAnthropicChatClient.call(BedrockAnthropicChatClient.java:69)
at com.thclouds.ai.KendraTest.testSpringAi(KendraTest.java:130)
at com.thclouds.ai.KendraTest.main(KendraTest.java:37)
Environment windows 10 Spring AI 0.8.0, Java 17
Steps to reproduce
AnthropicChatBedrockApi bedrockApi = new AnthropicChatBedrockApi(
AnthropicChatBedrockApi.AnthropicChatModel.CLAUDE_V21.id(),
()-> AwsBasicCredentials.create(BedrockTest.ak, BedrockTest.sk),
Region.US_WEST_2.id(),
new ObjectMapper()
);
BedrockAnthropicChatClient chatClient = new BedrockAnthropicChatClient(bedrockApi,
AnthropicChatOptions.builder()
.withTemperature(0.6f)
.withTopK(10)
.withTopP(0.8f)
.withMaxTokensToSample(100)
.withAnthropicVersion(AnthropicChatBedrockApi.DEFAULT_ANTHROPIC_VERSION)
.build());
String queryText = "What's Amazon Kendra and how do i use it with a chat bot?";
ChatResponse rsp = getChatClient().call(new Prompt(queryText));
Comment From: noday
Comment From: tzolov
@noday, thank you for reaching out.
However I'm unable to reproduce the issue. I'am attaching a project with your snippet (different Region) that i've successfully tested against 0.8.0 and 0.8.1-SNAPSHOT.
Please give it a try and let me know: issue_404.zip
Also in the snippet you've shared above your are not using the chatClient you create there but something else that comes from getChatClient()
Comment From: noday
[issue_404.zip] still have same exception, I try the demo in centos7 and successfully. It's very likely an issue with Windows.
Comment From: tzolov
@noday thank you for the feedback. I don't have windows to test but I'm assuming that it has to do with the new line character used. So this commit https://github.com/spring-projects/spring-ai/commit/ba940395b6d38a39c6e1c8d37870b44d433465b9 applies a platform independent new line character. Can you please try again with the 0.8.1-SNAPSHOT version?
Comment From: noday
Comment From: noday
payload = "{\"prompt\":\"\r\n\r\nHuman: What's Amazon Kendra and how do i use it with a chat bot?\r\nAssistant:\", \"temperature\":0.6, \"max_tokens_to_sample\":100, \"top_k\":10, \"top_p\":0.8, \"anthropic_version\":\"bedrock-2023-05-31\"}"; use payload as body will reproduce the issue.
payload = "{\"prompt\":\"\\r\\n\\r\\nHuman: What's Amazon Kendra and how do i use it with a chat bot?\\r\\nAssistant:\", \"temperature\":0.6, \"max_tokens_to_sample\":100, \"top_k\":10, \"top_p\":0.8, \"anthropic_version\":\"bedrock-2023-05-31\"}";
InvokeModelRequest req = InvokeModelRequest.builder()
.modelId("anthropic.claude-v2:1")
.body(SdkBytes.fromUtf8String(payload))
.build();
BedrockRuntimeClient client = BedrockRuntimeClient.builder()
.credentialsProvider(()-> AwsBasicCredentials.create(BedrockTest.ak, BedrockTest.sk))
.region(Region.US_WEST_2)
.build();
InvokeModelResponse rsp = client.invokeModel(req);
String text = rsp.body().asUtf8String();
System.out.println(text);
Comment From: tzolov
Heh, "%n" is supposed to be platform neutral. Can you please replace it with this line and let me know :
final String prompt = systemMessages + System.lineSeparator() + System.lineSeparator() + userMessages + System.lineSeparator() + ASSISTANT_PROMPT;
Comment From: tzolov
btw i merged the above change so can you please run your tests again.
Comment From: tzolov
@noday i got access to Win machine to run some tests. Apparently either Anthropic or the Bedrock SDK messes up the line separator in case of Wind + Anthropic. It work find for all other models. I will push a patch in soon.
Comment From: noday
好的