Hey team, First of all, thanks for the effort you are doing for this amazing project. I would like to ask for the support of a recently and very important addition in AWS Bedrock, that's the cross-region inference support. You can find all the tech information in AWS doc: https://aws.amazon.com/blogs/machine-learning/getting-started-with-cross-region-inference-in-amazon-bedrock/
Thanks in advance
Comment From: dafriz
You should be able to use withought any code changes by adding a "us." or "eu." prefix to the modelId.
Some extra enums could be added but wondering if the extra code is useful enough to add - something like this
/**
* Cross Region Inference mapping
* https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference-support.html
*/
public enum CrossRegion {
US("us", Set.of(Region.US_EAST_1, Region.US_WEST_2)),
EU("eu", Set.of(Region.EU_CENTRAL_1, Region.EU_WEST_1, Region.EU_WEST_3));
private final String prefix;
private final Set<Region> regions;
public String prefix() {
return prefix;
}
public Set<Region> regions() {
return regions;
}
CrossRegion(String prefix, Set<Region> regions) {
this.prefix = prefix;
this.regions = regions;
}
}
and then adding more enums for Llama and Anthropic like below
/**
* Cross Region Inference Llama models version.
* https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference-support.html
*/
public enum CrossRegionLlamaChatModel implements ChatModelDescription {
US_LLAMA3_2_1B_INSTRUCT_V1(CrossRegion.US, LlamaChatModel.LLAMA3_2_1B_INSTRUCT_V1),
US_LLAMA3_2_3B_INSTRUCT_V1(CrossRegion.US, LlamaChatModel.LLAMA3_2_3B_INSTRUCT_V1),
US_LLAMA3_2_11B_INSTRUCT_V1(CrossRegion.US, LlamaChatModel.LLAMA3_2_11B_INSTRUCT_V1),
US_LLAMA3_2_90B_INSTRUCT_V1(CrossRegion.US, LlamaChatModel.LLAMA3_2_90B_INSTRUCT_V1),
EU_LLAMA3_2_1B_INSTRUCT_V1(CrossRegion.EU, LlamaChatModel.LLAMA3_2_1B_INSTRUCT_V1),
EU_LLAMA3_2_3B_INSTRUCT_V1(CrossRegion.EU, LlamaChatModel.LLAMA3_2_3B_INSTRUCT_V1);
private final String id;
CrossRegionLlamaChatModel(CrossRegion crossRegion, LlamaChatModel model) {
this.id = crossRegion.prefix() + '.' + model.id;
}
public String id() {
return id;
}
@Override
public String getName() {
return this.id;
}
}
Thoughs on what support would be helpful ? Do we want to validate if someone uses a cross region inference modelId but then uses a region that's not supported in that cross-region ? ( That's where the mapping enum might be useful ).
Comment From: DEG-7
I agree, it sounds good to me. As you mentioned, it's not absolutely essential, and we could simply add the prefix 'us.' or 'eu.' to the model ID. However, having a specific enum for cross-region models (for those that support it) would be nicer and a better approach than just adding a string prefix.