[Enhancement]: Support AWS Bedrock custom inference profile #6710
Description
What features would you like to see added?
Currently, AWS bedrock allows you to create custom inference profiles for models. For example:
aws bedrock create-inference-profile --inference-profile-name "myappinferenceprofile" --model-source "copyFrom=arn:aws:bedrock:us-west-2:123456789123:inference-profile/us.anthropic.claude-3-7-sonnet-20250219-v1:0"
aws bedrock list-inference-profiles --type-equals "APPLICATION"
--This will list all application inference profiles showing the ARN, profile name and ID for each. Just an example, model names are different from real for Anthropic:
{
"inferenceProfileName": "ch-map-us-anthropic-claude-3-7-sonnet-20250219-v1-0",
"createdAt": "2025-04-01T12:00:46.784352+00:00",
"updatedAt": "2025-04-01T12:00:46.784352+00:00",
"inferenceProfileArn": "arn:aws:bedrock:us-east-1:123456789123:application-inference-profile/rf3zeruqfake",
"models": [
{
"modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-7-sonnet-20250219-v1:0"
},
{
"modelArn": "arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-3-7-sonnet-20250219-v1:0"
},
{
"modelArn": "arn:aws:bedrock:us-west-2::foundation-model/anthropic.claude-3-7-sonnet-20250219-v1:0"
}
],
"inferenceProfileId": "rf3zeruqfake",
"status": "ACTIVE",
"type": "APPLICATION"
}
Once the profile is created, to use it in LibreChat you need to add the new profile arn to the list of models:
BEDROCK_AWS_MODELS=arn:aws:bedrock:us-east-1:123456789123:application-inference-profile/rf3zeruqfake
But Librechat performs a check against model name:
if (/claude-3[-.]7/.test(model)) {...
and enables different parameters (thinking, temperature, topP and topK, etc) based on this check. You cannot control ARN name, and you cannot use profile name instead. All in all, this leads to a model that you cannot use in agents because of no settings are available:
More details
Provided above
Which components are impacted by your request?
Endpoints
Pictures
Provided above
Code of Conduct
- I agree to follow this project's Code of Conduct