New Models/Endpoints/Providers

[krrishdholakia]

Parent issue to track new models/endpoints/providers to add to litellm, comment below for new ones

• Vertex AI Mistral - [Feature]: Add vertexAI/mistral-large-2 #4874
• Vertex AI Codestral support - [Feature]: Add codestral on vertex support #4984 s/o @allen-cook
• Azure Batches API support - [Feature]: Support upstream batching on Azure OpenAI #5073
• [Feature]: LiteLLM SDK - Add support for Google AI Studio context caching  #5213 Add Gemini Context Caching in Anthropic Format
• Gemini Context Caching - Gemini API: Context Caching #4284
• Reka AI - [Feature]: Support for Reka AI  #3607
• Use Bedrock Guardrails [Feature]: Bedrock's guardrails with LiteLLM  #4938
• Vertex AI Multimodal embeddings - [Feature]: Support Vertex Multimodal embeddings #4622
• Fake streaming json mode for groq models - [Bug]: groq models do not support streaming when in JSON mode #4804
• Amazon Multimodal embedding models - [Feature]: Support new Amazon Multimodal Embedding model  #4353
• Support Bedrock document understanding
• Support using s3 buckets for gemini vision calls (requires uploading files to gcs for larger files)
• Prefill standardization - [Feature]: Unified API for prefill/prefix to stuff the start of the assistant's chat response #4881
• Add OpenAI fine tuning endpoints https://platform.openai.com/docs/api-reference/fine-tuning #4952
• Add Azure OpenAI fine tuning endpoints #4953
• Add Vertex AI fine tuning endpoints #4954
• Add Fireworks ai fine tuning endpoints #4955
• [Feature]: GET /v1/batches to return list of batches #4966
• Azure gpt-4o-mini support - s/o @seonghobae
• Support deepseek call caching - s/o @arvinxx #5017
• Natively support pass-through endpoints for vertex ai
• Github provider support - 4258295, s/o @lazyhope
• Support for amazon bedrock agents #5053

[guleng]

@krrishdholakia I have deployed the text graph model locally, but I am unable to configure it for use in Litellm. Besides Azure and other vendors' locally deployed text graph models, cannot they be used in Litellm?

Can you help me with how to configure it?

[seonghobae]

Please checkout Azure GPT-4o-mini ;) https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/openai-s-gpt-4o-mini-now-available-in-api-with-vision/ba-p/4200640

[guleng]

@seonghobae Does it not support locally deployed Wenshengtu models? Does it only support models from Azure and OpenAI manufacturers?

[krrishdholakia]

@guleng - your error message from the provider shows a 404 not found error.

• Is your provider openai-compatible? if so use this - https://docs.litellm.ai/docs/providers/openai_compatible
• If custom api, use this - https://docs.litellm.ai/docs/providers/custom_llm_server

If error continues, file a separate issue for this - with config.yaml, steps to repro, and error traceback received.

[lazyhope]

Github just released a new inference endpoint: https://models.inference.ai.azure.com, it has free tier for popular models like gpt-4o so would be worth adding to litellm.

Marketplace: https://github.com/marketplace/models

[krrishdholakia]

added @lazyhope

[lazyhope]

added @lazyhope

Awesome! Thank you!

[yigitkonur]

Hey Llama 3.1 - 405B is added to AWS Bedrock - model ID is meta.llama3-1-405b-instruct-v1:0

I appreciate if you add support and update docs. Thanks!

[krrishdholakia]

it's already supported @yigitkonur

litellm/model_prices_and_context_window.json

Line 4725 in 9a5115d

"meta.llama3-1-405b-instruct-v1:0": {

litellm/litellm/llms/bedrock/chat/converse_handler.py

Line 43 in 9a5115d

"meta.llama3-1-405b-instruct-v1:0",

[simonhir]

Is there already a way to add embedding models via azure_ai? In specific cohere.embed-multilingual-v3.0
Couldn't find anything in the documentation or code and the cohere provider seams to not allow setting the api_base.

[krrishdholakia]

cohere.embed-multilingual-v3.0

that model name looks like the bedrock one @simonhir

Re: azure ai embedding - not yet, can you create an issue, i'll add it tomorrow

[simonhir]

@krrishdholakia thanks for your fast answer.
I created #5861. Hope that helps.

[AdithyanI]

@krrishdholakia Google just upgraded the gemini-1.5-pro. There is a new model gemini-1.5-pro-002 in GA. Also the flash. Can we support this please? We are specifically looking for in vertex_ai support. Thank you :)

https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemini-1.5-pro-002?hl=en&authuser=1&rapt=AEjHL4O-gl6qi8hhL9DBsAA1wb_wVoUUEkXTkJeARCSgbGPb03zHz3AwlLM98ApP59b3JpuRF68lhNwehNMo6lq0CFkU4lc4AyjAXa-eVDAGfubP1PeTUP4&project=win-compute-407516

[jbellis]

Support for gemini embedding models, please. (text-embedding-004 is free!)

[ClancyDennis]

Are there any plans to add the /image/edits endpoint currently supported by Dalle-2?

[krrishdholakia]

Google just upgraded the gemini-1.5-pro. There is a new model gemini-1.5-pro-002 in GA. Also the flash. Can we support this please?

done -

litellm/model_prices_and_context_window.json

Line 2032 in e19bb55

"gemini-1.5-pro-002": {

Support for gemini embedding models, please. (text-embedding-004 is free!)

@jbellis can you create an issue for this? it would help to know if you're v0 is for google ai studio or vertex ai

Are there any plans to add the /image/edits endpoint currently supported by Dalle-2?

@ClancyDennis didn't realize that, please create an issue, so we can track this.

[FireballDWF]

Related to Add Amazon Titan Text Premier #3774

[igorlima]

Hey @jbellis and @krrishdholakia!

• Support for gemini embedding models, please. (text-embedding-004 is free!) ⁺

• can you create an issue for this? ⁺

Great news! litellm already has built-in support for Gemini embeddings, though it wasn't highlighted in the documentation until now. 🎉

I've just proposed that with the PR below, which adds Gemini to the documentation page's list of supported embeddings.

• Add Gemini Embeddings Support

code snippet

Here's a quick and easy code snippet to show you how to use Gemini embeddings with litellm:

import os
os.environ["GEMINI_API_KEY"] = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

from litellm import embedding
response = embedding(
  model="gemini/text-embedding-004",
  input=["good morning from litellm"],
)
print(response)

[KamilFatkhiev]

Hi @krrishdholakia, are you considering adding support for yandexgpt and gigachat family models?

[krrishdholakia]

Hey @KamilFatkhiev the api's look openai compatible - https://developers.sber.ru/docs/ru/gigachat/api/reference/rest/post-chat

so should already work - https://docs.litellm.ai/docs/providers/openai_compatible

[KamilFatkhiev]

Hey @KamilFatkhiev the api's look openai compatible - https://developers.sber.ru/docs/ru/gigachat/api/reference/rest/post-chat

so should already work - https://docs.litellm.ai/docs/providers/openai_compatible

Oh, cool! What about yandexgpt? Are the differences too big?
https://yandex.cloud/en-ru/docs/foundation-models/text-generation/api-ref/TextGeneration/completion

[krrishdholakia]

@KamilFatkhiev i don't have access to it - but if there's a mock endpoint we can use for testing it should be possible to add it