introduction-to-bedrock/Getting_started_with_Converse_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting started with the Converse API in Amazon Bedrock\n",
    "\n",
    "In this notebook, we'll explore the basics of the Converse API in Amazon Bedrock. The Converse or ConverseStream API is a unified structured text API action that allows you simplifying the invocations to Bedrock LLMs, using a universal syntax and message structured prompts for any of the supported model providers.\n",
    "\n",
    "Let's start by installing or updating boto3. You just need to run this cell the first time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install -qU boto3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sys\n",
    "print('Running boto3 version:', boto3.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the region and models to use. We can also setup our boto3 client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region = 'us-east-1'\n",
    "print('Using region: ', region)\n",
    "\n",
    "bedrock = boto3.client(\n",
    "    service_name = 'bedrock-runtime',\n",
    "    region_name = region,\n",
    "    )\n",
    "\n",
    "MODEL_IDS = [\n",
    "    \"amazon.titan-text-premier-v1:0\",\n",
    "    \"amazon.titan-text-express-v1\",\n",
    "    \"amazon.titan-text-lite-v1\",\n",
    "    \"ai21.j2-ultra-v1\",\n",
    "    \"ai21.j2-mid-v1\",\n",
    "    \"anthropic.claude-3-sonnet-20240229-v1:0\",\n",
    "    \"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    \"cohere.command-r-plus-v1:0\",\n",
    "    \"cohere.command-r-v1:0\",\n",
    "    \"meta.llama3-70b-instruct-v1:0\",\n",
    "    \"meta.llama3-8b-instruct-v1:0\",\n",
    "    \"mistral.mistral-large-2402-v1:0\",\n",
    "    \"mistral.mixtral-8x7b-instruct-v0:1\",\n",
    "    \"mistral.mistral-7b-instruct-v0:2\",\n",
    "    \"mistral.mistral-small-2402-v1:0\"\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're now ready to setup our Converse API action in Bedrock. Note that we use the same syntax for any model, including the messages-formatted prompts, and the inference parameters. Also note that we read the output in the same way independently of the model used.\n",
    "\n",
    "Optionally, we could define additional model specific request fields that are not common across all providers. For more information on this check the Bedrock Converse API documentation.\n",
    "\n",
    "### Converse for one-shot invocations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def invoke_bedrock_model(client, id, prompt, max_tokens=2000, temperature=0, top_p=0.9):\n",
    "    response = \"\"\n",
    "    try:\n",
    "        response = client.converse(\n",
    "            modelId=id,\n",
    "            messages=[\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": [\n",
    "                        {\n",
    "                            \"text\": prompt\n",
    "                        }\n",
    "                    ]\n",
    "                }\n",
    "            ],\n",
    "            inferenceConfig={\n",
    "                \"temperature\": temperature,\n",
    "                \"maxTokens\": max_tokens,\n",
    "                \"topP\": top_p\n",
    "            }\n",
    "            #additionalModelRequestFields={\n",
    "            #}\n",
    "        )\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        result = \"Model invocation error\"\n",
    "    try:\n",
    "        result = response['output']['message']['content'][0]['text'] \\\n",
    "        + '\\n--- Latency: ' + str(response['metrics']['latencyMs']) \\\n",
    "        + 'ms - Input tokens:' + str(response['usage']['inputTokens']) \\\n",
    "        + ' - Output tokens:' + str(response['usage']['outputTokens']) + ' ---\\n'\n",
    "        return result\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        result = \"Output parsing error\"\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can test our invocation.\n",
    "\n",
    "In this example, we run the same prompt across all the text models supported in Bedrock by the time of writing this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = (\"What is the capital of Italy?\")\n",
    "print(f'Prompt: {prompt}\\n')\n",
    "\n",
    "for i in MODEL_IDS:\n",
    "    response = invoke_bedrock_model(bedrock, i, prompt)\n",
    "    print(f'Model: {i}\\n{response}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ConverseStream for streaming invocations\n",
    "\n",
    "We can also use the Converse API for streaming invocations. In this case we rely on the ConverseStream action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_IDS = [\n",
    "    \"amazon.titan-text-premier-v1:0\",\n",
    "    \"amazon.titan-text-express-v1\",\n",
    "    \"amazon.titan-text-lite-v1\",\n",
    "    \"anthropic.claude-3-sonnet-20240229-v1:0\",\n",
    "    \"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    \"cohere.command-r-plus-v1:0\",\n",
    "    \"cohere.command-r-v1:0\",\n",
    "    \"meta.llama3-70b-instruct-v1:0\",\n",
    "    \"meta.llama3-8b-instruct-v1:0\",\n",
    "    \"mistral.mistral-large-2402-v1:0\",\n",
    "    \"mistral.mixtral-8x7b-instruct-v0:1\",\n",
    "    \"mistral.mistral-7b-instruct-v0:2\",\n",
    "    \"mistral.mistral-small-2402-v1:0\"\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def invoke_bedrock_model_stream(client, id, prompt, max_tokens=2000, temperature=0, top_p=0.9):\n",
    "    response = \"\"\n",
    "    response = client.converse_stream(\n",
    "        modelId=id,\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": [\n",
    "                    {\n",
    "                        \"text\": prompt\n",
    "                    }\n",
    "                ]\n",
    "            }\n",
    "        ],\n",
    "        inferenceConfig={\n",
    "            \"temperature\": temperature,\n",
    "            \"maxTokens\": max_tokens,\n",
    "            \"topP\": top_p\n",
    "        }\n",
    "    )\n",
    "    # Extract and print the response text in real-time.\n",
    "    for event in response['stream']:\n",
    "        if 'contentBlockDelta' in event:\n",
    "            chunk = event['contentBlockDelta']\n",
    "            sys.stdout.write(chunk['delta']['text'])\n",
    "            sys.stdout.flush()\n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = (\"What is the capital of Italy?\")\n",
    "print(f'Prompt: {prompt}\\n')\n",
    "\n",
    "for i in MODEL_IDS:\n",
    "    print(f'\\n\\nModel: {i}')\n",
    "    invoke_bedrock_model_stream(bedrock, i, prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the Converse API allow us to easily run the invocations with the same syntax across all the models."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bedrock-with-unified-api",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}