bedrock_finetuning_experiments (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "af2ee7fb-e888-4e38-a349-c7c40dfd2963",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "# Evaluate context based RAG responses from Knowledge Bases, Iterate and Monitor with TruEra"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0679163a-387a-4ba6-8ce0-d5571614c0dc",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "SageMaker JumpStart provides a variety of pretrained open source and proprietary models such as Llama-2, Anthropic’s Claude and Cohere Command that can be quickly deployed in the Sagemaker environment. In many cases however, these foundation models are not sufficient on their own for production use cases, needing to be adapted to a particular style or new tasks. One way to surface this need is by evaluating the model against a curated ground truth dataset. Once the need to adapt the foundation model is clear, one could leverage a set of techniques to carry that out. A popular approach is to fine-tune the model on a dataset that is tailored to the use case.\n",
    "\n",
    "One challenge with this approach is that curated ground truth datasets are expensive to create. In this blog post, we address this challenge by augmenting this workflow with a framework for extensible, automated evaluations. We start off with a baseline foundation model from SageMaker JumpStart and evaluate it with TruLens, an open source library for evaluating & tracking LLM apps. Once we identify the need for adaptation, we can leverage fine-tuning in Sagemaker Jumpstart and confirm improvement with TruLens.\n",
    "\n",
    "TruLens evaluations make use of an abstraction of feedback functions. These functions can be implemented in several ways, including BERT-style models, appropriately prompted Large Language Models, and more. TruLens’ integration with AWS Bedrock allows you to easily run evaluations using LLMs available from AWS Bedrock. The reliability of Bedrock’s infrastructure is particularly valuable for use in performing evaluations across development and production.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "251624f9-1eb6-4051-a774-0a4ba83cabf5",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "---\n",
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK to deploy pre-trained Llama 2 model as well as fine-tune it for your dataset in domain adaptation or instruction tuning format. We will also use TruLens to identify performance issues with the base model and validate improvement of the fine-tuned model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "85addd9d-ec89-44a7-9fb5-9bc24fe9993b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ! pip install trulens_eval==0.18.3 sagemaker datasets boto3 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89cce506-1783-4175-a9c0-f5c8d35ad2bf",
   "metadata": {},
   "source": [
    "## Install the necessary packages for SDK set up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8071a2d3",
   "metadata": {
    "collapsed": true,
    "jumpStartAlterations": [
     "modelIdVersion"
    ],
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Requirement already satisfied: pip in /opt/conda/lib/python3.10/site-packages (23.3.2)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mLooking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Collecting boto3==1.0.0\n",
      "  Downloading boto3-1.0.0-py2.py3-none-any.whl (94 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m94.4/94.4 kB\u001b[0m \u001b[31m68.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting botocore==1.0.0 (from boto3==1.0.0)\n",
      "  Downloading botocore-1.0.0-py2.py3-none-any.whl (1.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m82.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting jmespath<1.0.0,>=0.6.2 (from boto3==1.0.0)\n",
      "  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)\n",
      "Collecting futures==2.2.0 (from boto3==1.0.0)\n",
      "  Downloading futures-2.2.0-py2.py3-none-any.whl (16 kB)\n",
      "Collecting jmespath<1.0.0,>=0.6.2 (from boto3==1.0.0)\n",
      "  Downloading jmespath-0.7.1-py2.py3-none-any.whl (19 kB)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.10/site-packages (from botocore==1.0.0->boto3==1.0.0) (2.8.2)\n",
      "Requirement already satisfied: docutils>=0.10 in /opt/conda/lib/python3.10/site-packages (from botocore==1.0.0->boto3==1.0.0) (0.16)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.0.0->boto3==1.0.0) (1.16.0)\n",
      "Installing collected packages: jmespath, futures, botocore, boto3\n",
      "  Attempting uninstall: jmespath\n",
      "    Found existing installation: jmespath 1.0.1\n",
      "    Uninstalling jmespath-1.0.1:\n",
      "      Successfully uninstalled jmespath-1.0.1\n",
      "  Attempting uninstall: botocore\n",
      "    Found existing installation: botocore 1.34.9\n",
      "    Uninstalling botocore-1.34.9:\n",
      "      Successfully uninstalled botocore-1.34.9\n",
      "  Attempting uninstall: boto3\n",
      "    Found existing installation: boto3 1.34.9\n",
      "    Uninstalling boto3-1.34.9:\n",
      "      Successfully uninstalled boto3-1.34.9\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "awscli 1.31.9 requires botocore==1.33.9, but you have botocore 1.0.0 which is incompatible.\n",
      "awscli 1.31.9 requires s3transfer<0.9.0,>=0.8.0, but you have s3transfer 0.10.0 which is incompatible.\n",
      "itemloaders 1.0.4 requires jmespath>=0.9.5, but you have jmespath 0.7.1 which is incompatible.\n",
      "s3fs 0.4.2 requires botocore>=1.12.91, but you have botocore 1.0.0 which is incompatible.\n",
      "s3transfer 0.10.0 requires botocore<2.0a.0,>=1.33.2, but you have botocore 1.0.0 which is incompatible.\n",
      "sagemaker 2.202.0 requires boto3<2.0,>=1.33.3, but you have boto3 1.0.0 which is incompatible.\n",
      "sagemaker 2.202.0 requires urllib3<1.27, but you have urllib3 2.1.0 which is incompatible.\n",
      "sagemaker-studio-analytics-extension 0.0.20 requires boto3<2.0,>=1.26.49, but you have boto3 1.0.0 which is incompatible.\n",
      "sagemaker-studio-sparkmagic-lib 0.1.4 requires boto3<2.0,>=1.10.44, but you have boto3 1.0.0 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed boto3-1.0.0 botocore-1.0.0 futures-2.2.0 jmespath-0.7.1\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m/bin/bash: line 1: =1.15.3: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade pip\n",
    "\n",
    "# Install Boto3 version 1.0 specifically\n",
    "!pip install boto3==1.0.0\n",
    "\n",
    "# Make sure Boto3 is no older than version 1.15.0\n",
    "!pip install boto3>=1.15.0\n",
    "\n",
    "# Avoid versions of Boto3 newer than version 1.15.3\n",
    "!pip install boto3<=1.15.3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8017c4ef-eb89-4da6-8e28-c800adbfc4b8",
   "metadata": {
    "tags": []
   },
   "source": [
    "## RetrieveAndGenerate VS. Retrieve API calls on KB\n",
    "\n",
    "---\n",
    "Next, we invoke the KB APIs for retrieve and generate and retrieve API calls to make sure which option is the best for our use case\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5d676e44-68c0-475e-bec8-f7020871a90e",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Requirement already satisfied: boto3 in /opt/conda/lib/python3.10/site-packages (1.0.0)\n",
      "Collecting boto3\n",
      "  Downloading boto3-1.34.11-py3-none-any.whl.metadata (6.6 kB)\n",
      "Requirement already satisfied: botocore in /opt/conda/lib/python3.10/site-packages (1.0.0)\n",
      "Collecting botocore\n",
      "  Downloading botocore-1.34.11-py3-none-any.whl.metadata (5.6 kB)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from boto3) (0.7.1)\n",
      "Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /opt/conda/lib/python3.10/site-packages (from boto3) (0.10.0)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.10/site-packages (from botocore) (2.8.2)\n",
      "Collecting urllib3<2.1,>=1.25.4 (from botocore)\n",
      "  Downloading urllib3-2.0.7-py3-none-any.whl.metadata (6.6 kB)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore) (1.16.0)\n",
      "Downloading boto3-1.34.11-py3-none-any.whl (139 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m139.3/139.3 kB\u001b[0m \u001b[31m47.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading botocore-1.34.11-py3-none-any.whl (11.9 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.9/11.9 MB\u001b[0m \u001b[31m161.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading urllib3-2.0.7-py3-none-any.whl (124 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.2/124.2 kB\u001b[0m \u001b[31m119.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: urllib3, botocore, boto3\n",
      "  Attempting uninstall: urllib3\n",
      "    Found existing installation: urllib3 2.1.0\n",
      "    Uninstalling urllib3-2.1.0:\n",
      "      Successfully uninstalled urllib3-2.1.0\n",
      "  Attempting uninstall: botocore\n",
      "    Found existing installation: botocore 1.0.0\n",
      "    Uninstalling botocore-1.0.0:\n",
      "      Successfully uninstalled botocore-1.0.0\n",
      "  Attempting uninstall: boto3\n",
      "    Found existing installation: boto3 1.0.0\n",
      "    Uninstalling boto3-1.0.0:\n",
      "      Successfully uninstalled boto3-1.0.0\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "awscli 1.31.9 requires botocore==1.33.9, but you have botocore 1.34.11 which is incompatible.\n",
      "awscli 1.31.9 requires s3transfer<0.9.0,>=0.8.0, but you have s3transfer 0.10.0 which is incompatible.\n",
      "distributed 2022.7.0 requires tornado<6.2,>=6.0.3, but you have tornado 6.4 which is incompatible.\n",
      "sagemaker 2.202.0 requires urllib3<1.27, but you have urllib3 2.0.7 which is incompatible.\n",
      "sparkmagic 0.20.4 requires nest-asyncio==1.5.5, but you have nest-asyncio 1.5.8 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed boto3-1.34.11 botocore-1.34.11 urllib3-2.0.7\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade boto3 botocore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b795a085-048f-42b2-945f-0cd339c1cf91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import pprint\n",
    "from botocore.client import Config\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=2)\n",
    "\n",
    "bedrock_config = Config(connect_timeout=120, read_timeout=120, retries={'max_attempts': 0})\n",
    "bedrock_client = boto3.client('bedrock-runtime')\n",
    "bedrock_agent_client = boto3.client(\"bedrock-agent-runtime\",\n",
    "                              config=bedrock_config)\n",
    "\n",
    "model_id = \"anthropic.claude-instant-v1\" # try with both claude instant as well as claude-v2. for claude v2 - \"anthropic.claude-v2\"\n",
    "region_id = \"us-east-1\" # replace it with the region you're running sagemaker notebook\n",
    "kb_id = \"UAZGA1FONQ\" # replace it with the Knowledge base id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5dd833f8-1ddc-4805-80b2-19e7db629880",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def retrieveAndGenerate(input, kbId, sessionId=None, model_id = \"anthropic.claude-v2\", region_id = \"us-east-1\"):\n",
    "    model_arn = f'arn:aws:bedrock:{region_id}::foundation-model/{model_id}'\n",
    "    if sessionId:\n",
    "        return bedrock_agent_client.retrieve_and_generate(\n",
    "            input={\n",
    "                'text': input\n",
    "            },\n",
    "            retrieveAndGenerateConfiguration={\n",
    "                'type': 'KNOWLEDGE_BASE',\n",
    "                'knowledgeBaseConfiguration': {\n",
    "                    'knowledgeBaseId': kbId,\n",
    "                    'modelArn': model_arn\n",
    "                }\n",
    "            },\n",
    "            sessionId=sessionId\n",
    "        )\n",
    "    else:\n",
    "        return bedrock_agent_client.retrieve_and_generate(\n",
    "            input={\n",
    "                'text': input\n",
    "            },\n",
    "            retrieveAndGenerateConfiguration={\n",
    "                'type': 'KNOWLEDGE_BASE',\n",
    "                'knowledgeBaseConfiguration': {\n",
    "                    'knowledgeBaseId': kbId,\n",
    "                    'modelArn': model_arn\n",
    "                }\n",
    "            }\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cec9b024-211a-4873-ade6-8db314034cd0",
   "metadata": {},
   "source": [
    "### Now,Retrieving and Generating Responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "30ecd15d-dfc5-4267-97e3-c1e04337e0fe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Amazon SageMaker is a fully managed service that enables developers and data '\n",
      " 'scientists to quickly and easily build, train, and deploy machine learning '\n",
      " 'models at any scale. SageMaker removes all the barriers that typically slow '\n",
      " 'down developers who want to use machine learning.')\n"
     ]
    }
   ],
   "source": [
    "query = \"What is the amazon sagemaker?\"\n",
    "response = retrieveAndGenerate(query, kb_id,model_id=model_id,region_id=region_id)\n",
    "generated_text = response['output']['text']\n",
    "pp.pprint(generated_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80dfabe1-9980-4778-b631-39ed18c2391a",
   "metadata": {},
   "source": [
    "## Now, let's set up several questions in a dictionary to evaluate the KB responses on RetrieveAndGenerate API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c2818b07-c55d-4983-8a5d-dd33411b877e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query: What is Amazon EC2?\n",
      "('Amazon EC2 is a web service that provides secure, resizable compute capacity '\n",
      " 'in the cloud. It allows users to obtain and configure computing resources '\n",
      " 'like server instances quickly and easily.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n",
      "Query: How does Amazon S3 work?\n",
      "('Amazon S3 is an object storage service that allows users to store and '\n",
      " 'protect any amount of data. Users can organize their data and control access '\n",
      " 'permissions based on their business and compliance needs. Data stored in '\n",
      " 'Amazon S3 has 99.999999999% durability.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n",
      "Query: What are the benefits of Amazon RDS?\n",
      "('Amazon RDS makes it easy to set up, operate, and scale a relational database '\n",
      " 'in the cloud. It provides cost-efficient and resizable capacity while '\n",
      " 'automating time- consuming administration tasks such as hardware '\n",
      " 'provisioning, database setup, patching and backups. It frees you to focus on '\n",
      " 'your applications so you can give them the fast performance, high '\n",
      " 'availability, security and compatibility they need.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n",
      "Query: Explain Amazon Lambda usage.\n",
      "('Amazon Lambda allows you to run code without provisioning or managing '\n",
      " 'servers. With Lambda, you can run code for virtually any type of application '\n",
      " 'or backend service—all with zero administration. You pay only for the '\n",
      " 'compute time you consume—there is no charge when your code is not running. '\n",
      " 'You can set up your code to automatically trigger from other AWS services, '\n",
      " 'or you can call it directly from any web or mobile app.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n",
      "Query: What is Amazon DynamoDB?\n",
      "('Amazon DynamoDB is a key-value and document database that delivers '\n",
      " \"single-digit millisecond performance at any scale. It's a fully managed, \"\n",
      " 'multiregion, multimaster database with built-in security, backup and '\n",
      " 'restore, and in-memory caching for internet-scale applications.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n",
      "Query: Describe Amazon VPC.\n",
      "('Amazon Virtual Private Cloud (Amazon VPC) allows you to launch AWS resources '\n",
      " \"like EC2 instances into a virtual network that you've defined. With Amazon \"\n",
      " 'VPC, you have complete control over your virtual networking environment, '\n",
      " 'including selection of your own IP address range, creation of subnets, and '\n",
      " 'configuration of route tables and network gateways. You can easily customize '\n",
      " 'the network configuration for your Amazon VPC.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n",
      "Query: How to use Amazon SES?\n",
      "('Amazon Simple Email Service (Amazon SES) is a cost-effective, flexible, and '\n",
      " 'scalable email service that enables developers to send mail from within any '\n",
      " 'application. You can configure Amazon SES quickly to support several email '\n",
      " 'use cases, including transactional, marketing, or mass email communications.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n",
      "Query: What is Amazon Redshift?\n",
      "('Amazon Redshift is a data warehouse service offered by Amazon Web Services. '\n",
      " 'It allows users to run complex analytic queries against large datasets '\n",
      " 'stored in its columnar data warehouses using massively parallel query '\n",
      " 'execution.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n",
      "Query: Explain the purpose of Amazon EKS.\n",
      "('Amazon EKS makes it easy to deploy, manage, and scale containerized '\n",
      " 'applications using Kubernetes on AWS. Amazon EKS runs the Kubernetes '\n",
      " 'management infrastructure for you across multiple AWS availability zones to '\n",
      " 'eliminate a single point of failure.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n",
      "Query: What features does Amazon CloudFront offer?\n",
      "('Amazon CloudFront is a content delivery network (CDN) service that securely '\n",
      " 'delivers data, videos, applications, and APIs to customers globally with low '\n",
      " 'latency, high transfer speeds. CloudFront works seamlessly with other AWS '\n",
      " 'services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load '\n",
      " 'Balancing or Amazon EC2 as origins for applications, and Lambda@Edge to run '\n",
      " 'custom code closer to users.')\n",
      "\n",
      "--------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "import pprint\n",
    "\n",
    "# Define your queries\n",
    "queries = [\n",
    "    \"What is Amazon EC2?\",\n",
    "    \"How does Amazon S3 work?\",\n",
    "    \"What are the benefits of Amazon RDS?\",\n",
    "    \"Explain Amazon Lambda usage.\",\n",
    "    \"What is Amazon DynamoDB?\",\n",
    "    \"Describe Amazon VPC.\",\n",
    "    \"How to use Amazon SES?\",\n",
    "    \"What is Amazon Redshift?\",\n",
    "    \"Explain the purpose of Amazon EKS.\",\n",
    "    \"What features does Amazon CloudFront offer?\"\n",
    "]\n",
    "\n",
    "# Initialize the PrettyPrinter\n",
    "pp = pprint.PrettyPrinter(indent=2)\n",
    "\n",
    "# Loop through each query and get responses\n",
    "responses = []\n",
    "for query in queries:\n",
    "    response = retrieveAndGenerate(query, kb_id, model_id=model_id, region_id=region_id)\n",
    "    generated_text = response['output']['text']\n",
    "    responses.append(generated_text)\n",
    "\n",
    "# Print the responses\n",
    "for i, response in enumerate(responses):\n",
    "    print(f\"Query: {queries[i]}\")\n",
    "    pp.pprint(response)\n",
    "    print(\"\\n\" + \"-\"*50 + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "196d7ac2-95bc-4d28-a763-1d45d36c14f1",
   "metadata": {},
   "source": [
    "### Set up as responses from the KB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fef5a7d4-4fc2-43f1-bec5-6836a4e767ef",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Collecting trulens_eval\n",
      "  Downloading trulens_eval-0.20.0-py3-none-any.whl.metadata (3.0 kB)\n",
      "Requirement already satisfied: numpy>=1.23.5 in /opt/conda/lib/python3.10/site-packages (from trulens_eval) (1.26.2)\n",
      "Collecting frozendict>=2.3.8 (from trulens_eval)\n",
      "  Downloading frozendict-2.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (23 kB)\n",
      "Collecting munch>=3.0.0 (from trulens_eval)\n",
      "  Downloading munch-4.0.0-py2.py3-none-any.whl.metadata (5.9 kB)\n",
      "Requirement already satisfied: dill>=0.3.7 in /opt/conda/lib/python3.10/site-packages (from trulens_eval) (0.3.7)\n",
      "Requirement already satisfied: python-dotenv>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from trulens_eval) (1.0.0)\n",
      "Requirement already satisfied: pydantic<3,>=2 in /opt/conda/lib/python3.10/site-packages (from trulens_eval) (2.5.3)\n",
      "Collecting merkle-json>=1.0.0 (from trulens_eval)\n",
      "  Downloading merkle_json-1.0.0-py3-none-any.whl (5.2 kB)\n",
      "Requirement already satisfied: langchain>=0.0.335 in /opt/conda/lib/python3.10/site-packages (from trulens_eval) (0.0.352)\n",
      "Requirement already satisfied: typing-inspect>=0.8.0 in /opt/conda/lib/python3.10/site-packages (from trulens_eval) (0.9.0)\n",
      "Requirement already satisfied: typing-extensions>=4.5.0 in /opt/conda/lib/python3.10/site-packages (from trulens_eval) (4.9.0)\n",
      "Collecting millify>=0.1.1 (from trulens_eval)\n",
      "  Downloading millify-0.1.1.tar.gz (1.2 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting humanize>=4.6.0 (from trulens_eval)\n",
      "  Downloading humanize-4.9.0-py3-none-any.whl.metadata (7.9 kB)\n",
      "Collecting streamlit>=1.27.0 (from trulens_eval)\n",
      "  Downloading streamlit-1.29.0-py2.py3-none-any.whl.metadata (8.2 kB)\n",
      "Collecting streamlit-aggrid>=0.3.4.post3 (from trulens_eval)\n",
      "  Downloading streamlit_aggrid-0.3.4.post3-py3-none-any.whl (3.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m125.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting streamlit-extras>=0.2.7 (from trulens_eval)\n",
      "  Downloading streamlit_extras-0.3.6-py3-none-any.whl.metadata (3.7 kB)\n",
      "Requirement already satisfied: sqlalchemy>=2.0.19 in /opt/conda/lib/python3.10/site-packages (from trulens_eval) (2.0.24)\n",
      "Collecting alembic>=1.11.2 (from trulens_eval)\n",
      "  Downloading alembic-1.13.1-py3-none-any.whl.metadata (7.4 kB)\n",
      "Collecting Mako (from alembic>=1.11.2->trulens_eval)\n",
      "  Downloading Mako-1.3.0-py3-none-any.whl.metadata (2.9 kB)\n",
      "Requirement already satisfied: PyYAML>=5.3 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (6.0.1)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (3.9.1)\n",
      "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (4.0.3)\n",
      "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (0.6.3)\n",
      "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (1.33)\n",
      "Requirement already satisfied: langchain-community<0.1,>=0.0.2 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (0.0.6)\n",
      "Requirement already satisfied: langchain-core<0.2,>=0.1 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (0.1.3)\n",
      "Requirement already satisfied: langsmith<0.1.0,>=0.0.70 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (0.0.75)\n",
      "Requirement already satisfied: requests<3,>=2 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (2.31.0)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /opt/conda/lib/python3.10/site-packages (from langchain>=0.0.335->trulens_eval) (8.2.3)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /opt/conda/lib/python3.10/site-packages (from pydantic<3,>=2->trulens_eval) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.14.6 in /opt/conda/lib/python3.10/site-packages (from pydantic<3,>=2->trulens_eval) (2.14.6)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in /opt/conda/lib/python3.10/site-packages (from sqlalchemy>=2.0.19->trulens_eval) (3.0.3)\n",
      "Collecting altair<6,>=4.0 (from streamlit>=1.27.0->trulens_eval)\n",
      "  Downloading altair-5.2.0-py3-none-any.whl.metadata (8.7 kB)\n",
      "Requirement already satisfied: blinker<2,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (1.7.0)\n",
      "Collecting cachetools<6,>=4.0 (from streamlit>=1.27.0->trulens_eval)\n",
      "  Downloading cachetools-5.3.2-py3-none-any.whl.metadata (5.2 kB)\n",
      "Requirement already satisfied: click<9,>=7.0 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (8.1.7)\n",
      "Requirement already satisfied: importlib-metadata<7,>=1.4 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (4.11.3)\n",
      "Requirement already satisfied: packaging<24,>=16.8 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (23.2)\n",
      "Requirement already satisfied: pandas<3,>=1.3.0 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (2.1.3)\n",
      "Requirement already satisfied: pillow<11,>=7.1.0 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (10.1.0)\n",
      "Requirement already satisfied: protobuf<5,>=3.20 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (4.25.1)\n",
      "Requirement already satisfied: pyarrow>=6.0 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (14.0.1)\n",
      "Requirement already satisfied: python-dateutil<3,>=2.7.3 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (2.8.2)\n",
      "Requirement already satisfied: rich<14,>=10.14.0 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (13.7.0)\n",
      "Requirement already satisfied: toml<2,>=0.10.1 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (0.10.2)\n",
      "Collecting tzlocal<6,>=1.1 (from streamlit>=1.27.0->trulens_eval)\n",
      "  Downloading tzlocal-5.2-py3-none-any.whl.metadata (7.8 kB)\n",
      "Collecting validators<1,>=0.2 (from streamlit>=1.27.0->trulens_eval)\n",
      "  Downloading validators-0.22.0-py3-none-any.whl.metadata (4.7 kB)\n",
      "Collecting gitpython!=3.1.19,<4,>=3.0.7 (from streamlit>=1.27.0->trulens_eval)\n",
      "  Downloading GitPython-3.1.40-py3-none-any.whl.metadata (12 kB)\n",
      "Collecting pydeck<1,>=0.8.0b4 (from streamlit>=1.27.0->trulens_eval)\n",
      "  Downloading pydeck-0.8.1b0-py2.py3-none-any.whl (4.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.8/4.8 MB\u001b[0m \u001b[31m98.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tornado<7,>=6.0.3 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (6.4)\n",
      "Requirement already satisfied: watchdog>=2.1.5 in /opt/conda/lib/python3.10/site-packages (from streamlit>=1.27.0->trulens_eval) (2.1.6)\n",
      "Collecting python-decouple<4.0,>=3.6 (from streamlit-aggrid>=0.3.4.post3->trulens_eval)\n",
      "  Downloading python_decouple-3.8-py3-none-any.whl (9.9 kB)\n",
      "Requirement already satisfied: entrypoints>=0.4 in /opt/conda/lib/python3.10/site-packages (from streamlit-extras>=0.2.7->trulens_eval) (0.4)\n",
      "Collecting htbuilder>=0.6.2 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading htbuilder-0.6.2-py3-none-any.whl.metadata (5.9 kB)\n",
      "Collecting markdownlit>=0.0.5 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading markdownlit-0.0.7-py3-none-any.whl (15 kB)\n",
      "Collecting st-annotated-text>=3.0.0 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading st_annotated_text-4.0.1-py3-none-any.whl.metadata (2.2 kB)\n",
      "Collecting streamlit-camera-input-live>=0.2.0 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading streamlit_camera_input_live-0.2.0-py3-none-any.whl (6.6 kB)\n",
      "Collecting streamlit-card>=0.0.4 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading streamlit_card-1.0.0-py3-none-any.whl.metadata (4.0 kB)\n",
      "Collecting streamlit-embedcode>=0.1.2 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading streamlit_embedcode-0.1.2-py3-none-any.whl (3.5 kB)\n",
      "Collecting streamlit-faker>=0.0.2 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading streamlit_faker-0.0.3-py3-none-any.whl.metadata (2.0 kB)\n",
      "Collecting streamlit-image-coordinates<0.2.0,>=0.1.1 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading streamlit_image_coordinates-0.1.6-py3-none-any.whl.metadata (2.0 kB)\n",
      "Collecting streamlit-keyup>=0.1.9 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading streamlit_keyup-0.2.2-py3-none-any.whl.metadata (2.0 kB)\n",
      "Collecting streamlit-toggle-switch>=1.0.2 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading streamlit_toggle_switch-1.0.2-py3-none-any.whl (635 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m635.4/635.4 kB\u001b[0m \u001b[31m267.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting streamlit-vertical-slider>=1.0.2 (from streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading streamlit_vertical_slider-2.5.5-py3-none-any.whl.metadata (2.2 kB)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/lib/python3.10/site-packages (from typing-inspect>=0.8.0->trulens_eval) (1.0.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.335->trulens_eval) (23.1.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.335->trulens_eval) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.335->trulens_eval) (1.9.4)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.335->trulens_eval) (1.4.1)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.335->trulens_eval) (1.3.1)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from altair<6,>=4.0->streamlit>=1.27.0->trulens_eval) (3.1.2)\n",
      "Requirement already satisfied: jsonschema>=3.0 in /opt/conda/lib/python3.10/site-packages (from altair<6,>=4.0->streamlit>=1.27.0->trulens_eval) (4.20.0)\n",
      "Requirement already satisfied: toolz in /opt/conda/lib/python3.10/site-packages (from altair<6,>=4.0->streamlit>=1.27.0->trulens_eval) (0.11.2)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /opt/conda/lib/python3.10/site-packages (from dataclasses-json<0.7,>=0.5.7->langchain>=0.0.335->trulens_eval) (3.20.1)\n",
      "Collecting gitdb<5,>=4.0.1 (from gitpython!=3.1.19,<4,>=3.0.7->streamlit>=1.27.0->trulens_eval)\n",
      "  Downloading gitdb-4.0.11-py3-none-any.whl.metadata (1.2 kB)\n",
      "Collecting more-itertools (from htbuilder>=0.6.2->streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading more_itertools-10.1.0-py3-none-any.whl.metadata (33 kB)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.10/site-packages (from importlib-metadata<7,>=1.4->streamlit>=1.27.0->trulens_eval) (3.8.0)\n",
      "Requirement already satisfied: jsonpointer>=1.9 in /opt/conda/lib/python3.10/site-packages (from jsonpatch<2.0,>=1.33->langchain>=0.0.335->trulens_eval) (2.4)\n",
      "Requirement already satisfied: anyio<5,>=3 in /opt/conda/lib/python3.10/site-packages (from langchain-core<0.2,>=0.1->langchain>=0.0.335->trulens_eval) (4.2.0)\n",
      "Requirement already satisfied: markdown in /opt/conda/lib/python3.10/site-packages (from markdownlit>=0.0.5->streamlit-extras>=0.2.7->trulens_eval) (3.3.4)\n",
      "Requirement already satisfied: lxml in /opt/conda/lib/python3.10/site-packages (from markdownlit>=0.0.5->streamlit-extras>=0.2.7->trulens_eval) (4.9.3)\n",
      "Collecting favicon (from markdownlit>=0.0.5->streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading favicon-0.7.0-py2.py3-none-any.whl (5.9 kB)\n",
      "Collecting pymdown-extensions (from markdownlit>=0.0.5->streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading pymdown_extensions-10.7-py3-none-any.whl.metadata (3.0 kB)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas<3,>=1.3.0->streamlit>=1.27.0->trulens_eval) (2022.1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas<3,>=1.3.0->streamlit>=1.27.0->trulens_eval) (2023.3)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil<3,>=2.7.3->streamlit>=1.27.0->trulens_eval) (1.16.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests<3,>=2->langchain>=0.0.335->trulens_eval) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests<3,>=2->langchain>=0.0.335->trulens_eval) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests<3,>=2->langchain>=0.0.335->trulens_eval) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests<3,>=2->langchain>=0.0.335->trulens_eval) (2023.11.17)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/lib/python3.10/site-packages (from rich<14,>=10.14.0->streamlit>=1.27.0->trulens_eval) (3.0.0)\n",
      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/lib/python3.10/site-packages (from rich<14,>=10.14.0->streamlit>=1.27.0->trulens_eval) (2.17.2)\n",
      "Collecting faker (from streamlit-faker>=0.0.2->streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading Faker-22.0.0-py3-none-any.whl.metadata (15 kB)\n",
      "Requirement already satisfied: matplotlib in /opt/conda/lib/python3.10/site-packages (from streamlit-faker>=0.0.2->streamlit-extras>=0.2.7->trulens_eval) (3.5.2)\n",
      "Requirement already satisfied: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.10/site-packages (from Mako->alembic>=1.11.2->trulens_eval) (2.1.3)\n",
      "Requirement already satisfied: sniffio>=1.1 in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1->langchain>=0.0.335->trulens_eval) (1.3.0)\n",
      "Requirement already satisfied: exceptiongroup>=1.0.2 in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1->langchain>=0.0.335->trulens_eval) (1.2.0)\n",
      "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython!=3.1.19,<4,>=3.0.7->streamlit>=1.27.0->trulens_eval)\n",
      "  Downloading smmap-5.0.1-py3-none-any.whl.metadata (4.3 kB)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit>=1.27.0->trulens_eval) (2023.11.2)\n",
      "Requirement already satisfied: referencing>=0.28.4 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit>=1.27.0->trulens_eval) (0.31.1)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit>=1.27.0->trulens_eval) (0.13.2)\n",
      "Requirement already satisfied: mdurl~=0.1 in /opt/conda/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich<14,>=10.14.0->streamlit>=1.27.0->trulens_eval) (0.1.2)\n",
      "Requirement already satisfied: beautifulsoup4>=4.7.0 in /opt/conda/lib/python3.10/site-packages (from favicon->markdownlit>=0.0.5->streamlit-extras>=0.2.7->trulens_eval) (4.12.2)\n",
      "Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.10/site-packages (from matplotlib->streamlit-faker>=0.0.2->streamlit-extras>=0.2.7->trulens_eval) (0.11.0)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib->streamlit-faker>=0.0.2->streamlit-extras>=0.2.7->trulens_eval) (4.25.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib->streamlit-faker>=0.0.2->streamlit-extras>=0.2.7->trulens_eval) (1.4.2)\n",
      "Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib->streamlit-faker>=0.0.2->streamlit-extras>=0.2.7->trulens_eval) (3.0.9)\n",
      "Collecting markdown (from markdownlit>=0.0.5->streamlit-extras>=0.2.7->trulens_eval)\n",
      "  Downloading Markdown-3.5.1-py3-none-any.whl.metadata (7.1 kB)\n",
      "Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.10/site-packages (from beautifulsoup4>=4.7.0->favicon->markdownlit>=0.0.5->streamlit-extras>=0.2.7->trulens_eval) (2.3.1)\n",
      "Downloading trulens_eval-0.20.0-py3-none-any.whl (634 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m634.0/634.0 kB\u001b[0m \u001b[31m307.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading alembic-1.13.1-py3-none-any.whl (233 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m233.4/233.4 kB\u001b[0m \u001b[31m288.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading frozendict-2.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (116 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.7/116.7 kB\u001b[0m \u001b[31m282.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading humanize-4.9.0-py3-none-any.whl (126 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m126.8/126.8 kB\u001b[0m \u001b[31m311.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB)\n",
      "Downloading streamlit-1.29.0-py2.py3-none-any.whl (8.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.4/8.4 MB\u001b[0m \u001b[31m197.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading streamlit_extras-0.3.6-py3-none-any.whl (66 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m66.0/66.0 kB\u001b[0m \u001b[31m270.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading altair-5.2.0-py3-none-any.whl (996 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m996.9/996.9 kB\u001b[0m \u001b[31m342.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading cachetools-5.3.2-py3-none-any.whl (9.3 kB)\n",
      "Downloading GitPython-3.1.40-py3-none-any.whl (190 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m190.6/190.6 kB\u001b[0m \u001b[31m311.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading htbuilder-0.6.2-py3-none-any.whl (12 kB)\n",
      "Downloading st_annotated_text-4.0.1-py3-none-any.whl (9.0 kB)\n",
      "Downloading streamlit_card-1.0.0-py3-none-any.whl (680 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m680.8/680.8 kB\u001b[0m \u001b[31m343.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading streamlit_faker-0.0.3-py3-none-any.whl (14 kB)\n",
      "Downloading streamlit_image_coordinates-0.1.6-py3-none-any.whl (6.3 kB)\n",
      "Downloading streamlit_keyup-0.2.2-py3-none-any.whl (7.4 kB)\n",
      "Downloading streamlit_vertical_slider-2.5.5-py3-none-any.whl (1.8 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m344.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading tzlocal-5.2-py3-none-any.whl (17 kB)\n",
      "Downloading validators-0.22.0-py3-none-any.whl (26 kB)\n",
      "Downloading Mako-1.3.0-py3-none-any.whl (78 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.6/78.6 kB\u001b[0m \u001b[31m167.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading gitdb-4.0.11-py3-none-any.whl (62 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m222.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading Faker-22.0.0-py3-none-any.whl (1.7 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m206.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading more_itertools-10.1.0-py3-none-any.whl (55 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.8/55.8 kB\u001b[0m \u001b[31m255.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading pymdown_extensions-10.7-py3-none-any.whl (250 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m250.8/250.8 kB\u001b[0m \u001b[31m340.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading Markdown-3.5.1-py3-none-any.whl (102 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m102.2/102.2 kB\u001b[0m \u001b[31m283.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading smmap-5.0.1-py3-none-any.whl (24 kB)\n",
      "Building wheels for collected packages: millify\n",
      "  Building wheel for millify (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for millify: filename=millify-0.1.1-py3-none-any.whl size=1845 sha256=2a719e54ada53d7f721c3fda97be277662fb7150c626e8d591275032f4a339c8\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-jznmok3m/wheels/67/8f/53/2759feac2e247ce89c1165c3ff12d484de7714a875ea3464f0\n",
      "Successfully built millify\n",
      "Installing collected packages: python-decouple, millify, validators, tzlocal, smmap, munch, more-itertools, merkle-json, markdown, Mako, humanize, frozendict, cachetools, pymdown-extensions, pydeck, htbuilder, gitdb, favicon, faker, alembic, st-annotated-text, gitpython, altair, streamlit, streamlit-vertical-slider, streamlit-toggle-switch, streamlit-keyup, streamlit-image-coordinates, streamlit-embedcode, streamlit-card, streamlit-camera-input-live, streamlit-aggrid, streamlit-faker, markdownlit, streamlit-extras, trulens_eval\n",
      "  Attempting uninstall: markdown\n",
      "    Found existing installation: Markdown 3.3.4\n",
      "    Uninstalling Markdown-3.3.4:\n",
      "      Successfully uninstalled Markdown-3.3.4\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "panel 0.13.1 requires bokeh<2.5.0,>=2.4.0, but you have bokeh 3.3.2 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed Mako-1.3.0 alembic-1.13.1 altair-5.2.0 cachetools-5.3.2 faker-22.0.0 favicon-0.7.0 frozendict-2.4.0 gitdb-4.0.11 gitpython-3.1.40 htbuilder-0.6.2 humanize-4.9.0 markdown-3.5.1 markdownlit-0.0.7 merkle-json-1.0.0 millify-0.1.1 more-itertools-10.1.0 munch-4.0.0 pydeck-0.8.1b0 pymdown-extensions-10.7 python-decouple-3.8 smmap-5.0.1 st-annotated-text-4.0.1 streamlit-1.29.0 streamlit-aggrid-0.3.4.post3 streamlit-camera-input-live-0.2.0 streamlit-card-1.0.0 streamlit-embedcode-0.1.2 streamlit-extras-0.3.6 streamlit-faker-0.0.3 streamlit-image-coordinates-0.1.6 streamlit-keyup-0.2.2 streamlit-toggle-switch-1.0.2 streamlit-vertical-slider-2.5.5 trulens_eval-0.20.0 tzlocal-5.2 validators-0.22.0\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install trulens_eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3b1681da-05bf-4dcb-9257-717c59267d75",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from trulens_eval.feedback import GroundTruthAgreement, Groundedness\n",
    "from trulens_eval import TruBasicApp, Feedback, Tru, Select\n",
    "from trulens_eval import Bedrock"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a73faa93-b289-4c6a-81a2-18ddeff6731c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Set up the retriever for context based responses\n",
    "def retrieve(query, kbId, numberOfResults=5):\n",
    "    return bedrock_agent_client.retrieve(\n",
    "        retrievalQuery= {\n",
    "            'text': query\n",
    "        },\n",
    "        knowledgeBaseId=kbId,\n",
    "        retrievalConfiguration= {\n",
    "            'vectorSearchConfiguration': {\n",
    "                'numberOfResults': numberOfResults\n",
    "            }\n",
    "        }\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2de67af5-5bcc-4838-b25e-70e2e60f5c6c",
   "metadata": {},
   "source": [
    "## Setting up the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "17fc96a1-001d-47c9-b3b1-6a32438b50cb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define your queries\n",
    "queries = [\n",
    "    \"What is Amazon EC2?\",\n",
    "    \"How does Amazon S3 work?\",\n",
    "    \"What are the benefits of Amazon RDS?\",\n",
    "    \"Explain Amazon Lambda usage.\",\n",
    "    \"What is Amazon DynamoDB?\",\n",
    "    \"Describe Amazon VPC.\",\n",
    "    \"How to use Amazon SES?\",\n",
    "    \"What is Amazon Redshift?\",\n",
    "    \"Explain the purpose of Amazon EKS.\",\n",
    "    \"What features does Amazon CloudFront offer?\"\n",
    "]\n",
    "\n",
    "# Your knowledge base ID and other configuration details\n",
    "kb_id = \"UAZGA1FONQ\"\n",
    "model_id = \"anthropic.claude-v2\"\n",
    "region_id = \"us-east-1\"\n",
    "\n",
    "# Sample test dataset preparation\n",
    "test_dataset = []\n",
    "\n",
    "for query in queries:\n",
    "    # Retrieve context for the query\n",
    "    context = retrieve(query, kb_id)  # Assuming this returns a string or similar\n",
    "    \n",
    "    # Retrieve and generate response for the query\n",
    "    response = retrieveAndGenerate(query, kb_id, model_id=model_id, region_id=region_id)\n",
    "    \n",
    "    # Assuming both functions return a string, append to the test dataset\n",
    "    test_dataset.append({\n",
    "        \"query\": query,\n",
    "        \"context\": context,\n",
    "        \"response\": response\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73e4e6ef-5d6d-4cfa-af6f-03ff9e5475db",
   "metadata": {},
   "source": [
    "## Run the tru eval open source evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "8f5b01be-7beb-4777-acfa-9f9cefb4a4e6",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query: What is Amazon EC2?\n",
      "Response: {'ResponseMetadata': {'RequestId': 'f4e5de82-a6f5-49b3-a1f0-84c19bb993b6', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:42:00 GMT', 'content-type': 'application/json', 'content-length': '5221', 'connection': 'keep-alive', 'x-amzn-requestid': 'f4e5de82-a6f5-49b3-a1f0-84c19bb993b6'}, 'RetryAttempts': 0}, 'sessionId': 'c04ed4ec-6efe-42cc-a0af-48d09306688f', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 194}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience.', 'span': {'start': 196, 'end': 376}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 378, 'end': 483}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n",
      "Query: How does Amazon S3 work?\n",
      "Response: {'ResponseMetadata': {'RequestId': '565326e2-b1a9-4611-8c43-9252f22128c5', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:42:06 GMT', 'content-type': 'application/json', 'content-length': '5207', 'connection': 'keep-alive', 'x-amzn-requestid': '565326e2-b1a9-4611-8c43-9252f22128c5'}, 'RetryAttempts': 0}, 'sessionId': '935ea5d6-c20c-405a-a7d0-0794a0de6d96', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 188}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience.', 'span': {'start': 190, 'end': 370}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 372, 'end': 476}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n",
      "Query: What are the benefits of Amazon RDS?\n",
      "Response: {'ResponseMetadata': {'RequestId': 'a151ed90-4d09-4468-bde1-52022121471b', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:42:14 GMT', 'content-type': 'application/json', 'content-length': '5063', 'connection': 'keep-alive', 'x-amzn-requestid': 'a151ed90-4d09-4468-bde1-52022121471b'}, 'RetryAttempts': 0}, 'sessionId': '2e36cd2e-f80a-4863-abca-aedcbf34a1ad', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield, Amazon S3, Elastic Load Balancing, and Lambda@Edge. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 194}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield, Amazon S3, Elastic Load Balancing, and Lambda@Edge.', 'span': {'start': 196, 'end': 298}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 300, 'end': 404}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n",
      "Query: Explain Amazon Lambda usage.\n",
      "Response: {'ResponseMetadata': {'RequestId': 'efc96e0b-0b61-4de4-8108-bd82c4917beb', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:42:19 GMT', 'content-type': 'application/json', 'content-length': '5051', 'connection': 'keep-alive', 'x-amzn-requestid': 'efc96e0b-0b61-4de4-8108-bd82c4917beb'}, 'RetryAttempts': 0}, 'sessionId': 'e6f0df8b-da08-41fa-baa8-fa46b416476e', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield, Amazon S3, Elastic Load Balancing, and Lambda@Edge. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 188}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield, Amazon S3, Elastic Load Balancing, and Lambda@Edge.', 'span': {'start': 190, 'end': 292}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 294, 'end': 398}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n",
      "Query: What is Amazon DynamoDB?\n",
      "Response: {'ResponseMetadata': {'RequestId': '13b270b8-4148-4385-80c2-d56f51a85455', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:42:28 GMT', 'content-type': 'application/json', 'content-length': '5209', 'connection': 'keep-alive', 'x-amzn-requestid': '13b270b8-4148-4385-80c2-d56f51a85455'}, 'RetryAttempts': 0}, 'sessionId': '1cffac18-7289-4f41-ac87-99740592cc1e', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 188}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience.', 'span': {'start': 190, 'end': 370}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 372, 'end': 477}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n",
      "Query: Describe Amazon VPC.\n",
      "Response: {'ResponseMetadata': {'RequestId': '42487841-83c6-48f5-8dfd-bac8223d4c1a', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:42:39 GMT', 'content-type': 'application/json', 'content-length': '5207', 'connection': 'keep-alive', 'x-amzn-requestid': '42487841-83c6-48f5-8dfd-bac8223d4c1a'}, 'RetryAttempts': 0}, 'sessionId': 'e746552c-ccd5-4f2f-b30b-483780eef7ab', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 188}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience.', 'span': {'start': 190, 'end': 370}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 372, 'end': 476}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n",
      "Query: How to use Amazon SES?\n",
      "Response: {'ResponseMetadata': {'RequestId': '3a5dcdce-92e9-4fd8-bdd2-d27ec0c149ba', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:42:55 GMT', 'content-type': 'application/json', 'content-length': '5207', 'connection': 'keep-alive', 'x-amzn-requestid': '3a5dcdce-92e9-4fd8-bdd2-d27ec0c149ba'}, 'RetryAttempts': 0}, 'sessionId': 'e02ae458-8c8d-4a94-88ed-52e01649ce9e', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 188}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience.', 'span': {'start': 190, 'end': 370}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 372, 'end': 476}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n",
      "Query: What is Amazon Redshift?\n",
      "Response: {'ResponseMetadata': {'RequestId': 'fdaeea73-dd90-4c50-83c7-d2e1c5d5f9bd', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:43:05 GMT', 'content-type': 'application/json', 'content-length': '5221', 'connection': 'keep-alive', 'x-amzn-requestid': 'fdaeea73-dd90-4c50-83c7-d2e1c5d5f9bd'}, 'RetryAttempts': 0}, 'sessionId': '53755a83-c2a0-4902-8118-dc11f24ff9d1', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 194}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience.', 'span': {'start': 196, 'end': 376}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 378, 'end': 483}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n",
      "Query: Explain the purpose of Amazon EKS.\n",
      "Response: {'ResponseMetadata': {'RequestId': '3012c0c1-9bb3-4df3-a01a-e12d8f6b3a0d', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:43:14 GMT', 'content-type': 'application/json', 'content-length': '5209', 'connection': 'keep-alive', 'x-amzn-requestid': '3012c0c1-9bb3-4df3-a01a-e12d8f6b3a0d'}, 'RetryAttempts': 0}, 'sessionId': '861ca7bd-008b-4d89-92e8-0180e994e405', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 188}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience.', 'span': {'start': 190, 'end': 370}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 372, 'end': 477}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n",
      "Query: What features does Amazon CloudFront offer?\n",
      "Response: {'ResponseMetadata': {'RequestId': '38923937-0361-4373-a35b-81585c0a4c14', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 02 Jan 2024 15:43:30 GMT', 'content-type': 'application/json', 'content-length': '5221', 'connection': 'keep-alive', 'x-amzn-requestid': '38923937-0361-4373-a35b-81585c0a4c14'}, 'RetryAttempts': 0}, 'sessionId': 'b8d878bc-bc6a-4a94-a41c-4c386005a3bd', 'output': {'text': 'Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.'}, 'citations': [{'generatedResponsePart': {'textResponsePart': {'text': 'Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds.', 'span': {'start': 0, 'end': 194}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience.', 'span': {'start': 196, 'end': 376}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}, {'generatedResponsePart': {'textResponsePart': {'text': 'CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.', 'span': {'start': 378, 'end': 483}}}, 'retrievedReferences': [{'content': {'text': \"Amazon CloudFront Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront is integrated with AWS – both physical locations that are directly connected to the AWS global infrastructure, as well as other AWS services. CloudFront works seamlessly with services including AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins for your applications, and Lambda@Edge to run custom code closer to customers’ users and to customize the user experience.   You can get started with the Content Delivery Network in minutes, using the same AWS tools that you're already familiar with: APIs, AWS Management Console, AWS CloudFormation, CLIs, and SDKs. Amazon's CDN offers a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts, and support for the CDN is included in your existing AWS Support subscription.   Amazon Route 53 Amazon Route 53  is a highly available and scalable cloud Domain Name System (DNS) web service.\"}, 'location': {'type': 'S3', 's3Location': {'uri': 's3://s3awsdoc/aws-overview.pdf'}}}]}]}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from trulens_eval.feedback import GroundTruthAgreement, Groundedness\n",
    "from trulens_eval import TruBasicApp, Feedback, Tru, Select, Bedrock\n",
    "\n",
    "def KB_responses(instruction, context):\n",
    "    input_output_demarkation_key = \"\\n\\n### Response:\\n\"\n",
    "    payload = {\n",
    "        \"inputs\": template[\"prompt\"].format(instruction=instruction, context=context) + input_output_demarkation_key,\n",
    "        \"parameters\": {\"max_new_tokens\": 200},\n",
    "    }\n",
    "    return retrieveAndGenerate(query, kb_id, model_id=model_id, region_id=region_id)\n",
    "\n",
    "# Prepare test dataset\n",
    "test_dataset = []\n",
    "\n",
    "for query in queries:\n",
    "    context = retrieve(query, kb_id)  # Retrieve context for the query\n",
    "    response = retrieveAndGenerate(query, kb_id, model_id=model_id, region_id=region_id)  # Generate response\n",
    "    test_dataset.append({\"query\": query, \"context\": context, \"response\": response})\n",
    "\n",
    "# Process the dataset using KB_responses\n",
    "for item in test_dataset:\n",
    "    kb_response = KB_responses(item[\"query\"], item[\"context\"])\n",
    "    print(f\"Query: {item['query']}\\nResponse: {kb_response}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "59fa4a36-29f9-4e15-a0b6-6ceae5019907",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query: What is Amazon EC2?\n",
      "Response: Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n",
      "Query: How does Amazon S3 work?\n",
      "Response: Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n",
      "Query: What are the benefits of Amazon RDS?\n",
      "Response: Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n",
      "Query: Explain Amazon Lambda usage.\n",
      "Response: Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n",
      "Query: What is Amazon DynamoDB?\n",
      "Response: Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield, Amazon S3, Elastic Load Balancing, and Lambda@Edge. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n",
      "Query: Describe Amazon VPC.\n",
      "Response: Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield, Amazon S3, Elastic Load Balancing, and Lambda@Edge. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n",
      "Query: How to use Amazon SES?\n",
      "Response: Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple, pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n",
      "Query: What is Amazon Redshift?\n",
      "Response: Amazon CloudFront offers a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield, Amazon S3, Elastic Load Balancing, and Lambda@Edge. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n",
      "Query: Explain the purpose of Amazon EKS.\n",
      "Response: Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n",
      "Query: What features does Amazon CloudFront offer?\n",
      "Response: Amazon CloudFront offers a fast content delivery network service that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It is integrated with AWS services like AWS Shield for DDoS mitigation, Amazon S3, Elastic Load Balancing or Amazon EC2 as origins, and Lambda@Edge to customize the user experience. CloudFront has a simple pay-as-you-go pricing model with no upfront fees or required long-term contracts.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for item in test_dataset:\n",
    "    kb_response = KB_responses(item[\"query\"], item[\"context\"])\n",
    "    \n",
    "    # Extract the main text response\n",
    "    main_text = kb_response['output']['text']\n",
    "    \n",
    "    print(f\"Query: {item['query']}\\nResponse: {main_text}\\n\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "0462ca53-a8af-416c-a8bf-2cc052f605ef",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from trulens_eval.feedback import GroundTruthAgreement, Groundedness\n",
    "from trulens_eval import TruBasicApp, Feedback, Tru, Select\n",
    "import boto3\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "ebeeeb5f-2506-4ccc-ad63-1307102bc4a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Rename columns\n",
    "test_dataset = pd.DataFrame(test_dataset)\n",
    "test_dataset.rename(columns={\"instruction\": \"query\"}, inplace=True)\n",
    "\n",
    "# Convert DataFrame to a list of dictionaries\n",
    "golden_set = test_dataset[[\"query\",\"response\"]].to_dict(orient='records')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "a5145155-c2ee-4196-b806-42aede2cad31",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "OpenAIError",
     "evalue": "The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOpenAIError\u001b[0m                               Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[51], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# Create a Feedback object for ground truth similarity\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m ground_truth \u001b[38;5;241m=\u001b[39m \u001b[43mGroundTruthAgreement\u001b[49m\u001b[43m(\u001b[49m\u001b[43mgolden_set\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;66;03m# Call the agreement measure on the instruction and output\u001b[39;00m\n\u001b[1;32m      4\u001b[0m f_groundtruth \u001b[38;5;241m=\u001b[39m (Feedback(ground_truth\u001b[38;5;241m.\u001b[39magreement_measure, name \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGround Truth Agreement\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m      5\u001b[0m                  \u001b[38;5;241m.\u001b[39mon(Select\u001b[38;5;241m.\u001b[39mRecord\u001b[38;5;241m.\u001b[39mcalls[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39margs\u001b[38;5;241m.\u001b[39margs[\u001b[38;5;241m0\u001b[39m])\n\u001b[1;32m      6\u001b[0m                  \u001b[38;5;241m.\u001b[39mon_output()\n\u001b[1;32m      7\u001b[0m                 )\n",
      "File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/groundtruth.py:75\u001b[0m, in \u001b[0;36mGroundTruthAgreement.__init__\u001b[0;34m(self, ground_truth, provider, bert_scorer)\u001b[0m\n\u001b[1;32m     48\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Measures Agreement against a Ground Truth. \u001b[39;00m\n\u001b[1;32m     49\u001b[0m \n\u001b[1;32m     50\u001b[0m \u001b[38;5;124;03mUsage 1:\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     72\u001b[0m \n\u001b[1;32m     73\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     74\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m provider:\n\u001b[0;32m---> 75\u001b[0m     provider \u001b[38;5;241m=\u001b[39m \u001b[43mOpenAI\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     76\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(ground_truth, List):\n\u001b[1;32m     77\u001b[0m     ground_truth_imp \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/openai.py:48\u001b[0m, in \u001b[0;36mOpenAI.__init__\u001b[0;34m(self, endpoint, model_engine, *args, **kwargs)\u001b[0m\n\u001b[1;32m     46\u001b[0m self_kwargs\u001b[38;5;241m.\u001b[39mupdate(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m     47\u001b[0m self_kwargs[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmodel_engine\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m model_engine\n\u001b[0;32m---> 48\u001b[0m self_kwargs[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mendpoint\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[43mOpenAIEndpoint\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     50\u001b[0m \u001b[38;5;28msuper\u001b[39m()\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__init__\u001b[39m(\n\u001b[1;32m     51\u001b[0m     \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mself_kwargs\n\u001b[1;32m     52\u001b[0m )\n",
      "File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/openai.py:317\u001b[0m, in \u001b[0;36mOpenAIEndpoint.__init__\u001b[0;34m(self, rpm, client, **kwargs)\u001b[0m\n\u001b[1;32m    313\u001b[0m     \u001b[38;5;28;01mdel\u001b[39;00m kwargs[CLASS_INFO]\n\u001b[1;32m    315\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m client \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    316\u001b[0m     \u001b[38;5;66;03m# Pass kwargs to client.\u001b[39;00m\n\u001b[0;32m--> 317\u001b[0m     client \u001b[38;5;241m=\u001b[39m \u001b[43moai\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mOpenAI\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    318\u001b[0m     self_kwargs[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mclient\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m OpenAIClient(client\u001b[38;5;241m=\u001b[39mclient)\n\u001b[1;32m    320\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n",
      "File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/openai/_client.py:92\u001b[0m, in \u001b[0;36mOpenAI.__init__\u001b[0;34m(self, api_key, organization, base_url, timeout, max_retries, default_headers, default_query, http_client, _strict_response_validation)\u001b[0m\n\u001b[1;32m     90\u001b[0m     api_key \u001b[38;5;241m=\u001b[39m os\u001b[38;5;241m.\u001b[39menviron\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mOPENAI_API_KEY\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m     91\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m api_key \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m---> 92\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m OpenAIError(\n\u001b[1;32m     93\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mThe api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m     94\u001b[0m     )\n\u001b[1;32m     95\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mapi_key \u001b[38;5;241m=\u001b[39m api_key\n\u001b[1;32m     97\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m organization \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
      "\u001b[0;31mOpenAIError\u001b[0m: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable"
     ]
    }
   ],
   "source": [
    "# Create a Feedback object for ground truth similarity\n",
    "ground_truth = GroundTruthAgreement(golden_set)\n",
    "# Call the agreement measure on the instruction and output\n",
    "f_groundtruth = (Feedback(ground_truth.agreement_measure, name = \"Ground Truth Agreement\")\n",
    "                 .on(Select.Record.calls[0].args.args[0])\n",
    "                 .on_output()\n",
    "                )\n",
    "\n",
    "# Instantiate Bedrock\n",
    "from trulens_eval import Bedrock\n",
    "\n",
    "# Initialize Bedrock as feedback function provider\n",
    "bedrock = Bedrock(model_id = \"amazon.titan-tg1-large\", region_name=\"us-east-1\")\n",
    "\n",
    "# Answer Relevance\n",
    "f_answer_relevance = (Feedback(bedrock.relevance_with_cot_reasons, name = \"Answer Relevance\")\n",
    "                      .on(Select.Record.calls[0].args.args[0])\n",
    "                      .on_output()\n",
    "                      )\n",
    "\n",
    "# Context Relevance\n",
    "f_context_relevance = (Feedback(bedrock.qs_relevance_with_cot_reasons, name = \"Context Relevance\")\n",
    "                       .on(Select.Record.calls[0].args.args[0])\n",
    "                       .on(Select.Record.calls[0].args.args[1])\n",
    "                      )\n",
    "\n",
    "# Groundedness\n",
    "grounded = Groundedness(groundedness_provider=bedrock)\n",
    "f_groundedness = (Feedback(grounded.groundedness_measure_with_cot_reasons, name = \"Groundedness\")\n",
    "                .on(Select.Record.calls[0].args.args[1])\n",
    "                .on_output()\n",
    "                .aggregate(grounded.grounded_statements_aggregator)\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "339cb7e9-fe41-4c83-8aa6-585ce9c23397",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "OpenAIError",
     "evalue": "The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOpenAIError\u001b[0m                               Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[52], line 7\u001b[0m\n\u001b[1;32m      4\u001b[0m golden_set \u001b[38;5;241m=\u001b[39m test_dataset[[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mquery\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mresponse\u001b[39m\u001b[38;5;124m\"\u001b[39m]]\u001b[38;5;241m.\u001b[39mto_dict(orient\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrecords\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m      6\u001b[0m \u001b[38;5;66;03m# Setup feedback mechanisms\u001b[39;00m\n\u001b[0;32m----> 7\u001b[0m ground_truth \u001b[38;5;241m=\u001b[39m \u001b[43mGroundTruthAgreement\u001b[49m\u001b[43m(\u001b[49m\u001b[43mgolden_set\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      8\u001b[0m f_groundtruth \u001b[38;5;241m=\u001b[39m Feedback(ground_truth\u001b[38;5;241m.\u001b[39magreement_measure, name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGround Truth Agreement\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mon(Select\u001b[38;5;241m.\u001b[39mRecord\u001b[38;5;241m.\u001b[39mcalls[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39margs\u001b[38;5;241m.\u001b[39margs[\u001b[38;5;241m0\u001b[39m])\u001b[38;5;241m.\u001b[39mon_output()\n\u001b[1;32m     10\u001b[0m bedrock \u001b[38;5;241m=\u001b[39m Bedrock(model_id\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mamazon.titan-tg1-large\u001b[39m\u001b[38;5;124m\"\u001b[39m, region_name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mus-east-1\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/groundtruth.py:75\u001b[0m, in \u001b[0;36mGroundTruthAgreement.__init__\u001b[0;34m(self, ground_truth, provider, bert_scorer)\u001b[0m\n\u001b[1;32m     48\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Measures Agreement against a Ground Truth. \u001b[39;00m\n\u001b[1;32m     49\u001b[0m \n\u001b[1;32m     50\u001b[0m \u001b[38;5;124;03mUsage 1:\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     72\u001b[0m \n\u001b[1;32m     73\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     74\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m provider:\n\u001b[0;32m---> 75\u001b[0m     provider \u001b[38;5;241m=\u001b[39m \u001b[43mOpenAI\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     76\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(ground_truth, List):\n\u001b[1;32m     77\u001b[0m     ground_truth_imp \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/openai.py:48\u001b[0m, in \u001b[0;36mOpenAI.__init__\u001b[0;34m(self, endpoint, model_engine, *args, **kwargs)\u001b[0m\n\u001b[1;32m     46\u001b[0m self_kwargs\u001b[38;5;241m.\u001b[39mupdate(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m     47\u001b[0m self_kwargs[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmodel_engine\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m model_engine\n\u001b[0;32m---> 48\u001b[0m self_kwargs[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mendpoint\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[43mOpenAIEndpoint\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     50\u001b[0m \u001b[38;5;28msuper\u001b[39m()\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__init__\u001b[39m(\n\u001b[1;32m     51\u001b[0m     \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mself_kwargs\n\u001b[1;32m     52\u001b[0m )\n",
      "File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/trulens_eval/feedback/provider/endpoint/openai.py:317\u001b[0m, in \u001b[0;36mOpenAIEndpoint.__init__\u001b[0;34m(self, rpm, client, **kwargs)\u001b[0m\n\u001b[1;32m    313\u001b[0m     \u001b[38;5;28;01mdel\u001b[39;00m kwargs[CLASS_INFO]\n\u001b[1;32m    315\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m client \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    316\u001b[0m     \u001b[38;5;66;03m# Pass kwargs to client.\u001b[39;00m\n\u001b[0;32m--> 317\u001b[0m     client \u001b[38;5;241m=\u001b[39m \u001b[43moai\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mOpenAI\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    318\u001b[0m     self_kwargs[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mclient\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m OpenAIClient(client\u001b[38;5;241m=\u001b[39mclient)\n\u001b[1;32m    320\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n",
      "File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/openai/_client.py:92\u001b[0m, in \u001b[0;36mOpenAI.__init__\u001b[0;34m(self, api_key, organization, base_url, timeout, max_retries, default_headers, default_query, http_client, _strict_response_validation)\u001b[0m\n\u001b[1;32m     90\u001b[0m     api_key \u001b[38;5;241m=\u001b[39m os\u001b[38;5;241m.\u001b[39menviron\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mOPENAI_API_KEY\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m     91\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m api_key \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m---> 92\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m OpenAIError(\n\u001b[1;32m     93\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mThe api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m     94\u001b[0m     )\n\u001b[1;32m     95\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mapi_key \u001b[38;5;241m=\u001b[39m api_key\n\u001b[1;32m     97\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m organization \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
      "\u001b[0;31mOpenAIError\u001b[0m: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "# Prepare the dataset\n",
    "test_dataset = pd.DataFrame(test_dataset)\n",
    "test_dataset.rename(columns={\"instruction\": \"query\"}, inplace=True)\n",
    "golden_set = test_dataset[[\"query\", \"response\"]].to_dict(orient='records')\n",
    "\n",
    "# Setup feedback mechanisms\n",
    "ground_truth = GroundTruthAgreement(golden_set)\n",
    "f_groundtruth = Feedback(ground_truth.agreement_measure, name=\"Ground Truth Agreement\").on(Select.Record.calls[0].args.args[0]).on_output()\n",
    "\n",
    "bedrock = Bedrock(model_id=\"amazon.titan-tg1-large\", region_name=\"us-east-1\")\n",
    "f_answer_relevance = Feedback(bedrock.relevance_with_cot_reasons, name=\"Answer Relevance\").on(Select.Record.calls[0].args.args[0]).on_output()\n",
    "f_context_relevance = Feedback(bedrock.qs_relevance_with_cot_reasons, name=\"Context Relevance\").on(Select.Record.calls[0].args.args[0]).on(Select.Record.calls[0].args.args[1])\n",
    "\n",
    "grounded = Groundedness(groundedness_provider=bedrock)\n",
    "f_groundedness = Feedback(grounded.groundedness_measure_with_cot_reasons, name=\"Groundedness\").on(Select.Record.calls[0].args.args[1]).on_output().aggregate(grounded.grounded_statements_aggregator)\n",
    "\n",
    "# Setup TruLens App\n",
    "finetuned_recorder = TruBasicApp(KB_responses, app_id=\"KB retrieveandgenerate API\", feedbacks=[f_groundtruth, f_answer_relevance, f_context_relevance, f_groundedness])\n",
    "\n",
    "# Evaluate the dataset\n",
    "for i in range(len(test_dataset)):\n",
    "    with finetuned_recorder as recording:\n",
    "        finetuned_recorder.app(test_dataset[\"query\"][i], test_dataset[\"context\"][i])\n",
    "\n",
    "# Retrieve and display results\n",
    "records, feedback = Tru().get_leaderboard(app_ids=[\"KB retrieveandgenerate API\"])\n",
    "print(records)\n",
    "print(feedback)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7bc2829-1fab-4d48-8ff9-73614013ea10",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c431629-6208-4a2e-88f8-e47be40375df",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}