walkthroughs/OpenAI-EDA/BBC_OpenAI_EDA.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
        "\n",
        "Licensed under the MIT License.\n",
        "\n",
        "# OpenAI and Clustering Demo in Azure Synapse/Trident using the BBC Sports Dataset\n",
        "\n",
        "# To run this notebook, ensure you have run the deployment script in the Data Discovery Toolkit - [here](https://github.com/microsoft/Data-Discovery-Toolkit#if-you-do-not-have-a-synapse-workspace)\n",
        "\n",
        "## This notebook will showcase the following:\n",
        "\n",
        "- Extracting OpenAI embeddings against a dataframe\n",
        "- Clustering the dataset using KMEANS against OpenAI vectors\n",
        "- PCA Dimensionality Reduction for PowerBI Scatterplot visualisation against OpenAI vectors\n",
        "- Using OpenAI to automatically label the clusters in a dataset based on most common themes\n",
        "- OpenAI Retrieval Augmented Generation Pattern using Azure Cognitive Search and standard lucene keyword search\n",
        "- OpenAI Retrieval Augmented Generation Pattern using Azure Cognitive Search and Semantic Search\n",
        "- OpenAI Retrieval Augmented Generation Pattern using Azure Cognitive Search and Azure Synapse/Trident SQL\n",
        "- Search Result evaluation using Jaccard distance measure\n",
        "- Search Result evaluation using Manhattan distance measure\n",
        "- Search Result evaluation using Jaccard distance measure\n",
        "- Search Result evaluation using Euclidean distance measure\n",
        "- Search Result evaluation using Chebysev distance measure\n",
        "- Search Result evaluation using Cosine distance measure using spaCy vectors\n",
        "- Search Result evaluation using spaCy similarity\n",
        "- Search Result evaluation using roBERTa natural language entailment\n",
        "- Search Result evaluation using Jaccard distance measure\n",
        "- Search Result evaluation using the Spearman Ranking Correlation Coeeficient against spaCy vectors\n",
        "- Search Result evaluation using the Weighted Pearson Correlation Coefficient against spaCy vectors\n",
        "- Search Result evaluation using Levenshtein distance distance measure\n",
        "- Find the most similar content to any input data in the dataset using OpenAI vectors\n",
        "- Build a Knowledge Graph from the text in the dataset for PowerBI Network Navigator visualisation\n",
        "- Show Knowledge Graph Connectivity Degrees\n",
        "- Show Knowledge Graph Page Rank\n",
        "\n",
        "- TODO - Generate SQL using Codex\n",
        "- TODO - Run evaluation against OpenAI vectors\n",
        "\n"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "## This cell configures the spark session - Do not change (not needed for Trident)"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%configure -f\n",
        "{\n",
        "\"conf\": {\n",
        "     \"spark.rpc.message.maxSize\": 1024,\n",
        "     \"spark.kryoserializer.buffer.max\": \"1024m\"\n",
        "   }\n",
        "}"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": null,
              "session_id": "30",
              "statement_id": -1,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T13:11:30.0751689Z",
              "session_start_time": "2023-05-14T13:11:30.2049273Z",
              "execution_start_time": "2023-05-14T13:15:42.1765539Z",
              "execution_finish_time": "2023-05-14T13:15:42.1768371Z",
              "spark_jobs": null,
              "parent_msg_id": "56ccd12f-c3b0-433b-aba6-b5fc6ac058e0"
            },
            "text/plain": "StatementMeta(, 30, -1, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 45,
      "metadata": {
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "## These are the parameters that need to be changed to your values"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from pyspark.ml.feature import HashingTF, IDF, Tokenizer, CountVectorizer, StopWordsRemover, PCA, RegexTokenizer\n",
        "from pyspark.ml.clustering import LDA, KMeans, BisectingKMeans\n",
        "from pyspark.ml import Pipeline\n",
        "from pyspark import SparkContext, SparkConf\n",
        "import sys\n",
        "from pyspark.sql.functions import udf\n",
        "from pyspark.sql.types import StringType\n",
        "from pyspark.sql import SparkSession\n",
        "import ntpath\n",
        "import os\n",
        "import numpy as np\n",
        "import openai\n",
        "from math import sqrt\n",
        "from scipy.stats import spearmanr\n",
        "import torch\n",
        "import torch.nn.functional as F\n",
        "\n",
        "\n",
        "from pyspark.ml.linalg import Vectors, VectorUDT\n",
        "from pyspark.sql.functions import udf,col\n",
        "\n",
        "from itertools import combinations\n",
        "from operator import itemgetter\n",
        "\n",
        "from graphframes import *\n",
        "from pyspark.sql.functions import monotonically_increasing_id, lit\n",
        "\n",
        "\n",
        "import os\n",
        "import openai\n",
        "from azure.identity import DefaultAzureCredential\n",
        "from azure.search.documents import SearchClient\n",
        "from azure.search.documents.models import QueryType\n",
        "from azure.core.credentials import AzureKeyCredential\n",
        "\n",
        "import spacy\n",
        "import torch\n",
        "from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
        "from fuzzywuzzy import fuzz\n",
        "from IPython.display import display, Markdown\n",
        "\n",
        "# This will need to have been installed via the deployment script in the Data Discovery Toolkit\n",
        "nlp = spacy.load('en_core_web_lg')\n",
        "\n",
        "nli_model = AutoModelForSequenceClassification.from_pretrained('facebook/bart-large-mnli')\n",
        "tokenizer = AutoTokenizer.from_pretrained('facebook/bart-large-mnli')\n",
        "\n",
        "nli_weights = sc.broadcast(nli_model)\n",
        "\n",
        "\n",
        "# OpenAI deployment values - Add your values\n",
        "openai_chatgpt_deployment = \"chat\" # Currently a gpt-35-turbo model\n",
        "openai_gpt_deployment = \"davinci\" # Currently a text-davinci-003 model\n",
        "openai_service = \"\"\n",
        "openai_api_key = \"\"\n",
        "openai_api_type = \"azure\"\n",
        "openai_api_base = f\"https://{openai_service}.openai.azure.com\"\n",
        "openai_api_version = \"2022-12-01\"\n",
        "openai_embeddings_deployment = \"\" # Currently a text-embedding-ada-002 model\n",
        "openai_code_deployment = \"\" # Currently a code-davinci-002 model\n",
        "\n",
        "# If you have a RPM rate limit on your OpenAI account - sleep time in seconds\n",
        "openai_sleep_time = 80\n",
        "\n",
        "# The input file name - change to your BBC CSV file\n",
        "input_filename = 'abfss://share@datadiscoverypipeline2.dfs.core.windows.net/bbcsports/csv/sport_articles.csv'\n",
        "\n",
        "# The number of clusters - this can be automated or start with a guesstimate - set to 5 for BBC dataset\n",
        "number_of_clusters = 5\n",
        "\n",
        "# Concept Graph - # Get top N most connected nodes   \n",
        "number_of_connected_nodes = 5\n",
        "\n",
        "\n",
        "SCATTER_PLOT_3D = False\n"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "tags": []
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Distance Measures for Evaluation"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Jaccard Distance\n",
        "def get_jaccard_sim(str1, str2): \n",
        "    a = set(str1.split()) \n",
        "    b = set(str2.split())\n",
        "    c = a.intersection(b)\n",
        "    return float(len(c)) / (len(a) + len(b) - len(c))\n",
        "\n",
        "# Euclidean Distance\n",
        "def euclidean(x, y):\n",
        "    distance = 0\n",
        "    for a, b in zip(x, y):\n",
        "        distance += (sum([(pow((a-b),2))]))\n",
        "    return sqrt(distance)\n",
        "\n",
        "# Manhattan distance\n",
        "def manhattan(x, y):\n",
        "    distance=0\n",
        "    for a,b in zip(x,y):\n",
        "        distance += sum([abs(a-b)])\n",
        "    return distance\n",
        "\n",
        "#chebyshev distance\n",
        "def chebyshev(x,y):\n",
        "    distance = []\n",
        "    for a,b in zip(x,y):\n",
        "        distance.append(abs(a-b))\n",
        "    return max(distance)\n",
        "\n",
        "# Weighted Pearson Correlation Coefficient\n",
        "def m(x, w):\n",
        "    \"\"\"Weighted Mean - Weighted Pearson Correlation Coefficient\"\"\"\n",
        "    return np.sum(x * w) / np.sum(w)\n",
        "\n",
        "def cov(x, y, w):\n",
        "    \"\"\"Weighted Covariance - Weighted Pearson Correlation Coefficient\"\"\"\n",
        "    return np.sum(w * (x - m(x, w)) * (y - m(y, w))) / np.sum(w)\n",
        "\n",
        "def corr(x, y, w):\n",
        "    \"\"\"Weighted Correlation - Weighted Pearson Correlation Coefficient\"\"\"\n",
        "    return cov(x, y, w) / np.sqrt(cov(x, x, w) * cov(y, y, w))\n",
        "\n",
        "def spacy_tokenizer(sentence):\n",
        "    return [word.lemma_ for word in nlp(sentence) if not (word.like_num or word.is_stop or word.is_punct or word.is_space)] # remove numbers "
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "30",
              "statement_id": 3,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T13:17:09.5581614Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T13:18:36.1249555Z",
              "execution_finish_time": "2023-05-14T13:18:45.290036Z",
              "spark_jobs": null,
              "parent_msg_id": "4defe17a-4cfd-4b67-b3fd-2e4c20183d29"
            },
            "text/plain": "StatementMeta(DataDiscovery, 30, 3, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 47,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Optional: Clustering - Apply Data Preprocessing, OpenAI Embeddings, Kmeans clustering and PCA dimensionality reduction"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "df = spark.read.load(input_filename, header=True, format='csv')\n",
        "\n",
        "def openai_embeddings(text):\n",
        "\n",
        "    from time import sleep\n",
        "    sleep(openai_sleep_time) # Avoid rate limit if needed\n",
        "    openai.api_key = openai_api_key\n",
        "    openai.api_type = openai_api_type\n",
        "    openai.api_base = openai_api_base\n",
        "    openai.api_version = openai_api_version\n",
        "    deployment_id = openai_embeddings_deployment\n",
        "\n",
        "    embeddings = openai.Embedding.create(deployment_id=deployment_id,\n",
        "                                     input=text[:2000]) # Take first 2000 characters or calculate tokens\n",
        "    return Vectors.dense(embeddings['data'][0]['embedding'])\n",
        "\n",
        "# Vectorise processed text to OpenAI features\n",
        "udf_openai_vectoriser = udf(openai_embeddings, VectorUDT()) \n",
        "df = df.withColumn(\"OpenAI_features\", udf_openai_vectoriser(col(\"text\")))\n",
        "\n",
        "# Apply PCA and Kmeans in a pipeline\n",
        "pca = PCA(k=20, inputCol=\"OpenAI_features\")\n",
        "pca.setOutputCol(\"features\")\n",
        "\n",
        "if SCATTER_PLOT_3D:\n",
        "  pca_2 = PCA(k=3, inputCol=\"features\")\n",
        "else:  \n",
        "  pca_2 = PCA(k=2, inputCol=\"features\")\n",
        "  \n",
        "pca_2.setOutputCol(\"pca_scatterplot_features\")\n",
        "\n",
        "kmeans = KMeans(k=number_of_clusters, seed=42, initMode=\"k-means||\", distanceMeasure=\"euclidean\")\n",
        "\n",
        "pipeline = Pipeline(stages=[ pca, kmeans, pca_2])\n",
        "model = pipeline.fit(df)\n",
        "df_coords = model.transform(df)\n",
        "# Let's write the data to a SQL Table\n",
        "df_coords.write.saveAsTable(\"openai_bbc_coords_vectors\")"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "26",
              "statement_id": 3,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-13T20:30:36.3496782Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-13T20:35:29.3894045Z",
              "execution_finish_time": "2023-05-13T20:35:56.9246972Z",
              "spark_jobs": null,
              "parent_msg_id": "d3ab738b-6827-432d-ae19-e5c32abb9ae9"
            },
            "text/plain": "StatementMeta(DataDiscovery, 26, 3, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 36,
      "metadata": {
        "jupyter": {
          "source_hidden": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## If clustering is not run with vectors then load the data directly to a SQL table"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = spark.read.load(input_filename, header=True, format='csv')\n",
        "try:\n",
        "    df.write.saveAsTable(\"openai_bbc_coords2\")\n",
        "except Exception as TableAlreadyExists:\n",
        "    print(TableAlreadyExists)\n",
        "df = spark.sql(\"SELECT * FROM openai_bbc_coords2 LIMIT 1000\")\n",
        "display(df)"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "30",
              "statement_id": 4,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T13:17:16.2893739Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T13:18:45.4564759Z",
              "execution_finish_time": "2023-05-14T13:19:21.269795Z",
              "spark_jobs": null,
              "parent_msg_id": "02783dd9-2780-4e3d-817a-a0922f1d60b1"
            },
            "text/plain": "StatementMeta(DataDiscovery, 30, 4, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Table `openai_bbc_coords2` already exists.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "DataFrame[_c0: string, filename: string, text: string]"
          },
          "metadata": {}
        }
      ],
      "execution_count": 48,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Run a test SQL query"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "term_to_search_for = \"thumb\"\n",
        "df = spark.sql(f\"SELECT * FROM openai_bbc_coords2 where text like '%{term_to_search_for}%'\")\n",
        "display(df.show())"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "30",
              "statement_id": 5,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T13:17:34.8955383Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T13:19:21.4527792Z",
              "execution_finish_time": "2023-05-14T13:19:39.026269Z",
              "spark_jobs": null,
              "parent_msg_id": "44a835c3-3688-411d-8309-d33ba5e603eb"
            },
            "text/plain": "StatementMeta(DataDiscovery, 30, 5, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---+--------------+--------------------+\n|_c0|      filename|                text|\n+---+--------------+--------------------+\n|215|cricket093.txt|vaughan endures d...|\n|303|  rugby059.txt|\"robinson out of ...|\n|543|  rugby130.txt|\"celts savour gra...|\n|605|  rugby008.txt|\"thomas out of si...|\n+---+--------------+--------------------+\n\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "None"
          },
          "metadata": {}
        }
      ],
      "execution_count": 49,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Optional: Populate these values if using Azure Cognitive Search"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Add Search Parameters"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Azure Search Admin Key\n",
        "search_admin_key = \"\"\n",
        "# The name of the search service\n",
        "search_service_name = \"\"\n",
        "# The Azure Search Query Key\n",
        "search_query_key = \"\"\n",
        "search_index = \"bbc-index\"\n",
        "# This is the name of the semantic configuration on the search index\n",
        "semantic_configuration_name = \"config\"\n",
        "\n",
        "search_client = SearchClient(\n",
        "    endpoint=f\"https://{search_service_name}.search.windows.net\",\n",
        "    index_name=search_index,\n",
        "    credential=AzureKeyCredential(search_admin_key))"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "30",
              "statement_id": 6,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T13:19:44.54258Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T13:19:44.7094483Z",
              "execution_finish_time": "2023-05-14T13:19:44.8682858Z",
              "spark_jobs": null,
              "parent_msg_id": "9725046d-1034-4daf-b02d-2b93400308d9"
            },
            "text/plain": "StatementMeta(DataDiscovery, 30, 6, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 50,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## This code creates an Azure Cognitive Search Index "
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from synapse.ml.cognitive import *\n",
        "from pyspark.sql.functions import monotonically_increasing_id, lit\n",
        "\n",
        "df = df.drop(\"_c0\")\n",
        "\n",
        "(\n",
        "    df.withColumn(\"key\", monotonically_increasing_id().cast(\"string\"))\n",
        "    .withColumn(\"SearchAction\", lit(\"upload\"))\n",
        "    .writeToAzureSearch(\n",
        "        subscriptionKey=search_admin_key,\n",
        "        actionCol=\"SearchAction\",\n",
        "        serviceName=search_service_name,\n",
        "        indexName=search_index,  # Defaults to the notebook name\n",
        "        keyCol=\"key\",\n",
        "    )\n",
        ")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Optional: Test the Azure Cognitive Search Query"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "r = search_client.search(\"Whose thumb was fractured?\", top=5)\n",
        "\n",
        "scores = {}\n",
        "\n",
        "results = [doc[\"content\"].replace(\"\\n\", \"\").replace(\"\\r\", \"\") for doc in r]\n",
        "for i, doc in enumerate(results):\n",
        "    print(doc)"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "27",
              "statement_id": 13,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T09:08:06.855302Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T09:08:07.0113554Z",
              "execution_finish_time": "2023-05-14T09:08:07.5536455Z",
              "spark_jobs": null,
              "parent_msg_id": "daf9f5a0-0dbb-46bf-aa31-791008ca07d8"
            },
            "text/plain": "StatementMeta(DataDiscovery, 27, 13, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Thomas out of Six NationsWales captain Gareth Thomas has been ruled out of the rest of the Six Nations with a broken thumb.The full-back will have surgery on Monday after fracturing his thumb in the 24-18 win over France on Saturday. But Welsh legend Phil Bennett insisted Wales can cope without Thomas as they chase a first Grand Slam in 27 years. Bennett told BBC Sport: \"Such is the spirit in the camp, they'll put Kevin Morgan at 15, Rhys Williams at wing and just carry on.\" Thomas will miss the match against Scotland on 13 March, and what promises to be a huge encounter against the Irish six days later. Bennett added: \"It's a setback. He's a great captain, he leads from the front and the boys love him.\" Thomas was replaced at half-time by Williams as his side turned around a 15-6 deficit in Paris.\"With Gareth missing I would think Michael Owen will be our captain,\" said Wales coach Mike Ruddock. \"He did a great job in the second half in France. He has been vice-captain all along throughout the championship.\" Wales travel to Edinburgh to take on Scotland in a fortnight and then host Ireland in Cardiff in the final round of matches in what could be the Grand Slam and championship decider. Bennett, an inspirational fly-half for Llanelli and Wales in the 1970s, insisted the national team were entering a new golden period. \"It was a great game and a magnificent result for Wales,\" Bennett told BBC Radio Five Live's Sportsweek programme.\"The way this young team are blending, the glory days are on their way back. \"We couldn't get possession early on and France dominated and scored two tries. \"Had they been ruthless, Wales could have gone into the interval 30 points down. But they didn't take their chances. \"Wales defended fairly well but you cannot give that sort of quality ball to good sides. \"The All Blacks would have been ruthless and buried us in the first half. But the character we showed in the second half was quite outstanding.\"\nKenyon denies Robben Barca returnChelsea chief executive Peter Kenyon has played down reports that Arjen Robben will return for the Champions League match against Barcelona.\"He's been responding well to treatment and started running on Friday, but we'll have to wait and see,\" he told BBC Five Live's Sportsweek. \"We're looking to getting him back as soon as possible, but he'll be back when it's right for him and for us. \"There's no plans at the moment around the Barcelona game.\" His comments contradict those of chiropractor Jean Pierre Meersseman who treated the Dutchman after he fractured his foot at the start of February. Robben had been expected to be out for six weeks, but Meersseman hinted that the winger could be fit for the vital Stamford Bridge game on 8 March. \"I hope he can be back and I will try to help him make that happen,\" Meersseman told the Mail on Sunday. \"I put everything right with Arjen's foot the last time I saw him 12 days ago. It was an obvious correction and easy to perform. \"I know he was pleased with what I did and now that he is running again. I am due to see him one more time again in the next few days.\" Meersseman is the medical co-ordinator at Italian side AC Milan.\nVickery upbeat about arm injuryEngland prop Phil Vickery is staying positive despite a broken arm ruling him out of the RBS Six Nations.The 28-year-old fractured the radius in his right forearm during Gloucester's 17-16 win over Bath on Saturday. He will undergo an operation on Monday and is expected to be out for at least six weeks. He said: \"This isn't an injury that will stop me from working hard on the fitness elements and being around the lads.\" He added: \"I've got the operation this afternoon and I could be back doing fitness work after a week.\" \"As frustrating as it is, I've got to be positive.\"After the game, Vickery spoke with Bath prop David Barnes, who also broke his arm recently. \"I had a chat with David Barnes and it looks like a similar injury to him,\" he said. \"He said he had the operation and he was back running after a week. \"There's no doubt that I'm going to get involved and be around this place as soon as I can after the operation.\" Gloucester director of rugby Nigel Melville said: \"Phil has broken his radius, which is the large bone in his forearm. \"I don't really know how it happened, but Phil will definitely be out of action for at least six weeks. \"I feel very sorry for him, as he has been in great shape. He really needed 80 minutes of rugby this weekend, and then this happened. Mentally, it must be very hard for him.\"\nCole refuses to blame van PersieAshley Cole has refused to blame Robin van Persie for leaving Arsenal with no fully-fit strikers for the FA Cup fifth round replay at Sheffield United.Van Persie is suspended alongside Dennis Bergkamp and Jose Antonio Reyes after being sent off at Southampton when Arsenal had a numerical advantage. Thierry Henry is ruled out with an Achilles tendon injury but Cole said: \"No-one is putting the blame on Robin. \"It's just something that happens on the spur of the moment.\" Cole added: \"I've done it before and I hope they didn't blame me for anything. \"Of course he'll learn. I've been sent off a couple of times now and it's just one of those things when you go a bit crazy for one or two seconds. Freddie Ljungberg is likely to be used in an emergency striking role and will be partnered by either Arturo Lupoli, Quincy Owusu-Abeyie or Jeremie Aliadiere. Gunners boss Arsene Wenger said: \"Freddie is an option but we need a second striker. \"I have to decide whether it will be Aliadiere, Quincy or Lupoli who will start with him up front. Those three will be involved.\" Arsenal are also without winger Robert Pires, who sustained an ankle injury at St Mary's. Wenger added: \"It doesn't look like anything is fractured, but it is a good ankle sprain. \"It does not look like Pires will be ready for two to three weeks.\"\nWales get Williams fitness boostWales are hopeful that openside flanker Martyn Williams could be fit for Saturday's RBS 6 Nations championship opener against England in Cardiff.Williams was expected to miss the match with a disc problem in his neck, but has been making a speedy recovery. \"He will have tests in the next 48 hours and we are pretty optimistic he is getting there,\" Wales' team physiotherapist Mark Davies said. \"It has been frustrating but he is on the mend, he has made good progress.\" Last week Williams, along with fellow flanker Colin Charvis - who is unlikely to play for at least a month while he recovers from a foot injury - was all but ruled out of the Millennium Stadium clash. With Williams initially thought to be struggling, the signs pointed towards Wales coach Mike Ruddock handing a first cap to former Wales Under-21 skipper Richie Pugh.Cardiff Blues flanker Williams, 29, offers considerable experience and if he is declared fit then Ruddock might be tempted to include him in the back row. Charvis will be reviewed by the Wales medical staff next Monday, but Davies admitted that there was only an \"outside chance\" of him being fit to face France in Wales' third championship game on 26 February. Wales' other injury concern is Pugh's fellow Neath-Swansea Ospreys player Sonny Parker, as the centre has a trapped nerve in his neck. \"Sonny's injury is still an issue,\" Davies said. \"It is still painful and irritable. We will run the rule of thumb over him in the next couple of days.\" Ruddock will name his starting line-up for the England game at 1830 GMT on Tuesday evening, as Wales target their first victory in Cardiff over the world champions since 1993.\n"
          ]
        }
      ],
      "execution_count": 12,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Search the generated Azure Search Index via standard search and run evaluation"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "term_to_search_for = \"Whose thumb was fractured?\"\n",
        "\n",
        "r = search_client.search(term_to_search_for, top=5)\n",
        "\n",
        "scores = {}\n",
        "\n",
        "results = [doc for doc in r]\n",
        "for i, doc in enumerate(results):\n",
        "\n",
        "    hyp = doc[\"content\"].replace(\"\\n\", \" \")\n",
        "    display(Markdown(f'*Candidate answer: {hyp}*'))\n",
        "    scores[i] = 0\n",
        "\n",
        "    # roBERTA Entailment\n",
        "    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "    # run through model pre-trained on MNLI\n",
        "    x = tokenizer.encode(term_to_search_for, hyp, return_tensors='pt',\n",
        "                            truncation_strategy='only_first')\n",
        "    logits = nli_model(x.to(device))[0]\n",
        "\n",
        "    # we throw away \"neutral\" (dim 1) and take the probability of\n",
        "    # \"entailment\" (2) as the probability of the label being true\n",
        "    entail_contradiction_logits = logits[:, [0, 2]]\n",
        "    probs = entail_contradiction_logits.softmax(dim=1)\n",
        "    prob_label_is_true = probs[:, 1]\n",
        "    print(f\"Entailment score {prob_label_is_true[0]}\")\n",
        "    scores[i] += prob_label_is_true[0]\n",
        "\n",
        "    # Fuzz ratio (Levenschein Distance)\n",
        "    score = fuzz.ratio(term_to_search_for, hyp)\n",
        "    print(f\"Fuzz ratio {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # spaCy cosine similarity\n",
        "    doc1 = nlp(term_to_search_for)\n",
        "    doc2 = nlp(hyp)\n",
        "    score = doc1.similarity(doc2)\n",
        "    print(f\"spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Jaccard Distance\n",
        "    score = get_jaccard_sim(term_to_search_for, hyp)\n",
        "    print(f\"Jaccard distance {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Euclidean Distance\n",
        "    score = (100 - euclidean(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "    print(f\"Euclidean distance spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Manhattan Distance\n",
        "    score = (100 - manhattan(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "    print(f\"Manhattan distance spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Chebysev Distance\n",
        "    score = (100 - chebyshev(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "    print(f\"Chebyshev distance spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Spearman Ranking Correlation Coeeficient spaCy\n",
        "    score, _ = spearmanr(nlp(term_to_search_for).vector, nlp(hyp).vector)\n",
        "    print(f\"Spearman Ranking Correlation Coeeficient spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Weighted Pearson Correlation Coefficient spaCy\n",
        "    weights = nlp(hyp).vector.shape[0]\n",
        "    score = corr(nlp(term_to_search_for).vector, nlp(hyp).vector, weights)\n",
        "    print(f\"Weighted Pearson Correlation Coefficient spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    print(f'**Total Custom Distance Measure Score {scores[i]}**')\n",
        "    print(f'**Azure Cognitive Search Score {doc[\"@search.score\"]}**')\n"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "27",
              "statement_id": 14,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T09:08:46.6045854Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T09:08:46.7647805Z",
              "execution_finish_time": "2023-05-14T09:08:57.7936108Z",
              "spark_jobs": null,
              "parent_msg_id": "e0300403-122a-4064-abea-92668be94f66"
            },
            "text/plain": "StatementMeta(DataDiscovery, 27, 14, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Thomas out of Six Nations  Wales captain Gareth Thomas has been ruled out of the rest of the Six Nations with a broken thumb.  The full-back will have surgery on Monday after fracturing his thumb in the 24-18 win over France on Saturday. But Welsh legend Phil Bennett insisted Wales can cope without Thomas as they chase a first Grand Slam in 27 years. Bennett told BBC Sport: \"Such is the spirit in the camp, they'll put Kevin Morgan at 15, Rhys Williams at wing and just carry on.\" Thomas will miss the match against Scotland on 13 March, and what promises to be a huge encounter against the Irish six days later. Bennett added: \"It's a setback. He's a great captain, he leads from the front and the boys love him.\" Thomas was replaced at half-time by Williams as his side turned around a 15-6 deficit in Paris.  \"With Gareth missing I would think Michael Owen will be our captain,\" said Wales coach Mike Ruddock. \"He did a great job in the second half in France. He has been vice-captain all along throughout the championship.\" Wales travel to Edinburgh to take on Scotland in a fortnight and then host Ireland in Cardiff in the final round of matches in what could be the Grand Slam and championship decider. Bennett, an inspirational fly-half for Llanelli and Wales in the 1970s, insisted the national team were entering a new golden period. \"It was a great game and a magnificent result for Wales,\" Bennett told BBC Radio Five Live's Sportsweek programme.  \"The way this young team are blending, the glory days are on their way back. \"We couldn't get possession early on and France dominated and scored two tries. \"Had they been ruthless, Wales could have gone into the interval 30 points down. But they didn't take their chances. \"Wales defended fairly well but you cannot give that sort of quality ball to good sides. \"The All Blacks would have been ruthless and buried us in the first half. But the character we showed in the second half was quite outstanding.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2374: FutureWarning: The `truncation_strategy` argument is deprecated and will be removed in a future version, use `truncation=True` to truncate examples to a max length. You can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input size of the model (e.g. 512 for Bert).  If you have pairs of inputs, you can give a specific truncation strategy selected among `truncation='only_first'` (will only truncate the first sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence in the pairs).\n  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.9160066246986389\nFuzz ratio 0\nspaCy 0.4943371490776689\nJaccard distance 0.008771929824561403\nEuclidean distance spaCy 0.7367202900069381\nManhattan distance spaCy -2.6262820645794274\nChebyshev distance spaCy 0.958489465713501\nSpearman Ranking Correlation Coeeficient spaCy 0.4607033411482349\nWeighted Pearson Correlation Coefficient spaCy 0.8178629731922643\n**Total Custom Distance Measure Score 1.7666096687316895**\n**Azure Cognitive Search Score 5.5757585**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Kenyon denies Robben Barca return  Chelsea chief executive Peter Kenyon has played down reports that Arjen Robben will return for the Champions League match against Barcelona.  \"He's been responding well to treatment and started running on Friday, but we'll have to wait and see,\" he told BBC Five Live's Sportsweek. \"We're looking to getting him back as soon as possible, but he'll be back when it's right for him and for us. \"There's no plans at the moment around the Barcelona game.\" His comments contradict those of chiropractor Jean Pierre Meersseman who treated the Dutchman after he fractured his foot at the start of February. Robben had been expected to be out for six weeks, but Meersseman hinted that the winger could be fit for the vital Stamford Bridge game on 8 March. \"I hope he can be back and I will try to help him make that happen,\" Meersseman told the Mail on Sunday. \"I put everything right with Arjen's foot the last time I saw him 12 days ago. It was an obvious correction and easy to perform. \"I know he was pleased with what I did and now that he is running again. I am due to see him one more time again in the next few days.\" Meersseman is the medical co-ordinator at Italian side AC Milan.  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.02319514751434326\nFuzz ratio 0\nspaCy 0.5441085140968871\nJaccard distance 0.006535947712418301\nEuclidean distance spaCy 0.7441788759878958\nManhattan distance spaCy -2.579768744409084\nChebyshev distance spaCy 0.959799485206604\nSpearman Ranking Correlation Coeeficient spaCy 0.525475171946355\nWeighted Pearson Correlation Coefficient spaCy 0.7852365235676927\n**Total Custom Distance Measure Score 1.008760929107666**\n**Azure Cognitive Search Score 4.4981585**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Vickery upbeat about arm injury  England prop Phil Vickery is staying positive despite a broken arm ruling him out of the RBS Six Nations.  The 28-year-old fractured the radius in his right forearm during Gloucester's 17-16 win over Bath on Saturday. He will undergo an operation on Monday and is expected to be out for at least six weeks. He said: \"This isn't an injury that will stop me from working hard on the fitness elements and being around the lads.\" He added: \"I've got the operation this afternoon and I could be back doing fitness work after a week.\" \"As frustrating as it is, I've got to be positive.\"  After the game, Vickery spoke with Bath prop David Barnes, who also broke his arm recently. \"I had a chat with David Barnes and it looks like a similar injury to him,\" he said. \"He said he had the operation and he was back running after a week. \"There's no doubt that I'm going to get involved and be around this place as soon as I can after the operation.\" Gloucester director of rugby Nigel Melville said: \"Phil has broken his radius, which is the large bone in his forearm. \"I don't really know how it happened, but Phil will definitely be out of action for at least six weeks. \"I feel very sorry for him, as he has been in great shape. He really needed 80 minutes of rugby this weekend, and then this happened. Mentally, it must be very hard for him.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.7979953289031982\nFuzz ratio 0\nspaCy 0.5652431677987757\nJaccard distance 0.006211180124223602\nEuclidean distance spaCy 0.7491913911921283\nManhattan distance spaCy -2.51705733448267\nChebyshev distance spaCy 0.9626625299453735\nSpearman Ranking Correlation Coeeficient spaCy 0.541580239780442\nWeighted Pearson Correlation Coefficient spaCy 0.7972035895995842\n**Total Custom Distance Measure Score 1.9030301570892334**\n**Azure Cognitive Search Score 4.2438555**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Cole refuses to blame van Persie  Ashley Cole has refused to blame Robin van Persie for leaving Arsenal with no fully-fit strikers for the FA Cup fifth round replay at Sheffield United.  Van Persie is suspended alongside Dennis Bergkamp and Jose Antonio Reyes after being sent off at Southampton when Arsenal had a numerical advantage. Thierry Henry is ruled out with an Achilles tendon injury but Cole said: \"No-one is putting the blame on Robin. \"It's just something that happens on the spur of the moment.\" Cole added: \"I've done it before and I hope they didn't blame me for anything. \"Of course he'll learn. I've been sent off a couple of times now and it's just one of those things when you go a bit crazy for one or two seconds. Freddie Ljungberg is likely to be used in an emergency striking role and will be partnered by either Arturo Lupoli, Quincy Owusu-Abeyie or Jeremie Aliadiere. Gunners boss Arsene Wenger said: \"Freddie is an option but we need a second striker. \"I have to decide whether it will be Aliadiere, Quincy or Lupoli who will start with him up front. Those three will be involved.\" Arsenal are also without winger Robert Pires, who sustained an ankle injury at St Mary's. Wenger added: \"It doesn't look like anything is fractured, but it is a good ankle sprain. \"It does not look like Pires will be ready for two to three weeks.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.286815345287323\nFuzz ratio 0\nspaCy 0.5782458348190921\nJaccard distance 0.0\nEuclidean distance spaCy 0.7683413514889219\nManhattan distance spaCy -2.2697082729265095\nChebyshev distance spaCy 0.9633776330947876\nSpearman Ranking Correlation Coeeficient spaCy 0.5522430249224991\nWeighted Pearson Correlation Coefficient spaCy 0.8017443486362155\n**Total Custom Distance Measure Score 1.68105947971344**\n**Azure Cognitive Search Score 4.167914**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Wales get Williams fitness boost  Wales are hopeful that openside flanker Martyn Williams could be fit for Saturday's RBS 6 Nations championship opener against England in Cardiff.  Williams was expected to miss the match with a disc problem in his neck, but has been making a speedy recovery. \"He will have tests in the next 48 hours and we are pretty optimistic he is getting there,\" Wales' team physiotherapist Mark Davies said. \"It has been frustrating but he is on the mend, he has made good progress.\" Last week Williams, along with fellow flanker Colin Charvis - who is unlikely to play for at least a month while he recovers from a foot injury - was all but ruled out of the Millennium Stadium clash. With Williams initially thought to be struggling, the signs pointed towards Wales coach Mike Ruddock handing a first cap to former Wales Under-21 skipper Richie Pugh.  Cardiff Blues flanker Williams, 29, offers considerable experience and if he is declared fit then Ruddock might be tempted to include him in the back row. Charvis will be reviewed by the Wales medical staff next Monday, but Davies admitted that there was only an \"outside chance\" of him being fit to face France in Wales' third championship game on 26 February. Wales' other injury concern is Pugh's fellow Neath-Swansea Ospreys player Sonny Parker, as the centre has a trapped nerve in his neck. \"Sonny's injury is still an issue,\" Davies said. \"It is still painful and irritable. We will run the rule of thumb over him in the next couple of days.\" Ruddock will name his starting line-up for the England game at 1830 GMT on Tuesday evening, as Wales target their first victory in Cardiff over the world champions since 1993.  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.522243857383728\nFuzz ratio 0\nspaCy 0.5226353966638094\nJaccard distance 0.010582010582010581\nEuclidean distance spaCy 0.7444192930929813\nManhattan distance spaCy -2.5673284966498615\nChebyshev distance spaCy 0.9611926651000977\nSpearman Ranking Correlation Coeeficient spaCy 0.48937921532461465\nWeighted Pearson Correlation Coefficient spaCy 0.62481859178453\n**Total Custom Distance Measure Score 1.307942509651184**\n**Azure Cognitive Search Score 4.147021**\n"
          ]
        }
      ],
      "execution_count": 13,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Search the generated Azure Search Index with Semantic Search and run evaluation"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "1) [Enable Semantic Search](https://docs.microsoft.com/en-us/azure/search/semantic-search-overview#enable-semantic-search) on your search instance\n",
        "\n",
        "2) [Configure Semantic Search](https://docs.microsoft.com/en-us/azure/search/semantic-how-to-query-request?tabs=semanticConfiguration%2Cportal#create-a-semantic-configuration)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "term_to_search_for = \"Whose thumb was fractured?\"\n",
        "\n",
        "r = search_client.search(term_to_search_for,\n",
        "                                 #filter=filter,\n",
        "                                 query_type=\"semantic\",\n",
        "                                 query_language=\"en-us\",\n",
        "                                 query_speller=\"lexicon\",\n",
        "                                 semantic_configuration_name=semantic_configuration_name,\n",
        "                                 top=5)\n",
        "\n",
        "\n",
        "scores = {}\n",
        "\n",
        "results = [doc for doc in r]\n",
        "for i, doc in enumerate(results):\n",
        "\n",
        "    try:\n",
        "\n",
        "        hyp = doc[\"content\"].replace(\"\\n\", \" \")\n",
        "        display(Markdown(f'*Candidate answer: {hyp}*'))\n",
        "        scores[i] = 0\n",
        "\n",
        "        # roBERTA Entailment\n",
        "        device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "        # run through model pre-trained on MNLI\n",
        "        x = tokenizer.encode(term_to_search_for, hyp, return_tensors='pt',\n",
        "                                truncation_strategy='only_first')\n",
        "        logits = nli_model(x.to(device))[0]\n",
        "\n",
        "        # we throw away \"neutral\" (dim 1) and take the probability of\n",
        "        # \"entailment\" (2) as the probability of the label being true\n",
        "        entail_contradiction_logits = logits[:, [0, 2]]\n",
        "        probs = entail_contradiction_logits.softmax(dim=1)\n",
        "        prob_label_is_true = probs[:, 1]\n",
        "        print(f\"Entailment score {prob_label_is_true[0]}\")\n",
        "        scores[i] += prob_label_is_true[0]\n",
        "\n",
        "        # Fuzz ratio (Levenschein Distance)\n",
        "        score = fuzz.ratio(term_to_search_for, hyp)\n",
        "        print(f\"Fuzz ratio {score}\")\n",
        "        scores[i] += score\n",
        "\n",
        "        # spaCy cosine similarity\n",
        "        doc1 = nlp(term_to_search_for)\n",
        "        doc2 = nlp(hyp)\n",
        "        score = doc1.similarity(doc2)\n",
        "        print(f\"spaCy {score}\")\n",
        "        scores[i] += score\n",
        "\n",
        "        # Jaccard Distance\n",
        "        score = get_jaccard_sim(term_to_search_for, hyp)\n",
        "        print(f\"Jaccard distance {score}\")\n",
        "        scores[i] += score\n",
        "\n",
        "        # Euclidean Distance\n",
        "        score = (100 - euclidean(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "        print(f\"Euclidean distance spaCy {score}\")\n",
        "        scores[i] += score\n",
        "\n",
        "        # Manhattan Distance\n",
        "        score = (100 - manhattan(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "        print(f\"Manhattan distance spaCy {score}\")\n",
        "        scores[i] += score\n",
        "\n",
        "        # Chebysev Distance\n",
        "        score = (100 - chebyshev(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "        print(f\"Chebyshev distance spaCy {score}\")\n",
        "        scores[i] += score\n",
        "\n",
        "        # Spearman Ranking Correlation Coeeficient spaCy\n",
        "        score, _ = spearmanr(nlp(term_to_search_for).vector, nlp(hyp).vector)\n",
        "        print(f\"Spearman Ranking Correlation Coeeficient spaCy {score}\")\n",
        "        scores[i] += score\n",
        "\n",
        "        # Weighted Pearson Correlation Coefficient spaCy\n",
        "        weights = nlp(hyp).vector.shape[0]\n",
        "        score = corr(nlp(term_to_search_for).vector, nlp(hyp).vector, weights)\n",
        "        print(f\"Weighted Pearson Correlation Coefficient spaCy {score}\")\n",
        "        scores[i] += score\n",
        "\n",
        "        print(f'**Total Custom Distance Measure Score {scores[i]}**')\n",
        "        print(f'**Azure Cognitive Search Score {doc[\"@search.score\"]}**')\n",
        "\n",
        "    except Exception as swallowed:\n",
        "        print(f\"An error occured on record {i} {swallowed}\")\n",
        "        continue"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "29",
              "statement_id": 9,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T09:30:42.3214788Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T09:30:42.4815616Z",
              "execution_finish_time": "2023-05-14T09:30:59.6364742Z",
              "spark_jobs": null,
              "parent_msg_id": "9a10545e-f57d-45be-943b-4f1013524560"
            },
            "text/plain": "StatementMeta(DataDiscovery, 29, 9, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Thomas out of Six Nations  Wales captain Gareth Thomas has been ruled out of the rest of the Six Nations with a broken thumb.  The full-back will have surgery on Monday after fracturing his thumb in the 24-18 win over France on Saturday. But Welsh legend Phil Bennett insisted Wales can cope without Thomas as they chase a first Grand Slam in 27 years. Bennett told BBC Sport: \"Such is the spirit in the camp, they'll put Kevin Morgan at 15, Rhys Williams at wing and just carry on.\" Thomas will miss the match against Scotland on 13 March, and what promises to be a huge encounter against the Irish six days later. Bennett added: \"It's a setback. He's a great captain, he leads from the front and the boys love him.\" Thomas was replaced at half-time by Williams as his side turned around a 15-6 deficit in Paris.  \"With Gareth missing I would think Michael Owen will be our captain,\" said Wales coach Mike Ruddock. \"He did a great job in the second half in France. He has been vice-captain all along throughout the championship.\" Wales travel to Edinburgh to take on Scotland in a fortnight and then host Ireland in Cardiff in the final round of matches in what could be the Grand Slam and championship decider. Bennett, an inspirational fly-half for Llanelli and Wales in the 1970s, insisted the national team were entering a new golden period. \"It was a great game and a magnificent result for Wales,\" Bennett told BBC Radio Five Live's Sportsweek programme.  \"The way this young team are blending, the glory days are on their way back. \"We couldn't get possession early on and France dominated and scored two tries. \"Had they been ruthless, Wales could have gone into the interval 30 points down. But they didn't take their chances. \"Wales defended fairly well but you cannot give that sort of quality ball to good sides. \"The All Blacks would have been ruthless and buried us in the first half. But the character we showed in the second half was quite outstanding.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2374: FutureWarning: The `truncation_strategy` argument is deprecated and will be removed in a future version, use `truncation=True` to truncate examples to a max length. You can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input size of the model (e.g. 512 for Bert).  If you have pairs of inputs, you can give a specific truncation strategy selected among `truncation='only_first'` (will only truncate the first sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence in the pairs).\n  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.9160066246986389\nFuzz ratio 0\nspaCy 0.4943371490776689\nJaccard distance 0.008771929824561403\nEuclidean distance spaCy 0.7367202900069381\nManhattan distance spaCy -2.6262820645794274\nChebyshev distance spaCy 0.958489465713501\nSpearman Ranking Correlation Coeeficient spaCy 0.4607033411482349\nWeighted Pearson Correlation Coefficient spaCy 0.8178629731922643\n**Total Custom Distance Measure Score 1.7666096687316895**\n**Azure Cognitive Search Score 5.5757585**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Vickery upbeat about arm injury  England prop Phil Vickery is staying positive despite a broken arm ruling him out of the RBS Six Nations.  The 28-year-old fractured the radius in his right forearm during Gloucester's 17-16 win over Bath on Saturday. He will undergo an operation on Monday and is expected to be out for at least six weeks. He said: \"This isn't an injury that will stop me from working hard on the fitness elements and being around the lads.\" He added: \"I've got the operation this afternoon and I could be back doing fitness work after a week.\" \"As frustrating as it is, I've got to be positive.\"  After the game, Vickery spoke with Bath prop David Barnes, who also broke his arm recently. \"I had a chat with David Barnes and it looks like a similar injury to him,\" he said. \"He said he had the operation and he was back running after a week. \"There's no doubt that I'm going to get involved and be around this place as soon as I can after the operation.\" Gloucester director of rugby Nigel Melville said: \"Phil has broken his radius, which is the large bone in his forearm. \"I don't really know how it happened, but Phil will definitely be out of action for at least six weeks. \"I feel very sorry for him, as he has been in great shape. He really needed 80 minutes of rugby this weekend, and then this happened. Mentally, it must be very hard for him.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.7979953289031982\nFuzz ratio 0\nspaCy 0.5652431677987757\nJaccard distance 0.006211180124223602\nEuclidean distance spaCy 0.7491913911921283\nManhattan distance spaCy -2.51705733448267\nChebyshev distance spaCy 0.9626625299453735\nSpearman Ranking Correlation Coeeficient spaCy 0.541580239780442\nWeighted Pearson Correlation Coefficient spaCy 0.7972035895995842\n**Total Custom Distance Measure Score 1.9030301570892334**\n**Azure Cognitive Search Score 4.2438555**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Costin aims for comeback in 2006  Jamie Costin should be paralysed.  He says so himself in a matter-of-fact way as he recalls the car accident which occurred nine days before he was scheduled to step out into the Olympic Stadium in Athens for the 50K Walk. There is an ironic chuckle as he talks of his immediate thoughts after a lorry, driving on the wrong side of the road, had ploughed into his rental car. \"I was in a lot of pain and I guessed that one of my toes was broken,\" says the Waterford man. \"But I was thinking maybe with a cortisone injection you never know. \"In my back, it felt as though all the muscles had been ripped off my pelvis but I was thinking maybe we could do something with laser therapy and ultra sound and hopefully I'd be able to race.\" It took over 10 hours before Jamie knew with certainty that he would not be competing in his second Olympics. \"My back had been broken in two places and with one of my vertebrae, the bottom part had exploded so I'm fierce lucky not be paralysed. \"I'd fractured my big toe as well which was on the brake.\" Jamie didn't finally arrive at hospital in Athens until some nine and a half hours after the accident.  \"For the first nine hours, I had no pain killers which was ridiculous in 35 degrees heat. \"But once I got the scans and saw them it was a case of moving on and thinking:'OK, I've got a different set of circumstances now'.\" Within three days he was arriving back in Ireland by air ambulance. Doctors in Athens had wanted to operate on Jamie's back immediately but he insisted on delaying any surgery until he arrived back home - something he is now very relieved about. \"The Greek doctors were going to put three or four inch titanium rods either side of my spinal cord up through my vertebrae. \"That would have fused all my lower back and I would never have been able to race again. They were really putting a lot of pressure on me to agree to the surgery. \"But when I got to the Mater in Dublin they said it was possible for it to heal totally naturally which is giving me the chance to get back into competition which is very important to me. The people at the Mater have been absolutely fantastic.\" Jamie had to wear a body cast for three and a half months after the accident and spent most of that time flat on his back.  He then progressed to crutches for six weeks until he was finally able to walk unaided on 10 January. \"Walking without the crutches seemed like something finally really measurable in terms of my recovery.\" Physio sessions with Johnston McEvoy in Limerick have been a vital part of his recovery. \"Johnston uses an advanced type of acupuncture and it's very effective. \"Needles get put right close up to my spine. A two and a half inch needle went in yesterday and I'm fairly incapacitated today as a result.\" Jamie has also travelled to receive treatment at the Polish training centre in Spala where he has trained with triple Olympic champion Robert Korzeniowski over the past five years. \"I was there for over a fortnight earlier this month and underwent a fair extreme treatment called cryotherapy. \"Basically, there's a small room which is cooled by liquid nitrogen to minus 160 degrees centigrade and it promotes deep healing.\"  Jamie heads to Poland again on Sunday where he will be having daily cryotherapy in addition to twice-daily physio sessions and pool-work. All these sessions are small steps on the way to what Jamie hopes will be a return to racing in 2006. \"It's all about trying to get mobility in my back. Lying down for three and a half months didn't really help with the strength. \"There's a lot of work involved in my recovery. I'm doing about six hours a day between physio and pool work. \"I'm also going to the gym to lift very light weights to try and build up my muscles. I'm fairly full on with everything I do. \"I'd hope to be training regularly by March. But training is just part of the process of getting back. \"At the moment, every time I go and do a big bit of movement, my whole pelvic area all down my lower back just tightens up. \"It's a case of waiting and seeing how it reacts. Hopefully, after four or five months my back won't tighten up as much.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.796183168888092\nFuzz ratio 0\nspaCy 0.5705528714398947\nJaccard distance 0.002457002457002457\nEuclidean distance spaCy 0.743952954405338\nManhattan distance spaCy -2.4962075752019883\nChebyshev distance spaCy 0.9516753149032593\nSpearman Ranking Correlation Coeeficient spaCy 0.5492176579739774\nWeighted Pearson Correlation Coefficient spaCy 0.7938019818444078\n**Total Custom Distance Measure Score 1.9116334915161133**\n**Azure Cognitive Search Score 2.8234682**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Owen set for skipper role  Wales number eight Michael Owen says replacing Gareth Thomas as Wales' captain will be straightforward because of the leadership quality in the squad.  \"You dream about playing for Wales, but it never enters your head that you will one day captain your country,\" he said. \"It's an easy job really because there are so many experienced players in the side, but it will be a massive honour.\" Coach Mike Ruddock says the 24-year-old should take the armband from broken thumb victim Thomas against Scotland. Ruddock praised Owen's contribution as vice-captain and pack leader in this year's wins over England, Italy and France, notably his role in the second-half in Paris following Thomas' injury. The 22-cap Dragons man says that Thomas will be badly missed, though. \"Gareth is going to be a massive loss because he is a brilliant player and he has done a fantastic job since he has been captain,\" said Owen. \"He has put his own unique stamp on things. We are going to miss him but the strength of the squad is shown when you have someone like Kevin Morgan to step in.\"  Owen admitted the Grand Slam had been mentioned within the squad but that no-one was getting carried away with the three victories over England, Italy and France. \"We can't get ahead of ourselves,\" he said. \"We have a massive game next up and we are just looking forward to preparing for that. \"Scotland are improving all the time. Like us they did well in France and were unlucky not to get a win. \"Against Ireland they played fantastically well in the first 15 minutes before falling away. \"They are probably going on a similar journey to the one we have been on in the last couple of years.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.8652717471122742\nFuzz ratio 0\nspaCy 0.5380841791349585\nJaccard distance 0.010101010101010102\nEuclidean distance spaCy 0.7410224853019125\nManhattan distance spaCy -2.5730651529179887\nChebyshev distance spaCy 0.9595381164550781\nSpearman Ranking Correlation Coeeficient spaCy 0.5125963621818019\nWeighted Pearson Correlation Coefficient spaCy 0.8215269542644464\n**Total Custom Distance Measure Score 1.8750755786895752**\n**Azure Cognitive Search Score 3.8952117**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Aussies unhappy with pitch  Australian skipper Ricky Ponting was still able to raise a smile despite his side's 13-run defeat by India in the fourth Test at Mumbai.  They had already done enough to win the Border-Gavaskar Trophy with victories at Bangalore and Nagpur, their first series success in India since 1969-70. \"I said I thought it would be one of the all-time great series and the first three Tests turned out that way. \"This was nowhere near a Test pitch. It was terribly disappointing.\" He was less than thrilled, however, with a pitch in Mumbai which saw 38 wickets fall on the second and third days to bring the match to an early resolution.  Australia only needed 107 in the final innings but were bowled out for 93 in just 30.5 overs as the Indian spinners made the most of conditions. \"It was definitely gettable, but I don't think we played at our best with the bat today. \"Full credit to India though. They batted well in tough conditions and when the game was there to be won with the ball, they did what they had to do.\" Groundsman Polly Umrigar defended his pitch, saying: \"My verdict is that the batsmen did not apply themselves, \"The wicket was not as difficult as it is being made out to be. Batsmen willing to play shots got ample opportunities.\" Ponting also paid tribute to vice-captain Adam Gilchrist, who led the side in the first three Tests while he was recuperating from a broken thumb. \"When you look back through the series, everyone has contributed in one way or another. That's a pleasing thing - it's what you need in a good side,\" he added.  \"It's been a great tour. I've only been here a short time but we've certainly enjoyed ourselves - I think all of the players really enjoy touring India. \"It's a place we've been to quite a lot over the last few years. Every time we come back we enjoy it more and more and it certainly helps when you win as well.\" Michael Clarke took 6-9 on the final day and also contributed 400 runs in his first Test series. But the Man of the Series award went to team-mate Damien Martyn, who made 444 runs at an average of 55, although he ended it with a duck. \"I'd have liked 10 more runs in the second innings today,\" Martyn joked. \"We came over here to win and fight hard. It was a bit disappointing to finish off that way but we've won [the series] and the guiys have made a fantastic effort in the last month. \"We always know it's going to be hard over here. It's always a true test for a batsman.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.957051694393158\nFuzz ratio 0\nspaCy 0.5333328745669992\nJaccard distance 0.0035460992907801418\nEuclidean distance spaCy 0.7369176001588964\nManhattan distance spaCy -2.6169634030759337\nChebyshev distance spaCy 0.957062726020813\nSpearman Ranking Correlation Coeeficient spaCy 0.5078425315836842\nWeighted Pearson Correlation Coefficient spaCy 0.8025071997095297\n**Total Custom Distance Measure Score 1.8812973499298096**\n**Azure Cognitive Search Score 2.9979866**\n"
          ]
        }
      ],
      "execution_count": 30,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Now we implement the OpenAI Retrieval Augmented Generation Pattern using Azure Cognitive Search standard search and evaluate"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "openai.api_key = openai_api_key\n",
        "openai.api_type = openai_api_type\n",
        "openai.api_base = f\"https://{openai_service}.openai.azure.com\"\n",
        "openai.api_version = openai_api_version\n",
        "\n",
        "def query(user_input):\n",
        "   \n",
        "    prompt_prefix = \"\"\"<|im_start|>system\n",
        "    Let's work this out it a step by step to be sure we have the right answer\n",
        "\n",
        "    Sources:\n",
        "    {sources}\n",
        "    \n",
        "\n",
        "    <|im_end|>\"\"\"\n",
        "\n",
        "    turn_prefix = \"\"\"\n",
        "    <|im_start|>user\n",
        "    \"\"\"\n",
        "\n",
        "    turn_suffix = \"\"\"\n",
        "    <|im_end|>\n",
        "    <|im_start|>assistant\n",
        "    \"\"\"\n",
        "\n",
        "    prompt_history = turn_prefix\n",
        "\n",
        "    history = []\n",
        "\n",
        "    summary_prompt_template = \"\"\"Below is a summary of the conversation so far, and a new question asked by the user that needs to be answered by searching in a knowledge base. Generate a search query based on the conversation and the new question. Source names are not good search terms to include in the search query.\n",
        "\n",
        "    Summary:\n",
        "    {summary}\n",
        "\n",
        "    Question:\n",
        "    {question}\n",
        "\n",
        "    Search query:\n",
        "    \"\"\"\n",
        "\n",
        "    content = \"\"\n",
        "\n",
        "    # Exclude category, to simulate scenarios where there's a set of docs you can't see\n",
        "    exclude_category = None\n",
        "\n",
        "    if len(history) > 0:\n",
        "        completion = openai.Completion.create(\n",
        "            engine=openai_gpt_deployment,\n",
        "            prompt=summary_prompt_template.format(summary=\"\\n\".join(history), question=user_input),\n",
        "            temperature=0.9,\n",
        "            max_tokens=320,\n",
        "            stop=[\"\\n\"])\n",
        "        search = completion.choices[0].text\n",
        "        print(f\"search {search}\")\n",
        "    else:\n",
        "        search = user_input\n",
        "\n",
        "        # Alternatively simply use search_client.search(q, top=3) if not using semantic search\n",
        "        print(\"Searching:\", search)\n",
        "        print(\"-------------------\")\n",
        "        filter = \"category ne '{}'\".format(exclude_category.replace(\"'\", \"''\")) if exclude_category else None\n",
        "        r = search_client.search(search,\n",
        "                                 filter=filter,\n",
        "                                 top=5)\n",
        "\n",
        "        content = [doc for doc in r]\n",
        "\n",
        "    prompt = prompt_prefix.format(sources=content) + prompt_history + user_input + turn_suffix\n",
        "\n",
        "    completion = openai.Completion.create(\n",
        "        engine=openai_chatgpt_deployment,\n",
        "        prompt=prompt,\n",
        "        temperature=0.7,\n",
        "        max_tokens=1024,\n",
        "        stop=[\"<|im_end|>\", \"<|im_start|>\"])\n",
        "\n",
        "    prompt_history += user_input + turn_suffix + completion.choices[0].text + \"\\n<|im_end|>\" + turn_prefix\n",
        "    history.append(\"user: \" + user_input)\n",
        "    history.append(\"assistant: \" + completion.choices[0].text)\n",
        "\n",
        "    print(\"\\n-------------------\\n\".join(history))\n",
        "    print(\"\\n-------------------\\nPrompt:\\n\" + prompt)\n",
        "\n",
        "    # Now let's evaluate the results\n",
        "\n",
        "    scores = {}\n",
        "\n",
        "\n",
        "    for i, doc in enumerate(content):\n",
        "\n",
        "        try:\n",
        "\n",
        "            hyp = doc[\"content\"].replace(\"\\n\", \" \")\n",
        "            display(Markdown(f'*Candidate answer: {hyp}*'))\n",
        "            scores[i] = 0\n",
        "\n",
        "            # roBERTA Entailment\n",
        "            device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "            # run through model pre-trained on MNLI\n",
        "            x = tokenizer.encode(term_to_search_for, hyp, return_tensors='pt',\n",
        "                                    truncation_strategy='only_first')\n",
        "            logits = nli_model(x.to(device))[0]\n",
        "\n",
        "            # we throw away \"neutral\" (dim 1) and take the probability of\n",
        "            # \"entailment\" (2) as the probability of the label being true\n",
        "            entail_contradiction_logits = logits[:, [0, 2]]\n",
        "            probs = entail_contradiction_logits.softmax(dim=1)\n",
        "            prob_label_is_true = probs[:, 1]\n",
        "            print(f\"Entailment score {prob_label_is_true[0]}\")\n",
        "            scores[i] += prob_label_is_true[0]\n",
        "\n",
        "            # Fuzz ratio (Levenschein Distance)\n",
        "            score = fuzz.ratio(term_to_search_for, hyp)\n",
        "            print(f\"Fuzz ratio {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # spaCy cosine similarity\n",
        "            doc1 = nlp(term_to_search_for)\n",
        "            doc2 = nlp(hyp)\n",
        "            score = doc1.similarity(doc2)\n",
        "            print(f\"spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Jaccard Distance\n",
        "            score = get_jaccard_sim(term_to_search_for, hyp)\n",
        "            print(f\"Jaccard distance {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Euclidean Distance\n",
        "            score = (100 - euclidean(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "            print(f\"Euclidean distance spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Manhattan Distance\n",
        "            score = (100 - manhattan(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "            print(f\"Manhattan distance spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Chebysev Distance\n",
        "            score = (100 - chebyshev(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "            print(f\"Chebyshev distance spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Spearman Ranking Correlation Coeeficient spaCy\n",
        "            score, _ = spearmanr(nlp(term_to_search_for).vector, nlp(hyp).vector)\n",
        "            print(f\"Spearman Ranking Correlation Coeeficient spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Weighted Pearson Correlation Coefficient spaCy\n",
        "            weights = nlp(hyp).vector.shape[0]\n",
        "            score = corr(nlp(term_to_search_for).vector, nlp(hyp).vector, weights)\n",
        "            print(f\"Weighted Pearson Correlation Coefficient spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            print(f'**Total Custom Distance Measure Score {scores[i]}**')\n",
        "            print(f'**Azure Cognitive Search Score {doc[\"@search.score\"]}**')\n",
        "\n",
        "        except Exception as swallowed:\n",
        "            print(f\"An error occured on record {i} {swallowed}\")\n",
        "            continue\n"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "29",
              "statement_id": 11,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T09:31:47.5100717Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T09:31:47.6712683Z",
              "execution_finish_time": "2023-05-14T09:31:47.8436844Z",
              "spark_jobs": null,
              "parent_msg_id": "60dab6b6-fc2e-485e-807a-92084edc3080"
            },
            "text/plain": "StatementMeta(DataDiscovery, 29, 11, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 32,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Run the actual query"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "query(\"Whose thumb was fractured?\")"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "29",
              "statement_id": 12,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T09:32:30.4607317Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T09:32:30.6109389Z",
              "execution_finish_time": "2023-05-14T09:32:43.6297598Z",
              "spark_jobs": null,
              "parent_msg_id": "5bd55307-52e0-4d1f-bd33-9933b1a35335"
            },
            "text/plain": "StatementMeta(DataDiscovery, 29, 12, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Searching: Whose thumb was fractured?\n-------------------\nuser: Whose thumb was fractured?\n-------------------\nassistant:  Gareth Thomas' thumb was fractured.\n\n-------------------\nPrompt:\n<|im_start|>system\n    Let's work this out it a step by step to be sure we have the right answer\n\n    Sources:\n    [{'content': 'Thomas out of Six Nations\\n\\nWales captain Gareth Thomas has been ruled out of the rest of the Six Nations with a broken thumb.\\n\\nThe full-back will have surgery on Monday after fracturing his thumb in the 24-18 win over France on Saturday. But Welsh legend Phil Bennett insisted Wales can cope without Thomas as they chase a first Grand Slam in 27 years. Bennett told BBC Sport: \"Such is the spirit in the camp, they\\'ll put Kevin Morgan at 15, Rhys Williams at wing and just carry on.\" Thomas will miss the match against Scotland on 13 March, and what promises to be a huge encounter against the Irish six days later. Bennett added: \"It\\'s a setback. He\\'s a great captain, he leads from the front and the boys love him.\" Thomas was replaced at half-time by Williams as his side turned around a 15-6 deficit in Paris.\\n\\n\"With Gareth missing I would think Michael Owen will be our captain,\" said Wales coach Mike Ruddock. \"He did a great job in the second half in France. He has been vice-captain all along throughout the championship.\" Wales travel to Edinburgh to take on Scotland in a fortnight and then host Ireland in Cardiff in the final round of matches in what could be the Grand Slam and championship decider. Bennett, an inspirational fly-half for Llanelli and Wales in the 1970s, insisted the national team were entering a new golden period. \"It was a great game and a magnificent result for Wales,\" Bennett told BBC Radio Five Live\\'s Sportsweek programme.\\n\\n\"The way this young team are blending, the glory days are on their way back. \"We couldn\\'t get possession early on and France dominated and scored two tries. \"Had they been ruthless, Wales could have gone into the interval 30 points down. But they didn\\'t take their chances. \"Wales defended fairly well but you cannot give that sort of quality ball to good sides. \"The All Blacks would have been ruthless and buried us in the first half. But the character we showed in the second half was quite outstanding.\"\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9ydWdieTAwOC50eHQ1', '@search.score': 5.5757585, '@search.reranker_score': None, '@search.highlights': None, '@search.captions': None}, {'content': 'Kenyon denies Robben Barca return\\n\\nChelsea chief executive Peter Kenyon has played down reports that Arjen Robben will return for the Champions League match against Barcelona.\\n\\n\"He\\'s been responding well to treatment and started running on Friday, but we\\'ll have to wait and see,\" he told BBC Five Live\\'s Sportsweek. \"We\\'re looking to getting him back as soon as possible, but he\\'ll be back when it\\'s right for him and for us. \"There\\'s no plans at the moment around the Barcelona game.\" His comments contradict those of chiropractor Jean Pierre Meersseman who treated the Dutchman after he fractured his foot at the start of February. Robben had been expected to be out for six weeks, but Meersseman hinted that the winger could be fit for the vital Stamford Bridge game on 8 March. \"I hope he can be back and I will try to help him make that happen,\" Meersseman told the Mail on Sunday. \"I put everything right with Arjen\\'s foot the last time I saw him 12 days ago. It was an obvious correction and easy to perform. \"I know he was pleased with what I did and now that he is running again. I am due to see him one more time again in the next few days.\" Meersseman is the medical co-ordinator at Italian side AC Milan.\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9mb290YmFsbDAxNi50eHQ1', '@search.score': 4.4981585, '@search.reranker_score': None, '@search.highlights': None, '@search.captions': None}, {'content': 'Vickery upbeat about arm injury\\n\\nEngland prop Phil Vickery is staying positive despite a broken arm ruling him out of the RBS Six Nations.\\n\\nThe 28-year-old fractured the radius in his right forearm during Gloucester\\'s 17-16 win over Bath on Saturday. He will undergo an operation on Monday and is expected to be out for at least six weeks. He said: \"This isn\\'t an injury that will stop me from working hard on the fitness elements and being around the lads.\" He added: \"I\\'ve got the operation this afternoon and I could be back doing fitness work after a week.\" \"As frustrating as it is, I\\'ve got to be positive.\"\\n\\nAfter the game, Vickery spoke with Bath prop David Barnes, who also broke his arm recently. \"I had a chat with David Barnes and it looks like a similar injury to him,\" he said. \"He said he had the operation and he was back running after a week. \"There\\'s no doubt that I\\'m going to get involved and be around this place as soon as I can after the operation.\" Gloucester director of rugby Nigel Melville said: \"Phil has broken his radius, which is the large bone in his forearm. \"I don\\'t really know how it happened, but Phil will definitely be out of action for at least six weeks. \"I feel very sorry for him, as he has been in great shape. He really needed 80 minutes of rugby this weekend, and then this happened. Mentally, it must be very hard for him.\"\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9ydWdieTEzOC50eHQ1', '@search.score': 4.2438555, '@search.reranker_score': None, '@search.highlights': None, '@search.captions': None}, {'content': 'Cole refuses to blame van Persie\\n\\nAshley Cole has refused to blame Robin van Persie for leaving Arsenal with no fully-fit strikers for the FA Cup fifth round replay at Sheffield United.\\n\\nVan Persie is suspended alongside Dennis Bergkamp and Jose Antonio Reyes after being sent off at Southampton when Arsenal had a numerical advantage. Thierry Henry is ruled out with an Achilles tendon injury but Cole said: \"No-one is putting the blame on Robin. \"It\\'s just something that happens on the spur of the moment.\" Cole added: \"I\\'ve done it before and I hope they didn\\'t blame me for anything. \"Of course he\\'ll learn. I\\'ve been sent off a couple of times now and it\\'s just one of those things when you go a bit crazy for one or two seconds. Freddie Ljungberg is likely to be used in an emergency striking role and will be partnered by either Arturo Lupoli, Quincy Owusu-Abeyie or Jeremie Aliadiere. Gunners boss Arsene Wenger said: \"Freddie is an option but we need a second striker. \"I have to decide whether it will be Aliadiere, Quincy or Lupoli who will start with him up front. Those three will be involved.\" Arsenal are also without winger Robert Pires, who sustained an ankle injury at St Mary\\'s. Wenger added: \"It doesn\\'t look like anything is fractured, but it is a good ankle sprain. \"It does not look like Pires will be ready for two to three weeks.\"\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9mb290YmFsbDA4NC50eHQ1', '@search.score': 4.167914, '@search.reranker_score': None, '@search.highlights': None, '@search.captions': None}, {'content': 'Wales get Williams fitness boost\\n\\nWales are hopeful that openside flanker Martyn Williams could be fit for Saturday\\'s RBS 6 Nations championship opener against England in Cardiff.\\n\\nWilliams was expected to miss the match with a disc problem in his neck, but has been making a speedy recovery. \"He will have tests in the next 48 hours and we are pretty optimistic he is getting there,\" Wales\\' team physiotherapist Mark Davies said. \"It has been frustrating but he is on the mend, he has made good progress.\" Last week Williams, along with fellow flanker Colin Charvis - who is unlikely to play for at least a month while he recovers from a foot injury - was all but ruled out of the Millennium Stadium clash. With Williams initially thought to be struggling, the signs pointed towards Wales coach Mike Ruddock handing a first cap to former Wales Under-21 skipper Richie Pugh.\\n\\nCardiff Blues flanker Williams, 29, offers considerable experience and if he is declared fit then Ruddock might be tempted to include him in the back row. Charvis will be reviewed by the Wales medical staff next Monday, but Davies admitted that there was only an \"outside chance\" of him being fit to face France in Wales\\' third championship game on 26 February. Wales\\' other injury concern is Pugh\\'s fellow Neath-Swansea Ospreys player Sonny Parker, as the centre has a trapped nerve in his neck. \"Sonny\\'s injury is still an issue,\" Davies said. \"It is still painful and irritable. We will run the rule of thumb over him in the next couple of days.\" Ruddock will name his starting line-up for the England game at 1830 GMT on Tuesday evening, as Wales target their first victory in Cardiff over the world champions since 1993.\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9ydWdieTA3Mi50eHQ1', '@search.score': 4.147021, '@search.reranker_score': None, '@search.highlights': None, '@search.captions': None}]\n    \n\n    <|im_end|>\n    <|im_start|>user\n    Whose thumb was fractured?\n    <|im_end|>\n    <|im_start|>assistant\n    \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Thomas out of Six Nations  Wales captain Gareth Thomas has been ruled out of the rest of the Six Nations with a broken thumb.  The full-back will have surgery on Monday after fracturing his thumb in the 24-18 win over France on Saturday. But Welsh legend Phil Bennett insisted Wales can cope without Thomas as they chase a first Grand Slam in 27 years. Bennett told BBC Sport: \"Such is the spirit in the camp, they'll put Kevin Morgan at 15, Rhys Williams at wing and just carry on.\" Thomas will miss the match against Scotland on 13 March, and what promises to be a huge encounter against the Irish six days later. Bennett added: \"It's a setback. He's a great captain, he leads from the front and the boys love him.\" Thomas was replaced at half-time by Williams as his side turned around a 15-6 deficit in Paris.  \"With Gareth missing I would think Michael Owen will be our captain,\" said Wales coach Mike Ruddock. \"He did a great job in the second half in France. He has been vice-captain all along throughout the championship.\" Wales travel to Edinburgh to take on Scotland in a fortnight and then host Ireland in Cardiff in the final round of matches in what could be the Grand Slam and championship decider. Bennett, an inspirational fly-half for Llanelli and Wales in the 1970s, insisted the national team were entering a new golden period. \"It was a great game and a magnificent result for Wales,\" Bennett told BBC Radio Five Live's Sportsweek programme.  \"The way this young team are blending, the glory days are on their way back. \"We couldn't get possession early on and France dominated and scored two tries. \"Had they been ruthless, Wales could have gone into the interval 30 points down. But they didn't take their chances. \"Wales defended fairly well but you cannot give that sort of quality ball to good sides. \"The All Blacks would have been ruthless and buried us in the first half. But the character we showed in the second half was quite outstanding.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2374: FutureWarning: The `truncation_strategy` argument is deprecated and will be removed in a future version, use `truncation=True` to truncate examples to a max length. You can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input size of the model (e.g. 512 for Bert).  If you have pairs of inputs, you can give a specific truncation strategy selected among `truncation='only_first'` (will only truncate the first sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence in the pairs).\n  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.9160066246986389\nFuzz ratio 0\nspaCy 0.4943371490776689\nJaccard distance 0.008771929824561403\nEuclidean distance spaCy 0.7367202900069381\nManhattan distance spaCy -2.6262820645794274\nChebyshev distance spaCy 0.958489465713501\nSpearman Ranking Correlation Coeeficient spaCy 0.4607033411482349\nWeighted Pearson Correlation Coefficient spaCy 0.8178629731922643\n**Total Custom Distance Measure Score 1.7666096687316895**\n**Azure Cognitive Search Score 5.5757585**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Kenyon denies Robben Barca return  Chelsea chief executive Peter Kenyon has played down reports that Arjen Robben will return for the Champions League match against Barcelona.  \"He's been responding well to treatment and started running on Friday, but we'll have to wait and see,\" he told BBC Five Live's Sportsweek. \"We're looking to getting him back as soon as possible, but he'll be back when it's right for him and for us. \"There's no plans at the moment around the Barcelona game.\" His comments contradict those of chiropractor Jean Pierre Meersseman who treated the Dutchman after he fractured his foot at the start of February. Robben had been expected to be out for six weeks, but Meersseman hinted that the winger could be fit for the vital Stamford Bridge game on 8 March. \"I hope he can be back and I will try to help him make that happen,\" Meersseman told the Mail on Sunday. \"I put everything right with Arjen's foot the last time I saw him 12 days ago. It was an obvious correction and easy to perform. \"I know he was pleased with what I did and now that he is running again. I am due to see him one more time again in the next few days.\" Meersseman is the medical co-ordinator at Italian side AC Milan.  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.02319514751434326\nFuzz ratio 0\nspaCy 0.5441085140968871\nJaccard distance 0.006535947712418301\nEuclidean distance spaCy 0.7441788759878958\nManhattan distance spaCy -2.579768744409084\nChebyshev distance spaCy 0.959799485206604\nSpearman Ranking Correlation Coeeficient spaCy 0.525475171946355\nWeighted Pearson Correlation Coefficient spaCy 0.7852365235676927\n**Total Custom Distance Measure Score 1.008760929107666**\n**Azure Cognitive Search Score 4.4981585**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Vickery upbeat about arm injury  England prop Phil Vickery is staying positive despite a broken arm ruling him out of the RBS Six Nations.  The 28-year-old fractured the radius in his right forearm during Gloucester's 17-16 win over Bath on Saturday. He will undergo an operation on Monday and is expected to be out for at least six weeks. He said: \"This isn't an injury that will stop me from working hard on the fitness elements and being around the lads.\" He added: \"I've got the operation this afternoon and I could be back doing fitness work after a week.\" \"As frustrating as it is, I've got to be positive.\"  After the game, Vickery spoke with Bath prop David Barnes, who also broke his arm recently. \"I had a chat with David Barnes and it looks like a similar injury to him,\" he said. \"He said he had the operation and he was back running after a week. \"There's no doubt that I'm going to get involved and be around this place as soon as I can after the operation.\" Gloucester director of rugby Nigel Melville said: \"Phil has broken his radius, which is the large bone in his forearm. \"I don't really know how it happened, but Phil will definitely be out of action for at least six weeks. \"I feel very sorry for him, as he has been in great shape. He really needed 80 minutes of rugby this weekend, and then this happened. Mentally, it must be very hard for him.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.7979953289031982\nFuzz ratio 0\nspaCy 0.5652431677987757\nJaccard distance 0.006211180124223602\nEuclidean distance spaCy 0.7491913911921283\nManhattan distance spaCy -2.51705733448267\nChebyshev distance spaCy 0.9626625299453735\nSpearman Ranking Correlation Coeeficient spaCy 0.541580239780442\nWeighted Pearson Correlation Coefficient spaCy 0.7972035895995842\n**Total Custom Distance Measure Score 1.9030301570892334**\n**Azure Cognitive Search Score 4.2438555**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Cole refuses to blame van Persie  Ashley Cole has refused to blame Robin van Persie for leaving Arsenal with no fully-fit strikers for the FA Cup fifth round replay at Sheffield United.  Van Persie is suspended alongside Dennis Bergkamp and Jose Antonio Reyes after being sent off at Southampton when Arsenal had a numerical advantage. Thierry Henry is ruled out with an Achilles tendon injury but Cole said: \"No-one is putting the blame on Robin. \"It's just something that happens on the spur of the moment.\" Cole added: \"I've done it before and I hope they didn't blame me for anything. \"Of course he'll learn. I've been sent off a couple of times now and it's just one of those things when you go a bit crazy for one or two seconds. Freddie Ljungberg is likely to be used in an emergency striking role and will be partnered by either Arturo Lupoli, Quincy Owusu-Abeyie or Jeremie Aliadiere. Gunners boss Arsene Wenger said: \"Freddie is an option but we need a second striker. \"I have to decide whether it will be Aliadiere, Quincy or Lupoli who will start with him up front. Those three will be involved.\" Arsenal are also without winger Robert Pires, who sustained an ankle injury at St Mary's. Wenger added: \"It doesn't look like anything is fractured, but it is a good ankle sprain. \"It does not look like Pires will be ready for two to three weeks.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.286815345287323\nFuzz ratio 0\nspaCy 0.5782458348190921\nJaccard distance 0.0\nEuclidean distance spaCy 0.7683413514889219\nManhattan distance spaCy -2.2697082729265095\nChebyshev distance spaCy 0.9633776330947876\nSpearman Ranking Correlation Coeeficient spaCy 0.5522430249224991\nWeighted Pearson Correlation Coefficient spaCy 0.8017443486362155\n**Total Custom Distance Measure Score 1.68105947971344**\n**Azure Cognitive Search Score 4.167914**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Wales get Williams fitness boost  Wales are hopeful that openside flanker Martyn Williams could be fit for Saturday's RBS 6 Nations championship opener against England in Cardiff.  Williams was expected to miss the match with a disc problem in his neck, but has been making a speedy recovery. \"He will have tests in the next 48 hours and we are pretty optimistic he is getting there,\" Wales' team physiotherapist Mark Davies said. \"It has been frustrating but he is on the mend, he has made good progress.\" Last week Williams, along with fellow flanker Colin Charvis - who is unlikely to play for at least a month while he recovers from a foot injury - was all but ruled out of the Millennium Stadium clash. With Williams initially thought to be struggling, the signs pointed towards Wales coach Mike Ruddock handing a first cap to former Wales Under-21 skipper Richie Pugh.  Cardiff Blues flanker Williams, 29, offers considerable experience and if he is declared fit then Ruddock might be tempted to include him in the back row. Charvis will be reviewed by the Wales medical staff next Monday, but Davies admitted that there was only an \"outside chance\" of him being fit to face France in Wales' third championship game on 26 February. Wales' other injury concern is Pugh's fellow Neath-Swansea Ospreys player Sonny Parker, as the centre has a trapped nerve in his neck. \"Sonny's injury is still an issue,\" Davies said. \"It is still painful and irritable. We will run the rule of thumb over him in the next couple of days.\" Ruddock will name his starting line-up for the England game at 1830 GMT on Tuesday evening, as Wales target their first victory in Cardiff over the world champions since 1993.  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.522243857383728\nFuzz ratio 0\nspaCy 0.5226353966638094\nJaccard distance 0.010582010582010581\nEuclidean distance spaCy 0.7444192930929813\nManhattan distance spaCy -2.5673284966498615\nChebyshev distance spaCy 0.9611926651000977\nSpearman Ranking Correlation Coeeficient spaCy 0.48937921532461465\nWeighted Pearson Correlation Coefficient spaCy 0.62481859178453\n**Total Custom Distance Measure Score 1.307942509651184**\n**Azure Cognitive Search Score 4.147021**\n"
          ]
        }
      ],
      "execution_count": 33,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Now we implement the OpenAI Retrieval Augmented Generation Pattern using Azure Cognitive Search Semantic search and evaluate"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "openai.api_key = openai_api_key\n",
        "openai.api_type = openai_api_type\n",
        "openai.api_base = f\"https://{openai_service}.openai.azure.com\"\n",
        "openai.api_version = openai_api_version\n",
        "\n",
        "def query_semantic(user_input):\n",
        "   \n",
        "    prompt_prefix = \"\"\"<|im_start|>system\n",
        "    Let's work this out it a step by step to be sure we have the right answer\n",
        "\n",
        "    Sources:\n",
        "    {sources}\n",
        "    \n",
        "\n",
        "    <|im_end|>\"\"\"\n",
        "\n",
        "    turn_prefix = \"\"\"\n",
        "    <|im_start|>user\n",
        "    \"\"\"\n",
        "\n",
        "    turn_suffix = \"\"\"\n",
        "    <|im_end|>\n",
        "    <|im_start|>assistant\n",
        "    \"\"\"\n",
        "\n",
        "    prompt_history = turn_prefix\n",
        "\n",
        "    history = []\n",
        "\n",
        "    summary_prompt_template = \"\"\"Below is a summary of the conversation so far, and a new question asked by the user that needs to be answered by searching in a knowledge base. Generate a search query based on the conversation and the new question. Source names are not good search terms to include in the search query.\n",
        "\n",
        "    Summary:\n",
        "    {summary}\n",
        "\n",
        "    Question:\n",
        "    {question}\n",
        "\n",
        "    Search query:\n",
        "    \"\"\"\n",
        "\n",
        "    content = \"\"\n",
        "\n",
        "    # Exclude category, to simulate scenarios where there's a set of docs you can't see\n",
        "    exclude_category = None\n",
        "\n",
        "    if len(history) > 0:\n",
        "        completion = openai.Completion.create(\n",
        "            engine=openai_gpt_deployment,\n",
        "            prompt=summary_prompt_template.format(summary=\"\\n\".join(history), question=user_input),\n",
        "            temperature=0.9,\n",
        "            max_tokens=320,\n",
        "            stop=[\"\\n\"])\n",
        "        search = completion.choices[0].text\n",
        "        print(f\"search {search}\")\n",
        "    else:\n",
        "        search = user_input\n",
        "\n",
        "        # Alternatively simply use search_client.search(q, top=3) if not using semantic search\n",
        "        print(\"Searching:\", search)\n",
        "        print(\"-------------------\")\n",
        "        filter = \"category ne '{}'\".format(exclude_category.replace(\"'\", \"''\")) if exclude_category else None\n",
        "        r = search_client.search(search,\n",
        "                                 filter=filter,\n",
        "                                 query_type=\"semantic\",\n",
        "                                 query_language=\"en-us\",\n",
        "                                 query_speller=\"lexicon\",\n",
        "                                 semantic_configuration_name=semantic_configuration_name,\n",
        "                                 top=5)\n",
        "\n",
        "        content = [doc for doc in r]\n",
        "\n",
        "    prompt = prompt_prefix.format(sources=content) + prompt_history + user_input + turn_suffix\n",
        "\n",
        "    completion = openai.Completion.create(\n",
        "        engine=openai_chatgpt_deployment,\n",
        "        prompt=prompt,\n",
        "        temperature=0.7,\n",
        "        max_tokens=1024,\n",
        "        stop=[\"<|im_end|>\", \"<|im_start|>\"])\n",
        "\n",
        "    prompt_history += user_input + turn_suffix + completion.choices[0].text + \"\\n<|im_end|>\" + turn_prefix\n",
        "    history.append(\"user: \" + user_input)\n",
        "    history.append(\"assistant: \" + completion.choices[0].text)\n",
        "\n",
        "    print(\"\\n-------------------\\n\".join(history))\n",
        "    print(\"\\n-------------------\\nPrompt:\\n\" + prompt)\n",
        "\n",
        "    # Now let's evaluate the results\n",
        "\n",
        "    scores = {}\n",
        "\n",
        "\n",
        "    for i, doc in enumerate(content):\n",
        "\n",
        "        try:\n",
        "\n",
        "            hyp = doc[\"content\"].replace(\"\\n\", \" \")\n",
        "            display(Markdown(f'*Candidate answer: {hyp}*'))\n",
        "            scores[i] = 0\n",
        "\n",
        "            # roBERTA Entailment\n",
        "            device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "            # run through model pre-trained on MNLI\n",
        "            x = tokenizer.encode(term_to_search_for, hyp, return_tensors='pt',\n",
        "                                    truncation_strategy='only_first')\n",
        "            logits = nli_model(x.to(device))[0]\n",
        "\n",
        "            # we throw away \"neutral\" (dim 1) and take the probability of\n",
        "            # \"entailment\" (2) as the probability of the label being true\n",
        "            entail_contradiction_logits = logits[:, [0, 2]]\n",
        "            probs = entail_contradiction_logits.softmax(dim=1)\n",
        "            prob_label_is_true = probs[:, 1]\n",
        "            print(f\"Entailment score {prob_label_is_true[0]}\")\n",
        "            scores[i] += prob_label_is_true[0]\n",
        "\n",
        "            # Fuzz ratio (Levenschein Distance)\n",
        "            score = fuzz.ratio(term_to_search_for, hyp)\n",
        "            print(f\"Fuzz ratio {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # spaCy cosine similarity\n",
        "            doc1 = nlp(term_to_search_for)\n",
        "            doc2 = nlp(hyp)\n",
        "            score = doc1.similarity(doc2)\n",
        "            print(f\"spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Jaccard Distance\n",
        "            score = get_jaccard_sim(term_to_search_for, hyp)\n",
        "            print(f\"Jaccard distance {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Euclidean Distance\n",
        "            score = (100 - euclidean(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "            print(f\"Euclidean distance spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Manhattan Distance\n",
        "            score = (100 - manhattan(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "            print(f\"Manhattan distance spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Chebysev Distance\n",
        "            score = (100 - chebyshev(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "            print(f\"Chebyshev distance spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Spearman Ranking Correlation Coeeficient spaCy\n",
        "            score, _ = spearmanr(nlp(term_to_search_for).vector, nlp(hyp).vector)\n",
        "            print(f\"Spearman Ranking Correlation Coeeficient spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Weighted Pearson Correlation Coefficient spaCy\n",
        "            weights = nlp(hyp).vector.shape[0]\n",
        "            score = corr(nlp(term_to_search_for).vector, nlp(hyp).vector, weights)\n",
        "            print(f\"Weighted Pearson Correlation Coefficient spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            print(f'**Total Custom Distance Measure Score {scores[i]}**')\n",
        "            print(f'**Azure Cognitive Search Score {doc[\"@search.score\"]}**')\n",
        "\n",
        "        except Exception as swallowed:\n",
        "            print(f\"An error occured on record {i} {swallowed}\")\n",
        "            continue\n"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "29",
              "statement_id": 14,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T09:37:47.2831065Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T09:37:47.4370874Z",
              "execution_finish_time": "2023-05-14T09:37:47.6119172Z",
              "spark_jobs": null,
              "parent_msg_id": "6eebaa66-101e-4852-a6ec-32b1fc3fefc7"
            },
            "text/plain": "StatementMeta(DataDiscovery, 29, 14, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 35,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Run the actual query"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "query_semantic(\"Whose thumb was fractured?\")"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "29",
              "statement_id": 15,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T09:37:50.0767087Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T09:37:50.230852Z",
              "execution_finish_time": "2023-05-14T09:38:07.2808499Z",
              "spark_jobs": null,
              "parent_msg_id": "cde4ba46-637c-4bf3-85a7-54e6e904c972"
            },
            "text/plain": "StatementMeta(DataDiscovery, 29, 15, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Searching: Whose thumb was fractured?\n-------------------\nuser: Whose thumb was fractured?\n-------------------\nassistant:  Gareth Thomas' thumb was fractured.\n\n-------------------\nPrompt:\n<|im_start|>system\n    Let's work this out it a step by step to be sure we have the right answer\n\n    Sources:\n    [{'content': 'Thomas out of Six Nations\\n\\nWales captain Gareth Thomas has been ruled out of the rest of the Six Nations with a broken thumb.\\n\\nThe full-back will have surgery on Monday after fracturing his thumb in the 24-18 win over France on Saturday. But Welsh legend Phil Bennett insisted Wales can cope without Thomas as they chase a first Grand Slam in 27 years. Bennett told BBC Sport: \"Such is the spirit in the camp, they\\'ll put Kevin Morgan at 15, Rhys Williams at wing and just carry on.\" Thomas will miss the match against Scotland on 13 March, and what promises to be a huge encounter against the Irish six days later. Bennett added: \"It\\'s a setback. He\\'s a great captain, he leads from the front and the boys love him.\" Thomas was replaced at half-time by Williams as his side turned around a 15-6 deficit in Paris.\\n\\n\"With Gareth missing I would think Michael Owen will be our captain,\" said Wales coach Mike Ruddock. \"He did a great job in the second half in France. He has been vice-captain all along throughout the championship.\" Wales travel to Edinburgh to take on Scotland in a fortnight and then host Ireland in Cardiff in the final round of matches in what could be the Grand Slam and championship decider. Bennett, an inspirational fly-half for Llanelli and Wales in the 1970s, insisted the national team were entering a new golden period. \"It was a great game and a magnificent result for Wales,\" Bennett told BBC Radio Five Live\\'s Sportsweek programme.\\n\\n\"The way this young team are blending, the glory days are on their way back. \"We couldn\\'t get possession early on and France dominated and scored two tries. \"Had they been ruthless, Wales could have gone into the interval 30 points down. But they didn\\'t take their chances. \"Wales defended fairly well but you cannot give that sort of quality ball to good sides. \"The All Blacks would have been ruthless and buried us in the first half. But the character we showed in the second half was quite outstanding.\"\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9ydWdieTAwOC50eHQ1', '@search.score': 5.5757585, '@search.reranker_score': 0.7614212036132812, '@search.highlights': None, '@search.captions': None}, {'content': 'Vickery upbeat about arm injury\\n\\nEngland prop Phil Vickery is staying positive despite a broken arm ruling him out of the RBS Six Nations.\\n\\nThe 28-year-old fractured the radius in his right forearm during Gloucester\\'s 17-16 win over Bath on Saturday. He will undergo an operation on Monday and is expected to be out for at least six weeks. He said: \"This isn\\'t an injury that will stop me from working hard on the fitness elements and being around the lads.\" He added: \"I\\'ve got the operation this afternoon and I could be back doing fitness work after a week.\" \"As frustrating as it is, I\\'ve got to be positive.\"\\n\\nAfter the game, Vickery spoke with Bath prop David Barnes, who also broke his arm recently. \"I had a chat with David Barnes and it looks like a similar injury to him,\" he said. \"He said he had the operation and he was back running after a week. \"There\\'s no doubt that I\\'m going to get involved and be around this place as soon as I can after the operation.\" Gloucester director of rugby Nigel Melville said: \"Phil has broken his radius, which is the large bone in his forearm. \"I don\\'t really know how it happened, but Phil will definitely be out of action for at least six weeks. \"I feel very sorry for him, as he has been in great shape. He really needed 80 minutes of rugby this weekend, and then this happened. Mentally, it must be very hard for him.\"\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9ydWdieTEzOC50eHQ1', '@search.score': 4.2438555, '@search.reranker_score': 0.4542045593261719, '@search.highlights': None, '@search.captions': None}, {'content': 'Costin aims for comeback in 2006\\n\\nJamie Costin should be paralysed.\\n\\nHe says so himself in a matter-of-fact way as he recalls the car accident which occurred nine days before he was scheduled to step out into the Olympic Stadium in Athens for the 50K Walk. There is an ironic chuckle as he talks of his immediate thoughts after a lorry, driving on the wrong side of the road, had ploughed into his rental car. \"I was in a lot of pain and I guessed that one of my toes was broken,\" says the Waterford man. \"But I was thinking maybe with a cortisone injection you never know. \"In my back, it felt as though all the muscles had been ripped off my pelvis but I was thinking maybe we could do something with laser therapy and ultra sound and hopefully I\\'d be able to race.\" It took over 10 hours before Jamie knew with certainty that he would not be competing in his second Olympics. \"My back had been broken in two places and with one of my vertebrae, the bottom part had exploded so I\\'m fierce lucky not be paralysed. \"I\\'d fractured my big toe as well which was on the brake.\" Jamie didn\\'t finally arrive at hospital in Athens until some nine and a half hours after the accident.\\n\\n\"For the first nine hours, I had no pain killers which was ridiculous in 35 degrees heat. \"But once I got the scans and saw them it was a case of moving on and thinking:\\'OK, I\\'ve got a different set of circumstances now\\'.\" Within three days he was arriving back in Ireland by air ambulance. Doctors in Athens had wanted to operate on Jamie\\'s back immediately but he insisted on delaying any surgery until he arrived back home - something he is now very relieved about. \"The Greek doctors were going to put three or four inch titanium rods either side of my spinal cord up through my vertebrae. \"That would have fused all my lower back and I would never have been able to race again. They were really putting a lot of pressure on me to agree to the surgery. \"But when I got to the Mater in Dublin they said it was possible for it to heal totally naturally which is giving me the chance to get back into competition which is very important to me. The people at the Mater have been absolutely fantastic.\" Jamie had to wear a body cast for three and a half months after the accident and spent most of that time flat on his back.\\n\\nHe then progressed to crutches for six weeks until he was finally able to walk unaided on 10 January. \"Walking without the crutches seemed like something finally really measurable in terms of my recovery.\" Physio sessions with Johnston McEvoy in Limerick have been a vital part of his recovery. \"Johnston uses an advanced type of acupuncture and it\\'s very effective. \"Needles get put right close up to my spine. A two and a half inch needle went in yesterday and I\\'m fairly incapacitated today as a result.\" Jamie has also travelled to receive treatment at the Polish training centre in Spala where he has trained with triple Olympic champion Robert Korzeniowski over the past five years. \"I was there for over a fortnight earlier this month and underwent a fair extreme treatment called cryotherapy. \"Basically, there\\'s a small room which is cooled by liquid nitrogen to minus 160 degrees centigrade and it promotes deep healing.\"\\n\\nJamie heads to Poland again on Sunday where he will be having daily cryotherapy in addition to twice-daily physio sessions and pool-work. All these sessions are small steps on the way to what Jamie hopes will be a return to racing in 2006. \"It\\'s all about trying to get mobility in my back. Lying down for three and a half months didn\\'t really help with the strength. \"There\\'s a lot of work involved in my recovery. I\\'m doing about six hours a day between physio and pool work. \"I\\'m also going to the gym to lift very light weights to try and build up my muscles. I\\'m fairly full on with everything I do. \"I\\'d hope to be training regularly by March. But training is just part of the process of getting back. \"At the moment, every time I go and do a big bit of movement, my whole pelvic area all down my lower back just tightens up. \"It\\'s a case of waiting and seeing how it reacts. Hopefully, after four or five months my back won\\'t tighten up as much.\"\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9hdGhlbGV0aWNzMDkzLnR4dA2', '@search.score': 2.8234682, '@search.reranker_score': 0.3481903076171875, '@search.highlights': None, '@search.captions': None}, {'content': 'Owen set for skipper role\\n\\nWales number eight Michael Owen says replacing Gareth Thomas as Wales\\' captain will be straightforward because of the leadership quality in the squad.\\n\\n\"You dream about playing for Wales, but it never enters your head that you will one day captain your country,\" he said. \"It\\'s an easy job really because there are so many experienced players in the side, but it will be a massive honour.\" Coach Mike Ruddock says the 24-year-old should take the armband from broken thumb victim Thomas against Scotland. Ruddock praised Owen\\'s contribution as vice-captain and pack leader in this year\\'s wins over England, Italy and France, notably his role in the second-half in Paris following Thomas\\' injury. The 22-cap Dragons man says that Thomas will be badly missed, though. \"Gareth is going to be a massive loss because he is a brilliant player and he has done a fantastic job since he has been captain,\" said Owen. \"He has put his own unique stamp on things. We are going to miss him but the strength of the squad is shown when you have someone like Kevin Morgan to step in.\"\\n\\nOwen admitted the Grand Slam had been mentioned within the squad but that no-one was getting carried away with the three victories over England, Italy and France. \"We can\\'t get ahead of ourselves,\" he said. \"We have a massive game next up and we are just looking forward to preparing for that. \"Scotland are improving all the time. Like us they did well in France and were unlucky not to get a win. \"Against Ireland they played fantastically well in the first 15 minutes before falling away. \"They are probably going on a similar journey to the one we have been on in the last couple of years.\"\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9ydWdieTA2NS50eHQ1', '@search.score': 3.8952117, '@search.reranker_score': 0.3092231750488281, '@search.highlights': None, '@search.captions': None}, {'content': 'Aussies unhappy with pitch\\n\\nAustralian skipper Ricky Ponting was still able to raise a smile despite his side\\'s 13-run defeat by India in the fourth Test at Mumbai.\\n\\nThey had already done enough to win the Border-Gavaskar Trophy with victories at Bangalore and Nagpur, their first series success in India since 1969-70. \"I said I thought it would be one of the all-time great series and the first three Tests turned out that way. \"This was nowhere near a Test pitch. It was terribly disappointing.\" He was less than thrilled, however, with a pitch in Mumbai which saw 38 wickets fall on the second and third days to bring the match to an early resolution.\\n\\nAustralia only needed 107 in the final innings but were bowled out for 93 in just 30.5 overs as the Indian spinners made the most of conditions. \"It was definitely gettable, but I don\\'t think we played at our best with the bat today. \"Full credit to India though. They batted well in tough conditions and when the game was there to be won with the ball, they did what they had to do.\" Groundsman Polly Umrigar defended his pitch, saying: \"My verdict is that the batsmen did not apply themselves, \"The wicket was not as difficult as it is being made out to be. Batsmen willing to play shots got ample opportunities.\" Ponting also paid tribute to vice-captain Adam Gilchrist, who led the side in the first three Tests while he was recuperating from a broken thumb. \"When you look back through the series, everyone has contributed in one way or another. That\\'s a pleasing thing - it\\'s what you need in a good side,\" he added.\\n\\n\"It\\'s been a great tour. I\\'ve only been here a short time but we\\'ve certainly enjoyed ourselves - I think all of the players really enjoy touring India. \"It\\'s a place we\\'ve been to quite a lot over the last few years. Every time we come back we enjoy it more and more and it certainly helps when you win as well.\" Michael Clarke took 6-9 on the final day and also contributed 400 runs in his first Test series. But the Man of the Series award went to team-mate Damien Martyn, who made 444 runs at an average of 55, although he ended it with a duck. \"I\\'d have liked 10 more runs in the second innings today,\" Martyn joked. \"We came over here to win and fight hard. It was a bit disappointing to finish off that way but we\\'ve won [the series] and the guiys have made a fantastic effort in the last month. \"We always know it\\'s going to be hard over here. It\\'s always a true test for a batsman.\"\\n\\n', 'metadata_storage_path': 'aHR0cHM6Ly9zdHV4Ymp6aXl5MmFyM2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2JiYy9jcmlja2V0MDQ2LnR4dA2', '@search.score': 2.9979866, '@search.reranker_score': 0.2946434020996094, '@search.highlights': None, '@search.captions': None}]\n    \n\n    <|im_end|>\n    <|im_start|>user\n    Whose thumb was fractured?\n    <|im_end|>\n    <|im_start|>assistant\n    \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Thomas out of Six Nations  Wales captain Gareth Thomas has been ruled out of the rest of the Six Nations with a broken thumb.  The full-back will have surgery on Monday after fracturing his thumb in the 24-18 win over France on Saturday. But Welsh legend Phil Bennett insisted Wales can cope without Thomas as they chase a first Grand Slam in 27 years. Bennett told BBC Sport: \"Such is the spirit in the camp, they'll put Kevin Morgan at 15, Rhys Williams at wing and just carry on.\" Thomas will miss the match against Scotland on 13 March, and what promises to be a huge encounter against the Irish six days later. Bennett added: \"It's a setback. He's a great captain, he leads from the front and the boys love him.\" Thomas was replaced at half-time by Williams as his side turned around a 15-6 deficit in Paris.  \"With Gareth missing I would think Michael Owen will be our captain,\" said Wales coach Mike Ruddock. \"He did a great job in the second half in France. He has been vice-captain all along throughout the championship.\" Wales travel to Edinburgh to take on Scotland in a fortnight and then host Ireland in Cardiff in the final round of matches in what could be the Grand Slam and championship decider. Bennett, an inspirational fly-half for Llanelli and Wales in the 1970s, insisted the national team were entering a new golden period. \"It was a great game and a magnificent result for Wales,\" Bennett told BBC Radio Five Live's Sportsweek programme.  \"The way this young team are blending, the glory days are on their way back. \"We couldn't get possession early on and France dominated and scored two tries. \"Had they been ruthless, Wales could have gone into the interval 30 points down. But they didn't take their chances. \"Wales defended fairly well but you cannot give that sort of quality ball to good sides. \"The All Blacks would have been ruthless and buried us in the first half. But the character we showed in the second half was quite outstanding.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2374: FutureWarning: The `truncation_strategy` argument is deprecated and will be removed in a future version, use `truncation=True` to truncate examples to a max length. You can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input size of the model (e.g. 512 for Bert).  If you have pairs of inputs, you can give a specific truncation strategy selected among `truncation='only_first'` (will only truncate the first sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence in the pairs).\n  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.9160066246986389\nFuzz ratio 0\nspaCy 0.4943371490776689\nJaccard distance 0.008771929824561403\nEuclidean distance spaCy 0.7367202900069381\nManhattan distance spaCy -2.6262820645794274\nChebyshev distance spaCy 0.958489465713501\nSpearman Ranking Correlation Coeeficient spaCy 0.4607033411482349\nWeighted Pearson Correlation Coefficient spaCy 0.8178629731922643\n**Total Custom Distance Measure Score 1.7666096687316895**\n**Azure Cognitive Search Score 5.5757585**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Vickery upbeat about arm injury  England prop Phil Vickery is staying positive despite a broken arm ruling him out of the RBS Six Nations.  The 28-year-old fractured the radius in his right forearm during Gloucester's 17-16 win over Bath on Saturday. He will undergo an operation on Monday and is expected to be out for at least six weeks. He said: \"This isn't an injury that will stop me from working hard on the fitness elements and being around the lads.\" He added: \"I've got the operation this afternoon and I could be back doing fitness work after a week.\" \"As frustrating as it is, I've got to be positive.\"  After the game, Vickery spoke with Bath prop David Barnes, who also broke his arm recently. \"I had a chat with David Barnes and it looks like a similar injury to him,\" he said. \"He said he had the operation and he was back running after a week. \"There's no doubt that I'm going to get involved and be around this place as soon as I can after the operation.\" Gloucester director of rugby Nigel Melville said: \"Phil has broken his radius, which is the large bone in his forearm. \"I don't really know how it happened, but Phil will definitely be out of action for at least six weeks. \"I feel very sorry for him, as he has been in great shape. He really needed 80 minutes of rugby this weekend, and then this happened. Mentally, it must be very hard for him.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.7979953289031982\nFuzz ratio 0\nspaCy 0.5652431677987757\nJaccard distance 0.006211180124223602\nEuclidean distance spaCy 0.7491913911921283\nManhattan distance spaCy -2.51705733448267\nChebyshev distance spaCy 0.9626625299453735\nSpearman Ranking Correlation Coeeficient spaCy 0.541580239780442\nWeighted Pearson Correlation Coefficient spaCy 0.7972035895995842\n**Total Custom Distance Measure Score 1.9030301570892334**\n**Azure Cognitive Search Score 4.2438555**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Costin aims for comeback in 2006  Jamie Costin should be paralysed.  He says so himself in a matter-of-fact way as he recalls the car accident which occurred nine days before he was scheduled to step out into the Olympic Stadium in Athens for the 50K Walk. There is an ironic chuckle as he talks of his immediate thoughts after a lorry, driving on the wrong side of the road, had ploughed into his rental car. \"I was in a lot of pain and I guessed that one of my toes was broken,\" says the Waterford man. \"But I was thinking maybe with a cortisone injection you never know. \"In my back, it felt as though all the muscles had been ripped off my pelvis but I was thinking maybe we could do something with laser therapy and ultra sound and hopefully I'd be able to race.\" It took over 10 hours before Jamie knew with certainty that he would not be competing in his second Olympics. \"My back had been broken in two places and with one of my vertebrae, the bottom part had exploded so I'm fierce lucky not be paralysed. \"I'd fractured my big toe as well which was on the brake.\" Jamie didn't finally arrive at hospital in Athens until some nine and a half hours after the accident.  \"For the first nine hours, I had no pain killers which was ridiculous in 35 degrees heat. \"But once I got the scans and saw them it was a case of moving on and thinking:'OK, I've got a different set of circumstances now'.\" Within three days he was arriving back in Ireland by air ambulance. Doctors in Athens had wanted to operate on Jamie's back immediately but he insisted on delaying any surgery until he arrived back home - something he is now very relieved about. \"The Greek doctors were going to put three or four inch titanium rods either side of my spinal cord up through my vertebrae. \"That would have fused all my lower back and I would never have been able to race again. They were really putting a lot of pressure on me to agree to the surgery. \"But when I got to the Mater in Dublin they said it was possible for it to heal totally naturally which is giving me the chance to get back into competition which is very important to me. The people at the Mater have been absolutely fantastic.\" Jamie had to wear a body cast for three and a half months after the accident and spent most of that time flat on his back.  He then progressed to crutches for six weeks until he was finally able to walk unaided on 10 January. \"Walking without the crutches seemed like something finally really measurable in terms of my recovery.\" Physio sessions with Johnston McEvoy in Limerick have been a vital part of his recovery. \"Johnston uses an advanced type of acupuncture and it's very effective. \"Needles get put right close up to my spine. A two and a half inch needle went in yesterday and I'm fairly incapacitated today as a result.\" Jamie has also travelled to receive treatment at the Polish training centre in Spala where he has trained with triple Olympic champion Robert Korzeniowski over the past five years. \"I was there for over a fortnight earlier this month and underwent a fair extreme treatment called cryotherapy. \"Basically, there's a small room which is cooled by liquid nitrogen to minus 160 degrees centigrade and it promotes deep healing.\"  Jamie heads to Poland again on Sunday where he will be having daily cryotherapy in addition to twice-daily physio sessions and pool-work. All these sessions are small steps on the way to what Jamie hopes will be a return to racing in 2006. \"It's all about trying to get mobility in my back. Lying down for three and a half months didn't really help with the strength. \"There's a lot of work involved in my recovery. I'm doing about six hours a day between physio and pool work. \"I'm also going to the gym to lift very light weights to try and build up my muscles. I'm fairly full on with everything I do. \"I'd hope to be training regularly by March. But training is just part of the process of getting back. \"At the moment, every time I go and do a big bit of movement, my whole pelvic area all down my lower back just tightens up. \"It's a case of waiting and seeing how it reacts. Hopefully, after four or five months my back won't tighten up as much.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.796183168888092\nFuzz ratio 0\nspaCy 0.5705528714398947\nJaccard distance 0.002457002457002457\nEuclidean distance spaCy 0.743952954405338\nManhattan distance spaCy -2.4962075752019883\nChebyshev distance spaCy 0.9516753149032593\nSpearman Ranking Correlation Coeeficient spaCy 0.5492176579739774\nWeighted Pearson Correlation Coefficient spaCy 0.7938019818444078\n**Total Custom Distance Measure Score 1.9116334915161133**\n**Azure Cognitive Search Score 2.8234682**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Owen set for skipper role  Wales number eight Michael Owen says replacing Gareth Thomas as Wales' captain will be straightforward because of the leadership quality in the squad.  \"You dream about playing for Wales, but it never enters your head that you will one day captain your country,\" he said. \"It's an easy job really because there are so many experienced players in the side, but it will be a massive honour.\" Coach Mike Ruddock says the 24-year-old should take the armband from broken thumb victim Thomas against Scotland. Ruddock praised Owen's contribution as vice-captain and pack leader in this year's wins over England, Italy and France, notably his role in the second-half in Paris following Thomas' injury. The 22-cap Dragons man says that Thomas will be badly missed, though. \"Gareth is going to be a massive loss because he is a brilliant player and he has done a fantastic job since he has been captain,\" said Owen. \"He has put his own unique stamp on things. We are going to miss him but the strength of the squad is shown when you have someone like Kevin Morgan to step in.\"  Owen admitted the Grand Slam had been mentioned within the squad but that no-one was getting carried away with the three victories over England, Italy and France. \"We can't get ahead of ourselves,\" he said. \"We have a massive game next up and we are just looking forward to preparing for that. \"Scotland are improving all the time. Like us they did well in France and were unlucky not to get a win. \"Against Ireland they played fantastically well in the first 15 minutes before falling away. \"They are probably going on a similar journey to the one we have been on in the last couple of years.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.8652717471122742\nFuzz ratio 0\nspaCy 0.5380841791349585\nJaccard distance 0.010101010101010102\nEuclidean distance spaCy 0.7410224853019125\nManhattan distance spaCy -2.5730651529179887\nChebyshev distance spaCy 0.9595381164550781\nSpearman Ranking Correlation Coeeficient spaCy 0.5125963621818019\nWeighted Pearson Correlation Coefficient spaCy 0.8215269542644464\n**Total Custom Distance Measure Score 1.8750755786895752**\n**Azure Cognitive Search Score 3.8952117**\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: Aussies unhappy with pitch  Australian skipper Ricky Ponting was still able to raise a smile despite his side's 13-run defeat by India in the fourth Test at Mumbai.  They had already done enough to win the Border-Gavaskar Trophy with victories at Bangalore and Nagpur, their first series success in India since 1969-70. \"I said I thought it would be one of the all-time great series and the first three Tests turned out that way. \"This was nowhere near a Test pitch. It was terribly disappointing.\" He was less than thrilled, however, with a pitch in Mumbai which saw 38 wickets fall on the second and third days to bring the match to an early resolution.  Australia only needed 107 in the final innings but were bowled out for 93 in just 30.5 overs as the Indian spinners made the most of conditions. \"It was definitely gettable, but I don't think we played at our best with the bat today. \"Full credit to India though. They batted well in tough conditions and when the game was there to be won with the ball, they did what they had to do.\" Groundsman Polly Umrigar defended his pitch, saying: \"My verdict is that the batsmen did not apply themselves, \"The wicket was not as difficult as it is being made out to be. Batsmen willing to play shots got ample opportunities.\" Ponting also paid tribute to vice-captain Adam Gilchrist, who led the side in the first three Tests while he was recuperating from a broken thumb. \"When you look back through the series, everyone has contributed in one way or another. That's a pleasing thing - it's what you need in a good side,\" he added.  \"It's been a great tour. I've only been here a short time but we've certainly enjoyed ourselves - I think all of the players really enjoy touring India. \"It's a place we've been to quite a lot over the last few years. Every time we come back we enjoy it more and more and it certainly helps when you win as well.\" Michael Clarke took 6-9 on the final day and also contributed 400 runs in his first Test series. But the Man of the Series award went to team-mate Damien Martyn, who made 444 runs at an average of 55, although he ended it with a duck. \"I'd have liked 10 more runs in the second innings today,\" Martyn joked. \"We came over here to win and fight hard. It was a bit disappointing to finish off that way but we've won [the series] and the guiys have made a fantastic effort in the last month. \"We always know it's going to be hard over here. It's always a true test for a batsman.\"  *"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.957051694393158\nFuzz ratio 0\nspaCy 0.5333328745669992\nJaccard distance 0.0035460992907801418\nEuclidean distance spaCy 0.7369176001588964\nManhattan distance spaCy -2.6169634030759337\nChebyshev distance spaCy 0.957062726020813\nSpearman Ranking Correlation Coeeficient spaCy 0.5078425315836842\nWeighted Pearson Correlation Coefficient spaCy 0.8025071997095297\n**Total Custom Distance Measure Score 1.8812973499298096**\n**Azure Cognitive Search Score 2.9979866**\n"
          ]
        }
      ],
      "execution_count": 36,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Now we implement the OpenAI Retrieval Augmented Generation Pattern using simple SQL Synapse/Trident queries and evaluate"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "openai.api_key = openai_api_key\n",
        "openai.api_type = openai_api_type\n",
        "openai.api_base = f\"https://{openai_service}.openai.azure.com\"\n",
        "openai.api_version = openai_api_version\n",
        "\n",
        "\n",
        "def query_sql(user_input, sql_query):\n",
        "    # Use a breakpoint in the code line below to debug your script.\n",
        "    # ChatGPT uses a particular set of tokens to indicate turns in conversations\n",
        "    prompt_prefix = \"\"\"<|im_start|>system\n",
        "    An Assistant that helps the people with their questions regarding sports articles.\n",
        "    Answer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question. \n",
        "    Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brakets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [id][type].\n",
        "\n",
        "    Sources:\n",
        "    {sources}\n",
        "\n",
        "    <|im_end|>\"\"\"\n",
        "\n",
        "\n",
        "    turn_prefix = \"\"\"\n",
        "    <|im_start|>user\n",
        "    \"\"\"\n",
        "\n",
        "    turn_suffix = \"\"\"\n",
        "    <|im_end|>\n",
        "    <|im_start|>assistant\n",
        "    \"\"\"\n",
        "\n",
        "    prompt_history = turn_prefix\n",
        "\n",
        "    history = []\n",
        "\n",
        "    summary_prompt_template = \"\"\"Below is a summary of the conversation so far, and a new question asked by the user that needs to be answered by searching in a knowledge base. Generate a search query based on the conversation and the new question. Source names are not good search terms to include in the search query.\n",
        "\n",
        "    Summary:\n",
        "    {summary}\n",
        "\n",
        "    Question:\n",
        "    {question}\n",
        "\n",
        "    Search query:\n",
        "    \"\"\"\n",
        "\n",
        "    # Execute this cell multiple times updating user_input to accumulate chat history\n",
        "\n",
        "    content = \"\"\n",
        "\n",
        "    # Exclude category, to simulate scenarios where there's a set of docs you can't see\n",
        "    exclude_category = None\n",
        "\n",
        "    if len(history) > 0:\n",
        "        print(f\"search {search}\")\n",
        "    else:\n",
        "        search = user_input\n",
        "\n",
        "\n",
        "        print(\"SQL Searching:\", search)\n",
        "        print(\"-------------------\")\n",
        "        df = spark.sql(sql_query)\n",
        "        dfp = df.toPandas()\n",
        "        lst_query = dfp['text'].to_list()\n",
        "\n",
        "        content = \"\\n\".join(lst_query)\n",
        "\n",
        "    prompt = prompt_prefix.format(sources=content) + prompt_history + user_input + turn_suffix\n",
        "\n",
        "    completion = openai.Completion.create(\n",
        "        engine=openai_chatgpt_deployment,\n",
        "        prompt=prompt,\n",
        "        temperature=0.7,\n",
        "        max_tokens=1024,\n",
        "        stop=[\"<|im_end|>\", \"<|im_start|>\"])\n",
        "\n",
        "    prompt_history += user_input + turn_suffix + completion.choices[0].text + \"\\n<|im_end|>\" + turn_prefix\n",
        "    history.append(\"user: \" + user_input)\n",
        "    history.append(\"assistant: \" + completion.choices[0].text)\n",
        "\n",
        "    print(\"\\n-------------------\\n\".join(history))\n",
        "    print(\"\\n-------------------\\nPrompt:\\n\" + prompt)\n",
        "\n",
        "    # Now let's evaluate the results\n",
        "    scores = {}\n",
        "\n",
        "    for i, doc in enumerate(lst_query):\n",
        "\n",
        "        try:\n",
        "\n",
        "            hyp = doc.replace(\"\\n\", \" \")\n",
        "            display(Markdown(f'*Candidate answer: {hyp}*'))\n",
        "            scores[i] = 0\n",
        "\n",
        "            # roBERTA Entailment\n",
        "            device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "            # run through model pre-trained on MNLI\n",
        "            x = tokenizer.encode(term_to_search_for, hyp, return_tensors='pt',\n",
        "                                    truncation_strategy='only_first')\n",
        "            logits = nli_model(x.to(device))[0]\n",
        "\n",
        "            # we throw away \"neutral\" (dim 1) and take the probability of\n",
        "            # \"entailment\" (2) as the probability of the label being true\n",
        "            entail_contradiction_logits = logits[:, [0, 2]]\n",
        "            probs = entail_contradiction_logits.softmax(dim=1)\n",
        "            prob_label_is_true = probs[:, 1]\n",
        "            print(f\"Entailment score {prob_label_is_true[0]}\")\n",
        "            scores[i] += prob_label_is_true[0]\n",
        "\n",
        "            # Fuzz ratio (Levenschein Distance)\n",
        "            score = fuzz.ratio(term_to_search_for, hyp)\n",
        "            print(f\"Fuzz ratio {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # spaCy cosine similarity\n",
        "            doc1 = nlp(term_to_search_for)\n",
        "            doc2 = nlp(hyp)\n",
        "            score = doc1.similarity(doc2)\n",
        "            print(f\"spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Jaccard Distance\n",
        "            score = get_jaccard_sim(term_to_search_for, hyp)\n",
        "            print(f\"Jaccard distance {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Euclidean Distance\n",
        "            score = (100 - euclidean(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "            print(f\"Euclidean distance spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Manhattan Distance\n",
        "            score = (100 - manhattan(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "            print(f\"Manhattan distance spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Chebysev Distance\n",
        "            score = (100 - chebyshev(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "            print(f\"Chebyshev distance spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Spearman Ranking Correlation Coeeficient spaCy\n",
        "            score, _ = spearmanr(nlp(term_to_search_for).vector, nlp(hyp).vector)\n",
        "            print(f\"Spearman Ranking Correlation Coeeficient spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            # Weighted Pearson Correlation Coefficient spaCy\n",
        "            weights = nlp(hyp).vector.shape[0]\n",
        "            score = corr(nlp(term_to_search_for).vector, nlp(hyp).vector, weights)\n",
        "            print(f\"Weighted Pearson Correlation Coefficient spaCy {score}\")\n",
        "            scores[i] += score\n",
        "\n",
        "            print(f'**Total Custom Distance Measure Score {scores[i]}**')\n",
        "            \n",
        "        except Exception as swallowed:\n",
        "            print(f\"An error occured on record {i} {swallowed}\")\n",
        "            continue\n",
        "\n"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "30",
              "statement_id": 9,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T13:24:29.5772046Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T13:24:29.7391586Z",
              "execution_finish_time": "2023-05-14T13:24:30.0432311Z",
              "spark_jobs": null,
              "parent_msg_id": "23201c22-d45e-426e-966c-b1a539c0380e"
            },
            "text/plain": "StatementMeta(DataDiscovery, 30, 9, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 53,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Now let's go and run a query via a simple SQL statement against all non-stop words"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "user_input = \"Whose thumb was fractured?\"\n",
        "\n",
        "nlp_user_input = nlp(user_input)\n",
        "\n",
        "lst_words = []\n",
        "for token in nlp_user_input:\n",
        "    if token.is_stop == False and len(token.text) > 1:\n",
        "        lst_words.append(token.text)\n",
        "\n",
        "sql_query = \"SELECT text FROM openai_bbc_coords2 WHERE\"\n",
        "\n",
        "suffix = \"\"\n",
        "\n",
        "limit = \" LIMIT 5\"\n",
        "\n",
        "for i, word in enumerate(lst_words):\n",
        "    if i == 0:\n",
        "        suffix += \" text LIKE '%\" + word + \"%'\"  \n",
        "    else: \n",
        "        suffix += \"OR text LIKE '%\" + word + \"%'\"  \n",
        "\n",
        "\n",
        "sql_query = sql_query + suffix + limit\n",
        "print(f\"Manually generated SQL {sql_query}\")\n",
        "query_sql(user_input.lower(), sql_query.lower())"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "DataDiscovery",
              "session_id": "30",
              "statement_id": 20,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-14T13:33:49.9473734Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-14T13:33:50.1009214Z",
              "execution_finish_time": "2023-05-14T13:34:04.0467253Z",
              "spark_jobs": null,
              "parent_msg_id": "f0c516cf-7214-43f8-9986-edebeff9f449"
            },
            "text/plain": "StatementMeta(DataDiscovery, 30, 20, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Manually generated SQL SELECT text FROM openai_bbc_coords2 WHERE text LIKE '%thumb%'OR text LIKE '%fractured%' LIMIT 5\nSQL Searching: whose thumb was fractured?\n-------------------\nuser: whose thumb was fractured?\n-------------------\nassistant:  Sale full-back Jason Robinson's thumb was fractured. [robinson out of six nations]\n\n-------------------\nPrompt:\n<|im_start|>system\n    An Assistant that helps the people with their questions regarding sports articles.\n    Answer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question. \n    Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brakets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [id][type].\n\n    Sources:\n    \"vickery out of six nations  england tight-head prop phil vickery has been ruled out of the rest of the 2005 rbs six nations after breaking a bone in his right forearm.  vickery was injured as his club side, gloucester, beat bath 17-16 in the west country derby on saturday. he could be joined on the sidelines by bath centre olly barkley, who sat out the derby due to a leg injury. barkley will have a scan on sunday and might miss englands trip to six nations leaders ireland next weekend. the news is just the latest blow for coach andy robinson, who has seen his side lose their opening two matches in the 2005 six nations. robinson is already without world cup winners jonny wilkinson, will greenwood, mike tindall, richard hill and trevor woodman through injury. vickery has broken the radius, a large bone in his forearm. he only returned to the england side last weekend after a long-term back injury, which was followed by a fractured eye socket. and the gloucester prop was only recalled after leicester tight-head julian white suffered a neck injury which has already seen him ruled out of the ireland game.  bath prop matt stevens is the only remaining tight-head in englands training squad and could be involved against ireland. but he has to play second fiddle at club level to duncan bell, who excelled for england a against france and may now be called into the squad. the extent of barkleys injury is not yet clear but bath boss john connolly rates him no better than \"\"50-50\"\" to face ireland. barkley played at inside cente in englands defeat by france and if he is unable to play\nvaughan endures day to forget  this was englands worst day for a long time, and was capped by captain michael vaughan being fined 100% of his match fee.  match referee clive lloyds decision followed criticism from vaughan of the umpires handling of the bad light decisions on the second day.  the bowling throughout the day was awful and, two balls from the close, geraint jones dived directly in front of marcus trescothick at first slip and dropped herschelle gibbs on 136. having declared, and sacrificed the possibility of reaching a century, michael vaughan must have been horrified by the manner in which his bowlers responded. briefly the conditions were ideal, with low cloud hanging over the ground, and although this was soon replaced by clear blue sky which made batting easier, the direction of steve harmison in particular was woeful. his first eight overs conceded only nine runs, simply because the batsmen could not reach the ball. far too many deliveries went down the leg-side and even matthew hoggard was below his best, four wickets notwithstanding.  understandably james anderson was rusty having not played a match since 4 december but harmisons day was made complete when he hobbled off with a calf strain. graeme smith and gibbs got away to a flier but were separated in the 16th over when smith was trapped lbw to hoggard for 29. this has been a disappointing series so far for the south african captain and his demise was all the more embarrassing because he fell over and was flat on the ground when umpire bucknor finally raised his painfully slow finger. the bowling was still erratic, but hoggard struck again six overs later when rudolph laced a wide half volley to gully where ashley giles took a stinging catch that dislocated his right thumb.  jacques kallis again looked very well set, and the stand with gibbs during the afternoon was looking ominous until hoggard found some extra bounce and kallis, aiming to force off the back foot, chopped the ball into his stumps for 33. boeta dippenaar is one of the least exciting batsmen currently in test cricket, and after 37 minutes, he fell to a low catch at slip for a duck. the ball was so low to the ground when trescothick took it that dippenaar was reluctant to go. tv replays - which are only used now when the umpires are unsighted - would probably have given the batsman the benefit of the doubt, but mr dar sent him on his way to a chorus of boos from the crowd. ab devilliers joined gibbs, who was now strangely becalmed and only 65 runs were scored in the afternoon session, but after devilliers hooked hoggard and was caught at long leg for 19, gibbs reached his 14th century in almost five and a half hours. the second new ball failed to force an immediate breakthrough as gibbs and boucher launched a counter attack and added 120 crucial runs before boucher was caught at point for 64.\n\"vickery upbeat about arm injury  england prop phil vickery is staying positive despite a broken arm ruling him out of the rbs six nations.  the 28-year-old fractured the radius in his right forearm during gloucesters 17-16 win over bath on saturday. he will undergo an operation on monday and is expected to be out for at least six weeks. he said: \"\"this isnt an injury that will stop me from working hard on the fitness elements and being around the lads.\"\" he added: \"\"ive got the operation this afternoon and i could be back doing fitness work after a week.\"\" \"\"as frustrating as it is\n\"robinson out of six nations  england captain jason robinson will miss the rest of the six nations because of injury.  robinson, stand-in captain in the absence of jonny wilkinson, had been due to lead england in their final two games against italy and scotland. but the sale full-back pulled out of the squad on wednesday because of a torn ligament in his right thumb. the 30-year-old will undergo an operation on friday but england have yet to name a replacement skipper.  robinson said: \"\"this is very disappointing for me as this means i miss englands last two games in the six nations at twickenham and two games for my club\n\"celts savour grand slam prospect  the six nations has heralded a new order in northern hemisphere rugby this year and wales and ireland rather than traditional big guns france and england face a potential grand slam play-off in three weeks time.  but before that game in cardiff, wales must get past scotland at murrayfield, while ireland face the not insignificant task of a home fixture with the mercurial french. no-one knows what mood france will be in at lansdowne road on 12 march - sublime, as in the first half against wales, or ridiculous, like in the same period against england at twickenham. but how the mighty have fallen. england sat on rugbys summit 15 months ago as world champions and 2003 grand slam winners. but they have lost nine of their 14 matches since that heady night in sydney. and they face the ignominy of what could amount to a wooden spoon play-off against italy in a fortnight.  england are enduring their worst run in the championship since captain richard hill was dumped in favour of mike harrison after three straight losses in 1987. coach andy robinson, who took over from the successful sir clive woodward in september, has lost a phalanx of world cup stars. and he is enduring the toughest of teething problems in bedding down his own style with a new team. the same year that england ruled the roost, a woeful wales lost all five matches in the six nations. and they won only two games, against scotland and italy, in 2004. wales most recent championship title was in 1994, and their last grand slam success came in 1978 in the era of gareth edwards, phil bennett, jpr williams et al.  but welsh rugby fans remain on permanent tenterhooks for the blossoming of a new golden age. after several false dawns, coach mike ruddock may have come up with the team and philosophy to match expectations. the fresh verve is inspired by skipper gareth thomas, now out with a broken thumb, accurate kicking from either fly-half stephen jones or centre gavin henson, a rampant martyn williams leading the way up front, and exciting runners in the guise of henson and shane williams. ireland coach eddie osullivan and captain brian odriscoll have got their side buzzing too, and they are close to shedding the \"\"nearly-men\"\" tag that has dogged them for the past few years. the men from the emerald isle have been six nations runners-up for the past two years\n\n    <|im_end|>\n    <|im_start|>user\n    whose thumb was fractured?\n    <|im_end|>\n    <|im_start|>assistant\n    \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: \"vickery out of six nations  england tight-head prop phil vickery has been ruled out of the rest of the 2005 rbs six nations after breaking a bone in his right forearm.  vickery was injured as his club side, gloucester, beat bath 17-16 in the west country derby on saturday. he could be joined on the sidelines by bath centre olly barkley, who sat out the derby due to a leg injury. barkley will have a scan on sunday and might miss englands trip to six nations leaders ireland next weekend. the news is just the latest blow for coach andy robinson, who has seen his side lose their opening two matches in the 2005 six nations. robinson is already without world cup winners jonny wilkinson, will greenwood, mike tindall, richard hill and trevor woodman through injury. vickery has broken the radius, a large bone in his forearm. he only returned to the england side last weekend after a long-term back injury, which was followed by a fractured eye socket. and the gloucester prop was only recalled after leicester tight-head julian white suffered a neck injury which has already seen him ruled out of the ireland game.  bath prop matt stevens is the only remaining tight-head in englands training squad and could be involved against ireland. but he has to play second fiddle at club level to duncan bell, who excelled for england a against france and may now be called into the squad. the extent of barkleys injury is not yet clear but bath boss john connolly rates him no better than \"\"50-50\"\" to face ireland. barkley played at inside cente in englands defeat by france and if he is unable to play*"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2374: FutureWarning: The `truncation_strategy` argument is deprecated and will be removed in a future version, use `truncation=True` to truncate examples to a max length. You can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input size of the model (e.g. 512 for Bert).  If you have pairs of inputs, you can give a specific truncation strategy selected among `truncation='only_first'` (will only truncate the first sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence in the pairs).\n  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.6948408484458923\nFuzz ratio 0\nspaCy 0.11682736922266071\nJaccard distance 0.0\nEuclidean distance spaCy 0.5402123237740829\nManhattan distance spaCy -5.359750669971109\nChebyshev distance spaCy 0.9260266780853271\nSpearman Ranking Correlation Coeeficient spaCy 0.13969313874446476\nWeighted Pearson Correlation Coefficient spaCy 0.460615075852421\n**Total Custom Distance Measure Score -2.481534957885742**\nAn error occured on record 0 string indices must be integers\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: vaughan endures day to forget  this was englands worst day for a long time, and was capped by captain michael vaughan being fined 100% of his match fee.  match referee clive lloyds decision followed criticism from vaughan of the umpires handling of the bad light decisions on the second day.  the bowling throughout the day was awful and, two balls from the close, geraint jones dived directly in front of marcus trescothick at first slip and dropped herschelle gibbs on 136. having declared, and sacrificed the possibility of reaching a century, michael vaughan must have been horrified by the manner in which his bowlers responded. briefly the conditions were ideal, with low cloud hanging over the ground, and although this was soon replaced by clear blue sky which made batting easier, the direction of steve harmison in particular was woeful. his first eight overs conceded only nine runs, simply because the batsmen could not reach the ball. far too many deliveries went down the leg-side and even matthew hoggard was below his best, four wickets notwithstanding.  understandably james anderson was rusty having not played a match since 4 december but harmisons day was made complete when he hobbled off with a calf strain. graeme smith and gibbs got away to a flier but were separated in the 16th over when smith was trapped lbw to hoggard for 29. this has been a disappointing series so far for the south african captain and his demise was all the more embarrassing because he fell over and was flat on the ground when umpire bucknor finally raised his painfully slow finger. the bowling was still erratic, but hoggard struck again six overs later when rudolph laced a wide half volley to gully where ashley giles took a stinging catch that dislocated his right thumb.  jacques kallis again looked very well set, and the stand with gibbs during the afternoon was looking ominous until hoggard found some extra bounce and kallis, aiming to force off the back foot, chopped the ball into his stumps for 33. boeta dippenaar is one of the least exciting batsmen currently in test cricket, and after 37 minutes, he fell to a low catch at slip for a duck. the ball was so low to the ground when trescothick took it that dippenaar was reluctant to go. tv replays - which are only used now when the umpires are unsighted - would probably have given the batsman the benefit of the doubt, but mr dar sent him on his way to a chorus of boos from the crowd. ab devilliers joined gibbs, who was now strangely becalmed and only 65 runs were scored in the afternoon session, but after devilliers hooked hoggard and was caught at long leg for 19, gibbs reached his 14th century in almost five and a half hours. the second new ball failed to force an immediate breakthrough as gibbs and boucher launched a counter attack and added 120 crucial runs before boucher was caught at point for 64.*"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.004894575569778681\nFuzz ratio 0\nspaCy 0.11007811490426679\nJaccard distance 0.0\nEuclidean distance spaCy 0.535874579591665\nManhattan distance spaCy -5.399358833134174\nChebyshev distance spaCy 0.927357840538025\nSpearman Ranking Correlation Coeeficient spaCy 0.12781367077716085\nWeighted Pearson Correlation Coefficient spaCy -0.6612726507519184\n**Total Custom Distance Measure Score -4.354612350463867**\nAn error occured on record 1 string indices must be integers\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: \"vickery upbeat about arm injury  england prop phil vickery is staying positive despite a broken arm ruling him out of the rbs six nations.  the 28-year-old fractured the radius in his right forearm during gloucesters 17-16 win over bath on saturday. he will undergo an operation on monday and is expected to be out for at least six weeks. he said: \"\"this isnt an injury that will stop me from working hard on the fitness elements and being around the lads.\"\" he added: \"\"ive got the operation this afternoon and i could be back doing fitness work after a week.\"\" \"\"as frustrating as it is*"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.07602415233850479\nFuzz ratio 0\nspaCy 0.14530868070409286\nJaccard distance 0.0\nEuclidean distance spaCy 0.5504621554459502\nManhattan distance spaCy -5.180435049086809\nChebyshev distance spaCy 0.9251292705535888\nSpearman Ranking Correlation Coeeficient spaCy 0.16190672383176297\nWeighted Pearson Correlation Coefficient spaCy -0.9346487536211675\n**Total Custom Distance Measure Score -4.256253242492676**\nAn error occured on record 2 string indices must be integers\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: \"robinson out of six nations  england captain jason robinson will miss the rest of the six nations because of injury.  robinson, stand-in captain in the absence of jonny wilkinson, had been due to lead england in their final two games against italy and scotland. but the sale full-back pulled out of the squad on wednesday because of a torn ligament in his right thumb. the 30-year-old will undergo an operation on friday but england have yet to name a replacement skipper.  robinson said: \"\"this is very disappointing for me as this means i miss englands last two games in the six nations at twickenham and two games for my club*"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.5634950995445251\nFuzz ratio 0\nspaCy 0.0692187180187271\nJaccard distance 0.0\nEuclidean distance spaCy 0.5207955293268097\nManhattan distance spaCy -5.641562014892697\nChebyshev distance spaCy 0.9209987497329712\nSpearman Ranking Correlation Coeeficient spaCy 0.08046957865522673\nWeighted Pearson Correlation Coefficient spaCy 0.9843386537867757\n**Total Custom Distance Measure Score -2.5022454261779785**\nAn error occured on record 3 string indices must be integers\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<IPython.core.display.Markdown object>",
            "text/markdown": "*Candidate answer: \"celts savour grand slam prospect  the six nations has heralded a new order in northern hemisphere rugby this year and wales and ireland rather than traditional big guns france and england face a potential grand slam play-off in three weeks time.  but before that game in cardiff, wales must get past scotland at murrayfield, while ireland face the not insignificant task of a home fixture with the mercurial french. no-one knows what mood france will be in at lansdowne road on 12 march - sublime, as in the first half against wales, or ridiculous, like in the same period against england at twickenham. but how the mighty have fallen. england sat on rugbys summit 15 months ago as world champions and 2003 grand slam winners. but they have lost nine of their 14 matches since that heady night in sydney. and they face the ignominy of what could amount to a wooden spoon play-off against italy in a fortnight.  england are enduring their worst run in the championship since captain richard hill was dumped in favour of mike harrison after three straight losses in 1987. coach andy robinson, who took over from the successful sir clive woodward in september, has lost a phalanx of world cup stars. and he is enduring the toughest of teething problems in bedding down his own style with a new team. the same year that england ruled the roost, a woeful wales lost all five matches in the six nations. and they won only two games, against scotland and italy, in 2004. wales most recent championship title was in 1994, and their last grand slam success came in 1978 in the era of gareth edwards, phil bennett, jpr williams et al.  but welsh rugby fans remain on permanent tenterhooks for the blossoming of a new golden age. after several false dawns, coach mike ruddock may have come up with the team and philosophy to match expectations. the fresh verve is inspired by skipper gareth thomas, now out with a broken thumb, accurate kicking from either fly-half stephen jones or centre gavin henson, a rampant martyn williams leading the way up front, and exciting runners in the guise of henson and shane williams. ireland coach eddie osullivan and captain brian odriscoll have got their side buzzing too, and they are close to shedding the \"\"nearly-men\"\" tag that has dogged them for the past few years. the men from the emerald isle have been six nations runners-up for the past two years*"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Entailment score 0.15633727610111237\nFuzz ratio 0\nspaCy 0.06291821753591906\nJaccard distance 0.0\nEuclidean distance spaCy 0.5269126043771132\nManhattan distance spaCy -5.577703699916601\nChebyshev distance spaCy 0.9270708322525024\nSpearman Ranking Correlation Coeeficient spaCy 0.07906156269771251\nWeighted Pearson Correlation Coefficient spaCy 0.6606345910166012\n**Total Custom Distance Measure Score -3.1647684574127197**\nAn error occured on record 4 string indices must be integers\n"
          ]
        }
      ],
      "execution_count": 64,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Categoriser - Let's use OpenAI to label the cluster data for us so that we have a dataset for training an ML model - only works if the clustering step was run"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "user_input = \"Categorise these documents by the subject they have in common\"\n",
        "\n",
        "sql_query = \"SELECT text FROM openai_bbc_coords_vectors WHERE prediction == 4\"\n",
        "\n",
        "suffix = \"\"\n",
        "\n",
        "limit = \" LIMIT 10\"\n",
        "\n",
        "\n",
        "sql_query = sql_query + suffix + limit\n",
        "print(sql_query)\n",
        "query_sql(user_input.lower(), sql_query.lower())\n",
        "\n",
        "\n",
        "# Now the output from the assistant can be used to update the data in the SQL table for that prediction/cluster\n",
        "\n"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Sample:  Use CODEX to generate the SQL"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "user_input = \"Who was the England Captain\"\n",
        "prompt = f\"\"\"\n",
        "Table openai_bbc_coords2, columns = [text]\n",
        "Create a SQL query for all text that contains the phrase {user_input}\n",
        "\"\"\"\n",
        "\n",
        "completion = openai.Completion.create(\n",
        "        engine=openai_code_deployment,\n",
        "        prompt=prompt,\n",
        "        temperature=0.3,\n",
        "        max_tokens=1024,\n",
        "        stop=[\"<|im_end|>\", \"<|im_start|>\"])\n",
        "completion.choices[0].text"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": null,
              "session_id": "8eed670e-02ed-4000-b875-9b4e4156292d",
              "statement_id": 24,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-03T08:42:16.3830454Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-03T08:42:16.7297373Z",
              "execution_finish_time": "2023-05-03T08:42:34.0923315Z",
              "spark_jobs": {
                "numbers": {
                  "UNKNOWN": 0,
                  "FAILED": 0,
                  "SUCCEEDED": 0,
                  "RUNNING": 0
                },
                "jobs": [],
                "limit": 20,
                "rule": "ALL_DESC"
              },
              "parent_msg_id": "f72a10c8-a70b-4842-a54e-c8e6c0623a9c"
            },
            "text/plain": "StatementMeta(, 8eed670e-02ed-4000-b875-9b4e4156292d, 24, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "execution_count": 43,
          "data": {
            "text/plain": "'\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_bbc_coords2, columns = [text]\\nCreate a SQL query for all text that contains the phrase Who was the England Captain\\n\\nTable DataDiscovery.openai_'"
          },
          "metadata": {}
        }
      ],
      "execution_count": 20,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Get most similar document from any input string against the corpus using OpenAI embeddings - - only works if the clustering step was run"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "input_text = \"\"\"The manager of Ireland was quoted\"\"\"\n",
        "\n",
        "# Get the embedding for the input text\n",
        "input_vector = openai_embeddings(input_text)\n",
        "\n",
        "# Get all values in a list - obviously not good for large datasets\n",
        "df = spark.sql(\"SELECT * FROM openai_bbc_coords_vectors LIMIT 1000\")\n",
        "# Move to list\n",
        "candidates_list = df.select(\"OpenAI_features\").rdd.flatMap(lambda x: x).collect()\n",
        "\n",
        "# Create torch tensors\n",
        "st_candidate_embedding = []\n",
        "for query_embedding in candidates_list:\n",
        "    torch_tensor = torch.Tensor(query_embedding)\n",
        "    st_candidate_embedding.append(torch_tensor)\n",
        "    \n",
        "st_candidate_tensor = torch.stack(st_candidate_embedding)\n",
        "\n",
        "# Now for the input string\n",
        "ip = input_vector.toArray().tolist()\n",
        "input_tensor = torch.Tensor(ip)\n",
        "\n",
        "st_section_topk = {}\n",
        "st_section_score_topk = {}\n",
        "\n",
        "\n",
        "# Reshape tensors and get the cosine similarity\n",
        "for i, section_tensor in enumerate(st_candidate_tensor):\n",
        "\n",
        "    section_tensor = section_tensor.unsqueeze(0)\n",
        "    input_tensor = input_tensor.squeeze(0).squeeze(0)\n",
        "    cosine_distance = F.cosine_similarity(input_tensor, section_tensor)\n",
        "    st_section_topk[i] = cosine_distance\n",
        "\n",
        "# Display the top 5 most similar documents\n",
        "sim_res = sorted(st_section_topk.items(), key=lambda kv: kv[1], reverse=True)\n",
        "\n",
        "scores = {}\n",
        "\n",
        "for i in range(5):\n",
        "\n",
        "    df = spark.sql(f\"SELECT _c0, text FROM openai_bbc_coords_vectors where _c0 == {sim_res[i][0]}\")\n",
        "    lst_candidate = df.select(\"text\").rdd.flatMap(lambda x: x).collect()\n",
        "\n",
        "    hyp = lst_candidate[0].replace(\"\\n\", \" \")\n",
        "    display(Markdown(f'*Candidate answer: {hyp}*'))\n",
        "    scores[i] = 0\n",
        "\n",
        "    # roBERTA Entailment\n",
        "    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "    # run through model pre-trained on MNLI\n",
        "    x = tokenizer.encode(term_to_search_for, hyp, return_tensors='pt',\n",
        "                            truncation_strategy='only_first')\n",
        "    logits = nli_model(x.to(device))[0]\n",
        "\n",
        "    # we throw away \"neutral\" (dim 1) and take the probability of\n",
        "    # \"entailment\" (2) as the probability of the label being true\n",
        "    entail_contradiction_logits = logits[:, [0, 2]]\n",
        "    probs = entail_contradiction_logits.softmax(dim=1)\n",
        "    prob_label_is_true = probs[:, 1]\n",
        "    print(f\"Entailment score {prob_label_is_true[0]}\")\n",
        "    scores[i] += prob_label_is_true[0]\n",
        "\n",
        "    # Fuzz ratio (Levenschein Distance)\n",
        "    score = fuzz.ratio(term_to_search_for, hyp)\n",
        "    print(f\"Fuzz ratio {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # spaCy cosine similarity\n",
        "    doc1 = nlp(term_to_search_for)\n",
        "    doc2 = nlp(hyp)\n",
        "    score = doc1.similarity(doc2)\n",
        "    print(f\"spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Jaccard Distance\n",
        "    score = get_jaccard_sim(term_to_search_for, hyp)\n",
        "    print(f\"Jaccard distance {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Euclidean Distance\n",
        "    score = (100 - euclidean(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "    print(f\"Euclidean distance spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Manhattan Distance\n",
        "    score = (100 - manhattan(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "    print(f\"Manhattan distance spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Chebysev Distance\n",
        "    score = (100 - chebyshev(nlp(term_to_search_for).vector, nlp(hyp).vector)) / 100\n",
        "    print(f\"Chebyshev distance spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Spearman Ranking Correlation Coeeficient spaCy\n",
        "    score, _ = spearmanr(nlp(term_to_search_for).vector, nlp(hyp).vector)\n",
        "    print(f\"Spearman Ranking Correlation Coeeficient spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    # Weighted Pearson Correlation Coefficient spaCy\n",
        "    weights = nlp(hyp).vector.shape[0]\n",
        "    score = corr(nlp(term_to_search_for).vector, nlp(hyp).vector, weights)\n",
        "    print(f\"Weighted Pearson Correlation Coefficient spaCy {score}\")\n",
        "    scores[i] += score\n",
        "\n",
        "    print(f'**Total Custom Distance Measure Score {scores[i]}**')\n",
        "    print(f'**Azure Cognitive Search Score {doc[\"@search.score\"]}**')\n"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": null,
              "session_id": "5870d75e-7834-4fec-a8b7-a007a675f119",
              "statement_id": 44,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2023-05-04T11:51:33.4622961Z",
              "session_start_time": null,
              "execution_start_time": "2023-05-04T11:51:33.751818Z",
              "execution_finish_time": "2023-05-04T11:51:37.5226089Z",
              "spark_jobs": {
                "numbers": {
                  "FAILED": 0,
                  "SUCCEEDED": 0,
                  "UNKNOWN": 0,
                  "RUNNING": 0
                },
                "jobs": [],
                "limit": 20,
                "rule": "ALL_DESC"
              },
              "parent_msg_id": "64cec91a-c935-404d-b0e6-e8505833a26c"
            },
            "text/plain": "StatementMeta(, 5870d75e-7834-4fec-a8b7-a007a675f119, 44, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 40,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Optional: Build the Concept Graph"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Optional: Amend this section to build your concept graph"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_graph = spark.sql(\"SELECT * FROM openai_bbc_coords2 LIMIT 1000\")\n",
        "\n",
        "lst_text = df_graph.select('text').rdd.flatMap(lambda x: x).collect()\n",
        "\n",
        "lst_source_node = []\n",
        "lst_source_node_weight = []\n",
        "lst_source_node_label = []\n",
        "lst_target_node = []\n",
        "lst_target_node_weight = []\n",
        "lst_target_node_label = []\n",
        "lst_source_url = []\n",
        "lst_target_url = []\n",
        "lst_edge_weight = []\n",
        "lst_edge_colour_weight = []\n",
        "\n",
        "lst_g_nodes = []\n",
        "lst_g_edges = []\n",
        "\n",
        "dict_nodes = {}\n",
        "\n",
        "from itertools import combinations\n",
        "\n",
        "for i, row in enumerate(lst_text):\n",
        "    \n",
        "    combos = list(combinations(row, 2))\n",
        "   \n",
        "    for c in combos:\n",
        "        # First update edge weights\n",
        "        if (c[0] + \"_\" + c[1] not in dict_nodes) and (c[1] + \"_\" + c[0] not in dict_nodes):\n",
        "            dict_nodes[c[0] + \"_\" + c[1]] = 1 # initialise and create first combo\n",
        "        elif c[0] + \"_\" + c[1] in dict_nodes:\n",
        "            dict_nodes[c[0] + \"_\" + c[1]] += 1\n",
        "        elif c[1] + \"_\" + c[0] in dict_nodes:\n",
        "            dict_nodes[c[1] + \"_\" + c[0]] += 1\n",
        "\n",
        "    for c in combos:\n",
        "        lst_source_node.append(c[0])\n",
        "        lst_g_nodes.append((c[0], c[0]))\n",
        "        lst_target_node.append((c[1]))\n",
        "        lst_g_nodes.append((c[1], c[1]))\n",
        "        if c[0] + \"_\" + c[1] in dict_nodes:\n",
        "            lst_edge_weight.append(dict_nodes[c[0] + \"_\" + c[1]])\n",
        "            lst_source_node_weight.append(dict_nodes[c[0] + \"_\" + c[1]])\n",
        "            lst_target_node_weight.append(dict_nodes[c[0] + \"_\" + c[1]])\n",
        "            lst_g_edges.append((c[0],c[1], \"related\"))\n",
        "        else:\n",
        "            lst_edge_weight.append(dict_nodes[c[1] + \"_\" + c[0]])\n",
        "            lst_source_node_weight.append(dict_nodes[c[1] + \"_\" + c[0]])\n",
        "            lst_target_node_weight.append(dict_nodes[c[1] + \"_\" + c[0]])\n",
        "\n",
        "topn = dict(sorted(dict_nodes.items(), key = itemgetter(1), reverse = True)[:number_of_connected_nodes])\n",
        "\n",
        "# Assign edge weight colour\n",
        "for key in zip(lst_source_node, lst_target_node):\n",
        "    \n",
        "    if key[0] + \"_\" + key[1] in topn or key[1] + \"_\" + key[0] in topn:\n",
        "        lst_edge_colour_weight.append(\"red\")\n",
        "    else:\n",
        "        lst_edge_colour_weight.append(\"black\")\n",
        "\n",
        "\n",
        "# Create the Graph RDD\n",
        "columns = ['source', 'target', 'source_node_weight', 'target_node_weight', 'edge_weight', 'edge_colour']\n",
        "df_concept_graph = spark.createDataFrame(zip(lst_source_node, lst_target_node, lst_source_node_weight, lst_target_node_weight, lst_edge_weight, lst_edge_colour_weight), columns)\n",
        "  \n",
        "# Create a Vertex DataFrame with unique ID column \"id\"\n",
        "v = sqlContext.createDataFrame(lst_g_nodes, [\"id\", \"name\"])\n",
        "# Create an Edge DataFrame with \"src\" and \"dst\" columns\n",
        "e = sqlContext.createDataFrame(lst_g_edges, [\"src\", \"dst\", \"relationship\"])"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Show degree connectivity"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from graphframes import GraphFrame\n",
        "g = GraphFrame(v, e)\n",
        "\n",
        "# Query: Get in-degree of each vertex.\n",
        "df_degree = g.inDegrees\n",
        "df_degree.sort(['inDegree'], ascending=False).show()"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": null,
              "session_id": null,
              "statement_id": null,
              "state": "cancelled",
              "livy_statement_state": null,
              "queued_time": "2023-03-20T11:57:51.011691Z",
              "session_start_time": "2023-03-20T11:57:51.0662228Z",
              "execution_start_time": null,
              "execution_finish_time": "2023-03-20T11:57:51.8160924Z",
              "spark_jobs": null,
              "parent_msg_id": "aef23c4b-a479-4fee-9950-7c676f02dd9f"
            },
            "text/plain": "StatementMeta(, , , Cancelled, )"
          },
          "metadata": {}
        }
      ],
      "execution_count": 1,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Run PageRank "
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Query: Count the number of \"follow\" connections in the graph.\n",
        "g.edges.filter(\"relationship = 'relationship'\").count()\n",
        "\n",
        "# Run PageRank algorithm, and show results.\n",
        "results = g.pageRank(resetProbability=0.01, maxIter=20)\n",
        "results.vertices.select(\"id\", \"pagerank\").show()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Save the outcome with two operations (usual or coalesce) to display in the PowerBI Network Navigator"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "df_concept_graph.write.mode('overwrite').options(header='true').csv(os.path.join(output_directory, output_filename[:-4] + \"concept_graph.csv\"))"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "synapse_pyspark",
      "language": "Python",
      "display_name": "Synapse PySpark"
    },
    "language_info": {
      "name": "python"
    },
    "kernel_info": {
      "name": "synapse_pyspark"
    },
    "description": null,
    "save_output": true,
    "synapse_widget": {
      "version": "0.1",
      "state": {}
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}