intro_workshop_finished.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a7e8c835-8f00-4000-b458-a30a7d6a5cbe",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Weaviate](./img/01-cover.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "636dcf47-bc04-4900-923f-cce2a71036c4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Weaviate](./img/02-about-weaviate.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b264e3e3-55bb-4293-9d47-64f30cc34b1e",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Agenda:\n",
    "\n",
    "#### What you will see:\n",
    "\n",
    "- Examples of AI-powered searches\n",
    "- Create and build a vector database\n",
    "- Search with a vector database\n",
    "- Retrieval augmented generation (RAG)\n",
    "- Scalability considerations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eec4eec-eb4d-4397-a26b-f1bd546b52ae",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### You will learn:\n",
    "\n",
    "- About vector, keyword & hybrid searches\n",
    "    - When to use each one\n",
    "- How to perform RAG\n",
    "- How to build a scalable vector DB"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "893cf35f-fe83-4292-91ff-53bed6282f4b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Search: An Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7025930-ad27-4e49-a479-e17e5c1a7878",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Try searches using this (pre-populated) toy dataset. \n",
    "\n",
    "```json\n",
    "animal_objs = [\n",
    "    {\"description\": \"brown dog\"},\n",
    "    {\"description\": \"small domestic black cat\"},\n",
    "    {\"description\": \"orange cheetah\"},\n",
    "    {\"description\": \"black bear\"},\n",
    "    {\"description\": \"large white seagull\"},\n",
    "    {\"description\": \"yellow canary\"},\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "85bd1b33-db5d-4982-932b-01d90c5c4e5e",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BatchObjectReturn(_all_responses=[UUID('36416803-a5c5-45ef-8de5-dfe619830c24'), UUID('2d1945c4-6b1b-408d-b0e1-d04fc17dc3ef'), UUID('f4f5b6c0-f2d5-4db0-925b-cfcd2d9f60cb'), UUID('b5ce2c1f-9c45-4e1d-95a9-dc9ac17a74ee'), UUID('34a2db49-607a-428d-92e5-0f4d2356a85e'), UUID('98980a42-9342-48d5-84d4-f3aee4bd3718')], elapsed_seconds=0.9282820224761963, errors={}, uuids={0: UUID('36416803-a5c5-45ef-8de5-dfe619830c24'), 1: UUID('2d1945c4-6b1b-408d-b0e1-d04fc17dc3ef'), 2: UUID('f4f5b6c0-f2d5-4db0-925b-cfcd2d9f60cb'), 3: UUID('b5ce2c1f-9c45-4e1d-95a9-dc9ac17a74ee'), 4: UUID('34a2db49-607a-428d-92e5-0f4d2356a85e'), 5: UUID('98980a42-9342-48d5-84d4-f3aee4bd3718')}, has_errors=False)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ================================================================================\n",
    "# Prep script: Just run the cell for now - don't worry about the details\n",
    "# ================================================================================\n",
    "#\n",
    "# This script:\n",
    "#     connects to Weaviate,\n",
    "#     creates a collection,\n",
    "#     and populates it with the above demo dataset\n",
    "#\n",
    "# ================================================================================\n",
    "\n",
    "\n",
    "import weaviate\n",
    "from weaviate.classes.config import Configure, Property, DataType\n",
    "import os\n",
    "\n",
    "client = weaviate.connect_to_local()\n",
    "\n",
    "# Work with Weaviate\n",
    "\n",
    "animals = client.collections.delete(\"Animals\")\n",
    "\n",
    "animals = client.collections.create(\n",
    "    name=\"Animals\",\n",
    "    properties=[\n",
    "        Property(name=\"description\", data_type=DataType.TEXT),\n",
    "    ],\n",
    "    vectorizer_config=[\n",
    "        Configure.NamedVectors.text2vec_ollama(\n",
    "            name=\"description\",\n",
    "            source_properties=[\"description\"],\n",
    "            api_endpoint=\"http://host.docker.internal:11434\",  # If using Docker, use this to contact your local Ollama instance\n",
    "            model=\"nomic-embed-text\",  # The model to use, e.g. \"nomic-embed-text\"\n",
    "        )\n",
    "    ],\n",
    "    generative_config=Configure.Generative.ollama(\n",
    "        api_endpoint=\"http://host.docker.internal:11434\",  # If using Docker, use this to contact your local Ollama instance\n",
    "        model=\"gemma2:2b\"\n",
    "    ),\n",
    ")\n",
    "\n",
    "animal_objs = [\n",
    "    {\"description\": \"brown dog\"},\n",
    "    {\"description\": \"small domestic black cat\"},\n",
    "    {\"description\": \"orange cheetah\"},\n",
    "    {\"description\": \"black bear\"},\n",
    "    {\"description\": \"large white seagull\"},\n",
    "    {\"description\": \"yellow canary\"},\n",
    "]\n",
    "\n",
    "animals.data.insert_many(animal_objs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c63f3336-2a24-4b5b-8f99-485b77d0a714",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Traditional search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "92aff4d9-5c2a-4bf2-8cc9-d3b111772185",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 results returned:\n",
      "{'description': 'small domestic black cat'}\n"
     ]
    }
   ],
   "source": [
    "query = \"cat\"\n",
    "\n",
    "response = animals.query.bm25(query)\n",
    "\n",
    "print(f\"{len(response.objects)} results returned:\")\n",
    "for o in response.objects:\n",
    "    print(o.properties)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4be21eea-3eb8-452e-a472-652e7ff9bb38",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "But, traditional searches are not very robust. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c2d2c0f4-a845-423e-b8a2-5e6404f8c40e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 results returned:\n"
     ]
    }
   ],
   "source": [
    "query = \"kitty\"  # Try synonyms or even typos\n",
    "\n",
    "response = animals.query.bm25(query)\n",
    "\n",
    "print(f\"{len(response.objects)} results returned:\")\n",
    "for o in response.objects:\n",
    "    print(o.properties)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0118ba5-97a0-4241-a811-be2ddc1dd12c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Vector search\n",
    "\n",
    "But vector search is based on similarity, allowing more forgiving, nuanced search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a648b534-4029-4e4b-ae5e-e460b53533ed",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6 results returned:\n",
      "{'description': 'small domestic black cat'}\n",
      "{'description': 'orange cheetah'}\n",
      "{'description': 'yellow canary'}\n",
      "{'description': 'black bear'}\n",
      "{'description': 'large white seagull'}\n",
      "{'description': 'brown dog'}\n"
     ]
    }
   ],
   "source": [
    "query = \"cat\"\n",
    "\n",
    "response = animals.query.near_text(query)\n",
    "\n",
    "print(f\"{len(response.objects)} results returned:\")\n",
    "for o in response.objects:\n",
    "    print(o.properties)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e29faa18-fa8c-498a-90d4-26f80d40fd33",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6 results returned:\n",
      "{'description': 'small domestic black cat'}\n",
      "{'description': 'orange cheetah'}\n",
      "{'description': 'black bear'}\n",
      "{'description': 'yellow canary'}\n",
      "{'description': 'large white seagull'}\n",
      "{'description': 'brown dog'}\n"
     ]
    }
   ],
   "source": [
    "query = \"kitty\"  # Try synonyms or even typos\n",
    "\n",
    "response = animals.query.near_text(query)\n",
    "\n",
    "print(f\"{len(response.objects)} results returned:\")\n",
    "for o in response.objects:\n",
    "    print(o.properties)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aedde2c7-b92d-4655-8674-db494c1820b6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Vector searches provide forgiving, nuanced, meaning-based similarity search. \n",
    "\n",
    "But - what is a vector?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e86500e-c446-4fbe-916d-fb7af40c1b3b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Introduction to Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ede4c4b4-ffdf-4158-9511-8bfca1545d54",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Vectors](./img/04-vectors-intro-01.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbe73634-dcc2-4a3d-a4c0-4302645c1dc9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Vectors](./img/04-vectors-intro-02.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd06fb45-051e-4888-94ca-79bc87351777",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Vectors](./img/04-vectors-intro-03.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbbfcdc2-dc5a-41bf-89f6-7b32ef920307",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Vectors](./img/04-vectors-intro-04.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a7e0df1-6a07-4ec9-bfd4-d5e9e8bd3466",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to vectors](./img/04-vectors-intro-05.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9997230d-44a1-4240-8d4e-78f190d5a744",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Why use vector search?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4379e327-97f7-4f81-8773-ccae4c5f9f77",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- Better search\n",
    "    - Find contextually relevant info\n",
    "    - Allow synonyms, different languages\n",
    "    - More value from data\n",
    "- Work together with generative AI models\n",
    "    - Overcome hallucinations or lack of specific / prioprietary information"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d342e374-696f-4515-a761-c5c9e02bd778",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to RAG](./img/06-rag-intro-01.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3022a135-ba46-4139-b9b4-f5f408308e61",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to RAG](./img/06-rag-intro-02.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e3957db-7597-4cc1-8412-21c1caeee0d8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to RAG](./img/06-rag-intro-03.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6ed68ab-da1d-4e61-aa69-4ddc52c9f02d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to RAG](./img/06-rag-intro-04.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "602cab9b-b77f-4003-89ef-42330d06daa3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Example RAG prompts:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f1179f5-37b6-49ad-b0be-6ae8a37603f7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- Summarise the corporate strategy of ACME Co for FY2024-25.\n",
    "- What is our internal policy on food expenses?\n",
    "- What smartphone issues do users commonly complain about?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6c26e12-8b9e-4e6b-8bfd-216058190421",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "#### 🤔 How can we find data for these prompts with *just* keyword searches?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b13d58-99ad-40f5-9dcf-a1d5fc2a54ed",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "It's very difficult!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90018033-2823-4d77-bed1-9bb0b4ccf337",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Example:\n",
    "\n",
    "#### `What smartphone issues do users commonly complain about?`\n",
    "\n",
    "How would you search for \"smartphone\" issues in your data?\n",
    "\n",
    "- \"*phone*\"?\n",
    "- \"tablet\"?\n",
    "- \"android\" and \"iphone\"?\n",
    "- Include every smartphone maker, model and name?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ad2f6cb-6cf6-4fe6-afb5-b25c1ae9e6cb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Example:\n",
    "\n",
    "#### `What smartphone issues do users commonly complain about?`\n",
    "\n",
    "How would you search for \"smartphone\" issues in your data?\n",
    "\n",
    "With vector DBs - you can just use \"smartphone\" because semantic search takes these into account."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed2d4e72-4d39-48dc-89a5-8ab34162b6f0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to RAG](./img/06-rag-intro-05.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0393379-2b17-4029-875b-6405982825bf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Weaviate in practice"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24fc8e5f-7e4e-4bfc-91c5-90740c49bc41",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Build a database"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ba26bc3-0958-4051-b31e-9ce27555a8b7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Preparation: Get the data\n",
    "\n",
    "We'll use a dataset of movies from TMDB. Let's download the data, and preview it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4680be3b-e00f-4c3c-9d42-d9bd8ef1e5a9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-12T20:50:09.186617Z",
     "start_time": "2023-08-12T20:50:09.042514Z"
    },
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 1192,
    "lastExecutedAt": 1692032184432,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "import requests\nimport json\n\n# Load the data locally\nwith open(\"jeopardy_1k.json\", \"r\") as f:\n    raw_data = f.read()\n\n# Or download it from GitHub\nresponse = requests.get('https://raw.githubusercontent.com/databyjp/wv_demo_uploader/main/weaviate_datasets/data/jeopardy_1k.json')\nraw_data = response.text\n\n# Parse the JSON and preview it\ndata = json.loads(raw_data)\nprint(type(data), len(data))\nprint(json.dumps(data[0], indent=2))",
    "outputsMetadata": {
     "0": {
      "height": 217,
      "type": "stream"
     }
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>backdrop_path</th>\n",
       "      <th>genre_ids</th>\n",
       "      <th>id</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>poster_path</th>\n",
       "      <th>release_date</th>\n",
       "      <th>title</th>\n",
       "      <th>video</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>/rH0DPF7pB35jxLxKb3JRUgCrrnp.jpg</td>\n",
       "      <td>[10751, 14, 16, 10749]</td>\n",
       "      <td>11224</td>\n",
       "      <td>en</td>\n",
       "      <td>Cinderella</td>\n",
       "      <td>Cinderella has faith her dreams of a better li...</td>\n",
       "      <td>100.819</td>\n",
       "      <td>/avz6S9HYWs4O8Oe4PenBFNX4uDi.jpg</td>\n",
       "      <td>1950-02-22</td>\n",
       "      <td>Cinderella</td>\n",
       "      <td>False</td>\n",
       "      <td>7.044</td>\n",
       "      <td>6523</td>\n",
       "      <td>1950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>/p47ihFj4A7EpBjmPHdTj4ipyq1S.jpg</td>\n",
       "      <td>[18]</td>\n",
       "      <td>599</td>\n",
       "      <td>en</td>\n",
       "      <td>Sunset Boulevard</td>\n",
       "      <td>A hack screenwriter writes a screenplay for a ...</td>\n",
       "      <td>57.740</td>\n",
       "      <td>/sC4Dpmn87oz9AuxZ15Lmip0Ftgr.jpg</td>\n",
       "      <td>1950-08-10</td>\n",
       "      <td>Sunset Boulevard</td>\n",
       "      <td>False</td>\n",
       "      <td>8.312</td>\n",
       "      <td>2485</td>\n",
       "      <td>1950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>/zyO6j74DKMWfp5snWg6Hwo0T3Mz.jpg</td>\n",
       "      <td>[80, 18, 9648]</td>\n",
       "      <td>548</td>\n",
       "      <td>ja</td>\n",
       "      <td>羅生門</td>\n",
       "      <td>Brimming with action while incisively examinin...</td>\n",
       "      <td>21.011</td>\n",
       "      <td>/vL7Xw04nFMHwnvXRFCmYYAzMUvY.jpg</td>\n",
       "      <td>1950-08-26</td>\n",
       "      <td>Rashomon</td>\n",
       "      <td>False</td>\n",
       "      <td>8.091</td>\n",
       "      <td>2121</td>\n",
       "      <td>1950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>/b4yiLlIFuiULuuLTxT0Pt1QyT6J.jpg</td>\n",
       "      <td>[16, 10751, 14, 12]</td>\n",
       "      <td>12092</td>\n",
       "      <td>en</td>\n",
       "      <td>Alice in Wonderland</td>\n",
       "      <td>On a golden afternoon, young Alice follows a W...</td>\n",
       "      <td>75.465</td>\n",
       "      <td>/20cvfwfaFqNbe9Fc3VEHJuPRxmn.jpg</td>\n",
       "      <td>1951-07-28</td>\n",
       "      <td>Alice in Wonderland</td>\n",
       "      <td>False</td>\n",
       "      <td>7.200</td>\n",
       "      <td>5697</td>\n",
       "      <td>1951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>/mxf8hJJkHTCqZP3m4o8E1TtwHHs.jpg</td>\n",
       "      <td>[35, 10749]</td>\n",
       "      <td>872</td>\n",
       "      <td>en</td>\n",
       "      <td>Singin' in the Rain</td>\n",
       "      <td>In 1927 Hollywood, a silent film production co...</td>\n",
       "      <td>31.407</td>\n",
       "      <td>/w03EiJVHP8Un77boQeE7hg9DVdU.jpg</td>\n",
       "      <td>1952-04-09</td>\n",
       "      <td>Singin' in the Rain</td>\n",
       "      <td>False</td>\n",
       "      <td>8.200</td>\n",
       "      <td>3036</td>\n",
       "      <td>1952</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      backdrop_path               genre_ids     id  \\\n",
       "0  /rH0DPF7pB35jxLxKb3JRUgCrrnp.jpg  [10751, 14, 16, 10749]  11224   \n",
       "1  /p47ihFj4A7EpBjmPHdTj4ipyq1S.jpg                    [18]    599   \n",
       "2  /zyO6j74DKMWfp5snWg6Hwo0T3Mz.jpg          [80, 18, 9648]    548   \n",
       "3  /b4yiLlIFuiULuuLTxT0Pt1QyT6J.jpg     [16, 10751, 14, 12]  12092   \n",
       "4  /mxf8hJJkHTCqZP3m4o8E1TtwHHs.jpg             [35, 10749]    872   \n",
       "\n",
       "  original_language       original_title  \\\n",
       "0                en           Cinderella   \n",
       "1                en     Sunset Boulevard   \n",
       "2                ja                  羅生門   \n",
       "3                en  Alice in Wonderland   \n",
       "4                en  Singin' in the Rain   \n",
       "\n",
       "                                            overview  popularity  \\\n",
       "0  Cinderella has faith her dreams of a better li...     100.819   \n",
       "1  A hack screenwriter writes a screenplay for a ...      57.740   \n",
       "2  Brimming with action while incisively examinin...      21.011   \n",
       "3  On a golden afternoon, young Alice follows a W...      75.465   \n",
       "4  In 1927 Hollywood, a silent film production co...      31.407   \n",
       "\n",
       "                        poster_path release_date                title  video  \\\n",
       "0  /avz6S9HYWs4O8Oe4PenBFNX4uDi.jpg   1950-02-22           Cinderella  False   \n",
       "1  /sC4Dpmn87oz9AuxZ15Lmip0Ftgr.jpg   1950-08-10     Sunset Boulevard  False   \n",
       "2  /vL7Xw04nFMHwnvXRFCmYYAzMUvY.jpg   1950-08-26             Rashomon  False   \n",
       "3  /20cvfwfaFqNbe9Fc3VEHJuPRxmn.jpg   1951-07-28  Alice in Wonderland  False   \n",
       "4  /w03EiJVHP8Un77boQeE7hg9DVdU.jpg   1952-04-09  Singin' in the Rain  False   \n",
       "\n",
       "   vote_average  vote_count  year  \n",
       "0         7.044        6523  1950  \n",
       "1         8.312        2485  1950  \n",
       "2         8.091        2121  1950  \n",
       "3         7.200        5697  1951  \n",
       "4         8.200        3036  1952  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# movie_df = pd.read_csv(\"./data/movies.csv\")\n",
    "movie_df = pd.read_csv(\"https://raw.githubusercontent.com/weaviate-tutorials/intro-workshop/main/data/movies.csv\")\n",
    "movie_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d27642a1-e738-441c-9008-ce8cb267f28b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Step 1: Connect to Weaviate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a88ba7-6e35-4378-ad30-869537966f01",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "You can run Weaviate as:\n",
    "\n",
    "- a hosted instance on Weaviate Cloud, or\n",
    "- install Weaviate anywhere using the open-source distribution, such as on AWS, GCP, etc., or locally. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a8e3e2a-6727-48ba-99cc-77b7331e7d4e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Today, we will run a local instance on our own devices with Docker."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f4936a3-1ef5-47b9-88e8-71e96bed66a7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "In fact - you should have Weaviate running already (see the workshop README file). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ba40b5a3-d645-493b-9c93-e652c1057870",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-12T20:50:10.463278Z",
     "start_time": "2023-08-12T20:50:10.020181Z"
    },
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": null,
    "lastExecutedAt": null,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": null,
    "outputsMetadata": {
     "0": {
      "height": 97,
      "type": "stream"
     },
     "1": {
      "height": 257,
      "type": "stream"
     }
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import weaviate\n",
    "\n",
    "# If you have got Weaviate running locally Docker:\n",
    "client = weaviate.connect_to_local()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "388d970d-79a2-4591-a5f3-c6c17406df2a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Retrieve Weaviate instance information to check our configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b8a4226e-41fd-4ad7-9416-dcda4a443d58",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-12T20:50:11.726853Z",
     "start_time": "2023-08-12T20:50:11.722015Z"
    },
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 48,
    "lastExecutedAt": 1692032185634,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "jprint(client.get_meta())",
    "outputsMetadata": {
     "0": {
      "height": 597,
      "type": "stream"
     }
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.is_ready()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68b9706e-e11e-43f8-ae13-3741c5376b25",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Step 2: Add data to Weaviate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f01b498-455f-4e82-bdb3-df7b95363ca4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Add collection definition"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "310a7ecf-0f04-48a3-b622-dd3673e8302f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "The equivalent of a SQL \"table\" is called a \"collection\" in Weaviate."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9279cefa-ecaf-487c-a090-43a14383e9a1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "We'll create a new collection definition here for \"Movie\":\n",
    "- Two \"named vectors\" -> which will save different \"meanings\" of the data,\n",
    "- A \"generative\" module -> which will allow us to use LLMs with our data, and\n",
    "- Properties to save our movie data (which are like SQL columns).\n",
    "    - Just the title, overview, year and popularity for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5da0db4a-9f72-402c-9c23-49f1500ea995",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-12T20:50:24.018171Z",
     "start_time": "2023-08-12T20:50:23.993529Z"
    },
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": null,
    "lastExecutedAt": null,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": null,
    "outputsMetadata": {
     "0": {
      "height": 77,
      "type": "stream"
     }
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<weaviate.collections.collection.sync.Collection at 0x1102ecc50>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from weaviate.classes.config import Configure, DataType, Property\n",
    "\n",
    "# DO NOT DO THIS IN PRODUCTION - THIS IS TO DELETE DATA FROM MY PREVIOUS DEMOS\n",
    "if client.collections.exists(\"Movie\"):\n",
    "    client.collections.delete(\"Movie\")\n",
    "\n",
    "# Create a collection\n",
    "client.collections.create(\n",
    "    name=\"Movie\",\n",
    "    # ================================================================================\n",
    "    # Using our Ollama integration: https://weaviate.io/developers/weaviate/model-providers/ollama\n",
    "    # Many other integrations available. See https://weaviate.io/developers/weaviate/model-providers/\n",
    "    # ================================================================================\n",
    "    vectorizer_config=[\n",
    "        Configure.NamedVectors.text2vec_ollama(\n",
    "            name=\"title\",\n",
    "            source_properties=[\"title\"],\n",
    "            api_endpoint=\"http://host.docker.internal:11434\",  # If using Docker, use this to contact your local Ollama instance\n",
    "            model=\"nomic-embed-text\",  # The model to use, e.g. \"snowflake-arctic-embed\"\n",
    "        ),\n",
    "        Configure.NamedVectors.text2vec_ollama(\n",
    "            name=\"all_text\",\n",
    "            source_properties=[\"title\", \"overview\"],\n",
    "            api_endpoint=\"http://host.docker.internal:11434\",  # If using Docker, use this to contact your local Ollama instance\n",
    "            model=\"nomic-embed-text\",  # The model to use, e.g. \"snowflake-arctic-embed\"\n",
    "        ),\n",
    "    ],\n",
    "    generative_config=Configure.Generative.ollama(\n",
    "        api_endpoint=\"http://host.docker.internal:11434\",\n",
    "        model=\"gemma2:2b\"\n",
    "    ),\n",
    "    # ================================================================================\n",
    "    # OPTIONAL - SPECIFY YOUR DATA SCHEMA OR HAVE IT INFERRED BY WEAVIATE\n",
    "    # ================================================================================\n",
    "    # properties=[\n",
    "    #     Property(\n",
    "    #         name=\"title\",\n",
    "    #         data_type=DataType.TEXT,\n",
    "    #     ),\n",
    "    #     Property(\n",
    "    #         name=\"overview\",\n",
    "    #         data_type=DataType.TEXT,\n",
    "    #     ),\n",
    "    #     Property(\n",
    "    #         name=\"popularity\",\n",
    "    #         data_type=DataType.NUMBER,\n",
    "    #     ),\n",
    "    #     Property(\n",
    "    #         name=\"year\",\n",
    "    #         data_type=DataType.INT,\n",
    "    #     ),\n",
    "    # ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd17a8e9-da25-4619-9e37-0787383dafe1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Was our collection created successfully? Let's take a look"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cfed74df-84a7-43a6-bd79-960b0681be7e",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 48,
    "lastExecutedAt": 1692032185778,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "jprint(client.schema.get(\"Question\"))",
    "outputsMetadata": {
     "0": {
      "height": 616,
      "type": "stream"
     }
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.collections.exists(\"Movie\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1ec4752-6428-4db0-b5b3-c36db1b6caaf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Add data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d809f0b6-d816-448a-9619-b8f660b0eef7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Let's add objects (SQL rows) to our data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "92d8d242",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UUID('7dadf9df-fed7-430b-acb0-399e8045e3ae')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies = client.collections.get(\"Movie\")\n",
    "\n",
    "movies.data.insert({\"title\": \"SpongeBob in vector space\", \"year\": 2055})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8738e941-dbec-40c1-a2cc-e0b4429910ab",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(movies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ecd6f59f-2dda-4996-966f-6cd1c52b60c3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BatchObjectReturn(_all_responses=[UUID('1a87ea83-3da5-4bd9-8319-5a141dc58273'), UUID('6f23b768-ff86-45c3-aa64-b1eae0c71703'), UUID('7a3696bc-d373-4982-a346-969bb918e9ce'), UUID('8319a603-4f02-40fe-b3bb-03640047c59e'), UUID('40e813ff-3393-4f92-8f39-773abbd11797')], elapsed_seconds=0.14514994621276855, errors={}, uuids={0: UUID('1a87ea83-3da5-4bd9-8319-5a141dc58273'), 1: UUID('6f23b768-ff86-45c3-aa64-b1eae0c71703'), 2: UUID('7a3696bc-d373-4982-a346-969bb918e9ce'), 3: UUID('8319a603-4f02-40fe-b3bb-03640047c59e'), 4: UUID('40e813ff-3393-4f92-8f39-773abbd11797')}, has_errors=False)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_movies_to_add = [\n",
    "    {\"title\": f\"SpongeBob in vector space {i+2}\", \"year\": 2056 + i} for i in range(5)\n",
    "]\n",
    "\n",
    "movies.data.insert_many(new_movies_to_add)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "776c0960-f7b8-4768-b8c3-fcfc2362f255",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(movies)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd6162e3-4a51-41a7-96a5-ac0049dec09f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "But... what if we have lots of data? (like our movies)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36544a4e-84d7-4ae0-997b-3599f24342e2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Use `batch` imports!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2adf2de-b9e5-4d9b-876d-bf4af926ef8a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Batch imports"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2940cda1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "First, let's build objects to add - and take a look at a couple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a4453f38-67ac-4de6-9f3d-decc3527149a",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 47,
    "lastExecutedAt": 1692032185826,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "for o in data[:2]:\n    obj_body = {\n        \"question\": o[\"Question\"],\n        \"answer\": o[\"Answer\"],\n    }\n    print(obj_body)",
    "outputsMetadata": {
     "0": {
      "height": 97,
      "type": "stream"
     }
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>overview</th>\n",
       "      <th>year</th>\n",
       "      <th>popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cinderella</td>\n",
       "      <td>Cinderella has faith her dreams of a better li...</td>\n",
       "      <td>1950</td>\n",
       "      <td>100.819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Sunset Boulevard</td>\n",
       "      <td>A hack screenwriter writes a screenplay for a ...</td>\n",
       "      <td>1950</td>\n",
       "      <td>57.740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Rashomon</td>\n",
       "      <td>Brimming with action while incisively examinin...</td>\n",
       "      <td>1950</td>\n",
       "      <td>21.011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Alice in Wonderland</td>\n",
       "      <td>On a golden afternoon, young Alice follows a W...</td>\n",
       "      <td>1951</td>\n",
       "      <td>75.465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Singin' in the Rain</td>\n",
       "      <td>In 1927 Hollywood, a silent film production co...</td>\n",
       "      <td>1952</td>\n",
       "      <td>31.407</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 title                                           overview  \\\n",
       "0           Cinderella  Cinderella has faith her dreams of a better li...   \n",
       "1     Sunset Boulevard  A hack screenwriter writes a screenplay for a ...   \n",
       "2             Rashomon  Brimming with action while incisively examinin...   \n",
       "3  Alice in Wonderland  On a golden afternoon, young Alice follows a W...   \n",
       "4  Singin' in the Rain  In 1927 Hollywood, a silent film production co...   \n",
       "\n",
       "   year  popularity  \n",
       "0  1950     100.819  \n",
       "1  1950      57.740  \n",
       "2  1950      21.011  \n",
       "3  1951      75.465  \n",
       "4  1952      31.407  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_columns = ['title', 'overview', 'year', 'popularity']\n",
    "\n",
    "df = movie_df[data_columns]\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0d65333-ce25-4833-92b7-57ca8113bdb2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "> If it all looks fine - let's add objects:\n",
    "> - https://weaviate.io/developers/weaviate/manage-data/import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1b445597-3655-4606-99e8-8398c50b632d",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 6734,
    "lastExecutedAt": 1692032192560,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "with client.batch() as batch:\n    for o in data:\n        obj_body = {\n            \"question\": o[\"Question\"],\n            \"answer\": o[\"Answer\"],\n        }\n        batch.add_data_object(\n            data_object=obj_body,\n            class_name=\"Question\"\n        )",
    "outputsMetadata": {
     "0": {
      "height": 97,
      "type": "stream"
     }
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1322it [00:17, 76.63it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "with movies.batch.fixed_size(200) as batch:\n",
    "    for i, row in tqdm(df.iterrows()):\n",
    "        obj_body = {\n",
    "            c: row[c] for c in data_columns\n",
    "        }\n",
    "        batch.add_object(\n",
    "            properties=obj_body\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "216d67ac-f92d-4239-b1cb-8c76ea2cf6ca",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Confirm data load\n",
    "\n",
    "Do we have data? \n",
    "\n",
    "Let's get an object count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "748cf869-67cd-4b39-b230-23b043217aa3",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 48,
    "lastExecutedAt": 1692032192610,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "jprint(client.query.aggregate(\"Question\").with_meta_count().do())",
    "outputsMetadata": {
     "0": {
      "height": 277,
      "type": "stream"
     }
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1328\n"
     ]
    }
   ],
   "source": [
    "print(len(movies))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1233f9b5-5676-4dc3-bcc8-86df0f86553f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Does the data look right?\n",
    "\n",
    "Let's grab a few objects from Weaviate!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "146e3376-9b3d-4cd6-b27f-e835a9f4eb76",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'overview': 'In October of 1994 three student filmmakers disappeared in the woods near Burkittsville, Maryland, while shooting a documentary. A year later their footage was found.', 'year': 1999.0, 'title': 'The Blair Witch Project', 'popularity': 87.325}\n",
      "{'overview': \"In director Baz Luhrmann's contemporary take on William Shakespeare's classic tragedy, the Montagues and Capulets have moved their ongoing feud to the sweltering suburb of Verona Beach, where Romeo and Juliet fall in love and secretly wed. Though the film is visually modern, the bard's dialogue remains.\", 'year': 1996.0, 'title': 'Romeo + Juliet', 'popularity': 31.488}\n",
      "{'year': 1981.0, 'title': 'An American Werewolf in London', 'overview': 'American tourists David and Jack are savaged by an unidentified vicious animal whilst hiking on the Yorkshire Moors. Retiring to the home of a beautiful nurse to recuperate, David soon experiences disturbing changes to his mind and body.', 'popularity': 40.785}\n"
     ]
    }
   ],
   "source": [
    "response = movies.query.fetch_objects(limit=3)\n",
    "for o in response.objects:\n",
    "    print(o.properties)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "663c77a3-b0f2-429b-b2fe-78cef9d6ac40",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Let's pause for a second - because we've done a lot!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4b1dbad-dfa7-4f88-a5f5-ea6a0f39e413",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### What did we just do?\n",
    "\n",
    "Here is a conceptual diagram"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "436522dc-8cd0-47c4-8286-f839e5a785d4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![img](https://github.com/weaviate-tutorials/intro-workshop/blob/main/images/object_import_process_full.png?raw=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ac15074-c5f6-48e9-9950-35ee265e14d0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Step 3: Work with the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38c5e47f-c408-4fc4-83f8-4e133d040b3b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Let's try a few more involved queries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd5740b2-2846-4462-b140-1b1b9c4544ec",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Filtering (similar to WHERE filter in SQL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c640bf-9f35-4d1e-96e5-455b2fb69fde",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "A filter reduces the number of objects based on specific criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "db451d4b-fcdf-440f-8502-854d970c98d6",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 48,
    "lastExecutedAt": 1692032192706,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "where_filter = {\n    \"path\": [\"question\"],\n    \"operator\": \"Like\",\n    \"valueText\": \"*history*\"\n}\n\nresponse = (\n    client.query\n    .get(\"Question\", [\"question\", \"answer\"])\n    .with_where(where_filter)\n    .with_limit(3)\n    .do()\n)\n\njprint(response)",
    "outputsMetadata": {
     "0": {
      "height": 477,
      "type": "stream"
     }
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jackie\n",
      "The Magnificent Seven\n",
      "Fantastic Beasts and Where to Find Them\n"
     ]
    }
   ],
   "source": [
    "from weaviate.classes.query import Filter\n",
    "\n",
    "response = movies.query.fetch_objects(\n",
    "    filters=Filter.by_property(\"year\").greater_than(2015),\n",
    "    limit=3\n",
    ")\n",
    "\n",
    "for o in response.objects:\n",
    "    print(o.properties[\"title\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4bd3737-068a-4a88-8386-0148999947ed",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "But this does not rank the result in any meaningful way. \n",
    "\n",
    "For that, we need a keyword search (as opposed to a keyword *filter*)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6553f176-47b5-4c2b-88e2-5c8fea71fe7e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Keyword search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7a3d7c7-d376-465e-8a23-841203d54055",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Keyword search ranks results based on keyword match \"scores\", according to the BM25 algorithm. These scores are based on how often tokens in the query appear in each data object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "62d78900-199a-4b8d-8da1-3223c3ada047",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 52,
    "lastExecutedAt": 1692032192810,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "response = (\n    client.query\n    .get(\"Question\", [\"question\", \"answer\"])\n    .with_bm25(\"history\")\n    .with_limit(3)\n    .do()\n)\n\njprint(response)",
    "outputsMetadata": {
     "0": {
      "height": 477,
      "type": "stream"
     }
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.2652101516723633\n",
      "2024-09-17 08:46:24.670000+00:00\n",
      "{'year': 2017.0, 'title': 'Guardians of the Galaxy Vol. 2', 'overview': \"The Guardians must fight to keep their newfound family together as they unravel the mysteries of Peter Quill's true parentage.\", 'popularity': 142.267}\n",
      "3.2652101516723633\n",
      "2024-09-17 08:46:32.760000+00:00\n",
      "{'year': 2023.0, 'title': 'Guardians of the Galaxy Vol. 3', 'overview': 'Peter Quill, still reeling from the loss of Gamora, must rally his team around him to defend the universe along with protecting one of their own. A mission that, if not completed successfully, could quite possibly lead to the end of the Guardians as we know them.', 'popularity': 165.416}\n",
      "2.1303980350494385\n",
      "2024-09-17 08:46:15.943000+00:00\n",
      "{'year': 2002.0, 'title': 'Star Wars: Episode II - Attack of the Clones', 'overview': 'Following an assassination attempt on Senator Padmé Amidala, Jedi Knights Anakin Skywalker and Obi-Wan Kenobi investigate a mysterious plot that could change the galaxy forever.', 'popularity': 44.575}\n",
      "2.0588436126708984\n",
      "2024-09-17 08:46:24.670000+00:00\n",
      "{'title': 'Alien: Covenant', 'overview': 'The crew of the colony ship Covenant, bound for a remote planet on the far side of the galaxy, discovers what they think is an uncharted paradise but is actually a dark, dangerous world.', 'year': 2017.0, 'popularity': 235.941}\n",
      "1.9034849405288696\n",
      "2024-09-17 08:46:15.943000+00:00\n",
      "{'year': 1999.0, 'title': 'Galaxy Quest', 'overview': 'For four years, the courageous crew of the NSEA protector - \"Commander Peter Quincy Taggart\" (Tim Allen), \"Lt. Tawny Madison (Sigourney Weaver) and \"Dr.Lazarus\" (Alan Rickman) - set off on a thrilling and often dangerous mission in space...and then their series was cancelled! Now, twenty years later, aliens under attack have mistaken the Galaxy Quest television transmissions for \"historical documents\" and beam up the crew of has-been actors to save the universe. With no script, no director and no clue, the actors must turn in the performances of their lives.', 'popularity': 62.01}\n"
     ]
    }
   ],
   "source": [
    "from weaviate.classes.query import MetadataQuery\n",
    "\n",
    "response = movies.query.bm25(\n",
    "    query=\"galaxy\",\n",
    "    limit=5,\n",
    "    return_metadata=MetadataQuery(score=True, last_update_time=True)\n",
    ")\n",
    "\n",
    "for o in response.objects:\n",
    "    print(o.metadata.score)\n",
    "    print(o.metadata.last_update_time)\n",
    "    print(o.properties)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ada69268-32aa-4175-99e8-516e355d5703",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Semantic search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa422c04-ae07-48e4-8202-1ea0615db2a0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "A semantic search, on the other hand, searches objects based on similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "295bcedc-496d-49a0-b8c1-b3a398d728b8",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 159,
    "lastExecutedAt": 1692032192969,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "response = (\n    client.query\n    .get(\"Question\", [\"question\", \"answer\"])\n    .with_near_text({\"concepts\": [\"history\"]})\n    .with_limit(3)\n    .do()\n)\n\njprint(response)",
    "outputsMetadata": {
     "0": {
      "height": 477,
      "type": "stream"
     }
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"year\": 1999.0,\n",
      "  \"title\": \"Galaxy Quest\",\n",
      "  \"overview\": \"For four years, the courageous crew of the NSEA protector - \\\"Commander Peter Quincy Taggart\\\" (Tim Allen), \\\"Lt. Tawny Madison (Sigourney Weaver) and \\\"Dr.Lazarus\\\" (Alan Rickman) - set off on a thrilling and often dangerous mission in space...and then their series was cancelled! Now, twenty years later, aliens under attack have mistaken the Galaxy Quest television transmissions for \\\"historical documents\\\" and beam up the crew of has-been actors to save the universe. With no script, no director and no clue, the actors must turn in the performances of their lives.\",\n",
      "  \"popularity\": 62.01\n",
      "}\n",
      "{\n",
      "  \"year\": 1977.0,\n",
      "  \"title\": \"Star Wars\",\n",
      "  \"overview\": \"Princess Leia is captured and held hostage by the evil Imperial forces in their effort to take over the galactic Empire. Venturesome Luke Skywalker and dashing captain Han Solo team together with the loveable robot duo R2-D2 and C-3PO to rescue the beautiful princess and restore peace and justice in the Empire.\",\n",
      "  \"popularity\": 250.715\n",
      "}\n",
      "{\n",
      "  \"year\": 2014.0,\n",
      "  \"title\": \"Interstellar\",\n",
      "  \"overview\": \"The adventures of a group of explorers who make use of a newly discovered wormhole to surpass the limitations on human space travel and conquer the vast distances involved in an interstellar voyage.\",\n",
      "  \"popularity\": 283.624\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "response = movies.query.near_text(\n",
    "    query=\"galaxy\",\n",
    "    limit=3,\n",
    "    target_vector=\"title\",\n",
    ")\n",
    "\n",
    "for o in response.objects:\n",
    "    print(json.dumps(o.properties, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ab8beb0-ab45-4590-a4de-84b68370c026",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### How does this work?\n",
    "\n",
    "- Under the hood, this uses a vector search. It looks for objects which are the most similar to a text input.\n",
    "- We can inspect the similarity along with the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "185a77e4-b27a-4b80-9981-fd975f92e67a",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 142,
    "lastExecutedAt": 1692032193111,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "response = (\n    client.query\n    .get(\"Question\", [\"question\", \"answer\"])\n    .with_near_text({\"concepts\": [\"history\"]})\n    .with_additional(\"distance\")\n    .with_limit(3)\n    .do()\n)\n\njprint(response)",
    "outputsMetadata": {
     "0": {
      "height": 616,
      "type": "stream"
     }
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MetadataReturn(creation_time=None, last_update_time=None, distance=0.2529594898223877, certainty=None, score=None, explain_score=None, is_consistent=None, rerank_score=None)\n",
      "{\n",
      "  \"year\": 1999.0,\n",
      "  \"title\": \"Galaxy Quest\",\n",
      "  \"overview\": \"For four years, the courageous crew of the NSEA protector - \\\"Commander Peter Quincy Taggart\\\" (Tim Allen), \\\"Lt. Tawny Madison (Sigourney Weaver) and \\\"Dr.Lazarus\\\" (Alan Rickman) - set off on a thrilling and often dangerous mission in space...and then their series was cancelled! Now, twenty years later, aliens under attack have mistaken the Galaxy Quest television transmissions for \\\"historical documents\\\" and beam up the crew of has-been actors to save the universe. With no script, no director and no clue, the actors must turn in the performances of their lives.\",\n",
      "  \"popularity\": 62.01\n",
      "}\n",
      "MetadataReturn(creation_time=None, last_update_time=None, distance=0.4024823307991028, certainty=None, score=None, explain_score=None, is_consistent=None, rerank_score=None)\n",
      "{\n",
      "  \"overview\": \"Princess Leia is captured and held hostage by the evil Imperial forces in their effort to take over the galactic Empire. Venturesome Luke Skywalker and dashing captain Han Solo team together with the loveable robot duo R2-D2 and C-3PO to rescue the beautiful princess and restore peace and justice in the Empire.\",\n",
      "  \"year\": 1977.0,\n",
      "  \"title\": \"Star Wars\",\n",
      "  \"popularity\": 250.715\n",
      "}\n",
      "MetadataReturn(creation_time=None, last_update_time=None, distance=0.429984450340271, certainty=None, score=None, explain_score=None, is_consistent=None, rerank_score=None)\n",
      "{\n",
      "  \"year\": 2014.0,\n",
      "  \"title\": \"Interstellar\",\n",
      "  \"overview\": \"The adventures of a group of explorers who make use of a newly discovered wormhole to surpass the limitations on human space travel and conquer the vast distances involved in an interstellar voyage.\",\n",
      "  \"popularity\": 283.624\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "response = movies.query.near_text(\n",
    "    query=\"galaxy\",\n",
    "    limit=3,\n",
    "    target_vector=\"title\",\n",
    "    return_metadata=MetadataQuery(distance=True)\n",
    ")\n",
    "\n",
    "for o in response.objects:\n",
    "    print(o.metadata)\n",
    "    print(json.dumps(o.properties, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6df9e24c-edf7-4f25-ac30-6e291b3ffcad",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "This is where \"vectors\" come in. \n",
    "\n",
    "Each object in Weaviate includes a vector - like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "1d75ad29-ce85-4bca-beaf-da708b20fb57",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 54,
    "lastExecutedAt": 1692032193165,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "response = (\n    client.query\n    .get(\"Question\", [\"question\", \"answer\"])\n    .with_additional(\"vector\")\n    .with_limit(1)\n    .do()\n)\n\njprint(response)",
    "outputsMetadata": {
     "0": {
      "height": 616,
      "type": "stream"
     }
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2529594898223877\n",
      "{\n",
      "  \"title\": \"Galaxy Quest\",\n",
      "  \"overview\": \"For four years, the courageous crew of the NSEA protector - \\\"Commander Peter Quincy Taggart\\\" (Tim Allen), \\\"Lt. Tawny Madison (Sigourney Weaver) and \\\"Dr.Lazarus\\\" (Alan Rickman) - set off on a thrilling and often dangerous mission in space...and then their series was cancelled! Now, twenty years later, aliens under attack have mistaken the Galaxy Quest television transmissions for \\\"historical documents\\\" and beam up the crew of has-been actors to save the universe. With no script, no director and no clue, the actors must turn in the performances of their lives.\",\n",
      "  \"year\": 1999.0,\n",
      "  \"popularity\": 62.01\n",
      "}\n",
      "[-0.5893528461456299, 0.5016435980796814, -3.231532096862793, 0.6001119017601013, -0.4437905550003052]\n",
      "0.4024823307991028\n",
      "{\n",
      "  \"year\": 1977.0,\n",
      "  \"title\": \"Star Wars\",\n",
      "  \"overview\": \"Princess Leia is captured and held hostage by the evil Imperial forces in their effort to take over the galactic Empire. Venturesome Luke Skywalker and dashing captain Han Solo team together with the loveable robot duo R2-D2 and C-3PO to rescue the beautiful princess and restore peace and justice in the Empire.\",\n",
      "  \"popularity\": 250.715\n",
      "}\n",
      "[-0.9034830927848816, -0.7712558507919312, -3.110103130340576, 0.34885266423225403, -0.9137812256813049]\n",
      "0.429984450340271\n",
      "{\n",
      "  \"year\": 2014.0,\n",
      "  \"title\": \"Interstellar\",\n",
      "  \"overview\": \"The adventures of a group of explorers who make use of a newly discovered wormhole to surpass the limitations on human space travel and conquer the vast distances involved in an interstellar voyage.\",\n",
      "  \"popularity\": 283.624\n",
      "}\n",
      "[0.024126943200826645, -0.013838309794664383, -2.8055012226104736, -0.14428117871284485, -0.09647063910961151]\n"
     ]
    }
   ],
   "source": [
    "response = movies.query.near_text(\n",
    "    query=\"galaxy\",\n",
    "    limit=3,\n",
    "    target_vector=\"title\",  # or \"overview\"\n",
    "    include_vector=True,\n",
    "    return_metadata=MetadataQuery(distance=True)\n",
    ")\n",
    "\n",
    "for o in response.objects:\n",
    "    print(o.metadata.distance)\n",
    "    print(json.dumps(o.properties, indent=2))\n",
    "    print(o.vector[\"title\"][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4097db1-76b0-459c-b5c3-2ee1e0f90abe",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "These vector representations come from deep learning models to those that power LLMs. They capture meaning, and are called vector \"embeddings\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a055536-649b-4840-8443-c7f15c4f961c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Generative search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8bbeb19-3d05-47f4-960f-6f8e9e5dfcc8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "A generative search transforms your data at retrieval time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "6eb0d7a5-d31b-4b5a-a7d8-96719947a69e",
   "metadata": {
    "editable": true,
    "executionCancelledAt": null,
    "executionTime": 4342,
    "lastExecutedAt": 1692032197507,
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "response = (\n    client.query\n    .get(\"Question\", [\"question\", \"answer\"])\n    .with_near_text({\"concepts\": [\"history\"]})\n    .with_generate(single_prompt=\"Write a tweet about {question} as an interesting factoid.\")\n    .with_limit(3)\n    .do()\n)\n\njprint(response)",
    "outputsMetadata": {
     "0": {
      "height": 616,
      "type": "stream"
     }
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = movies.generate.near_text(\n",
    "    query=\"galaxy\",\n",
    "    limit=5,\n",
    "    target_vector=\"title\",\n",
    "    single_prompt=\"Write a tweet promoting the movie with TITLE: {title} and OVERVIEW: {overview}.\",\n",
    "    grouped_task=\"What audience demographic might enjoy this group of movies?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c1bcf81c-04f8-476d-b1aa-ccbace93f192",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These movies appeal to a **broad audience** but primarily resonate with those who enjoy **genre-based escapism**:\n",
      "\n",
      "* **Sci-Fi/Space Opera:**  The core themes of space exploration, aliens, and intergalactic conflict are staples in the genre. Each movie has its own unique take on this common trope.\n",
      "* **Comedic Relief:** \"Galaxy Quest\" offers a hilariously lighthearted approach to science fiction, playing on pop culture references. This appeals to those who appreciate clever comedy mixed with action.\n",
      "* **Classic Sci-Fi Fans:**  \"Star Wars,\" specifically the original trilogy (and its prequel, Episode I) holds nostalgic significance for many fans and remains relevant in terms of storytelling and special effects. \n",
      "* **Fans of Adventure/Fantasy:** \"Interstellar\" appeals to those who enjoy space exploration with a touch of fantastical elements, while \"Guardians of the Galaxy Vol.2\" offers a more lighthearted but emotional journey of familial bonds.\n",
      "\n",
      "**In summary,** this group of films caters to audiences interested in science fiction, fantasy, and adventure narratives, particularly those drawn to humor, spectacle, and nostalgia for classic space operas. \n",
      "\n",
      "\n",
      "Let me know if you'd like to delve deeper into any specific film or demographic! \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(response.generated)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b7f1dc7f-731b-424c-9399-d64c02fc4107",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🚀 Still haven't seen #GalaxyQuest?  Time travel through the cosmos! 🌌 This hilarious cult classic about washed-up actors saving the galaxy is a must-watch! 😂\n",
      "\n",
      "Check it out now! 👉 [link to movie]  #TimAllen #SigourneyWeaver #AlanRickman #SpaceAdventure #ComedyGold 🎬 \n",
      "\n",
      "{\n",
      "  \"overview\": \"For four years, the courageous crew of the NSEA protector - \\\"Commander Peter Quincy Taggart\\\" (Tim Allen), \\\"Lt. Tawny Madison (Sigourney Weaver) and \\\"Dr.Lazarus\\\" (Alan Rickman) - set off on a thrilling and often dangerous mission in space...and then their series was cancelled! Now, twenty years later, aliens under attack have mistaken the Galaxy Quest television transmissions for \\\"historical documents\\\" and beam up the crew of has-been actors to save the universe. With no script, no director and no clue, the actors must turn in the performances of their lives.\",\n",
      "  \"year\": 1999.0,\n",
      "  \"title\": \"Galaxy Quest\",\n",
      "  \"popularity\": 62.01\n",
      "}\n",
      "🚀🌌 **Get ready for a galactic adventure!** 💥  Princess Leia is captured by the Empire, & it's up to Luke Skywalker, Han Solo, R2-D2, & C-3PO to save her! #StarWars is out now! ✨ [link to movie]  \n",
      "\n",
      "#starwars #disneyplus #movietime #adventure #heroes \n",
      "\n",
      "{\n",
      "  \"year\": 1977.0,\n",
      "  \"title\": \"Star Wars\",\n",
      "  \"overview\": \"Princess Leia is captured and held hostage by the evil Imperial forces in their effort to take over the galactic Empire. Venturesome Luke Skywalker and dashing captain Han Solo team together with the loveable robot duo R2-D2 and C-3PO to rescue the beautiful princess and restore peace and justice in the Empire.\",\n",
      "  \"popularity\": 250.715\n",
      "}\n",
      "🚀🌌 **Interstellar** - Humanity's last hope for survival lies beyond the stars. 🌍 Witness breathtaking visuals and emotional journeys as explorers embark on a daring mission through a wormhole, defying time itself! #InterstellarMovie #SpaceTravel #SciFi  [Link to movie]\n",
      "\n",
      "**---**\n",
      "You can further personalize this tweet by:\n",
      "* **Adding an eye-catching image or GIF from the film.**\n",
      "* **Mentioning if it's playing in theaters or streaming now.**\n",
      "* **Including relevant hashtags for a wider reach.** \n",
      "\n",
      "{\n",
      "  \"year\": 2014.0,\n",
      "  \"title\": \"Interstellar\",\n",
      "  \"overview\": \"The adventures of a group of explorers who make use of a newly discovered wormhole to surpass the limitations on human space travel and conquer the vast distances involved in an interstellar voyage.\",\n",
      "  \"popularity\": 283.624\n",
      "}\n",
      "💥 Get ready for galactic adventures and heart-stopping revelations! 🌌 #GuardiansOfTheGalaxyVol2 hits theaters soon, where the Guardians face their toughest challenge yet.  \n",
      "\n",
      "Is there more to Peter Quill than meets the eye? Find out why this movie is a must-watch! 🚀➡️ [Link to trailer/tickets] \n",
      "\n",
      "#MarvelMovies #SpaceOdyssey #TeamQuill #NewMovieAlert \n",
      "\n",
      "{\n",
      "  \"overview\": \"The Guardians must fight to keep their newfound family together as they unravel the mysteries of Peter Quill's true parentage.\",\n",
      "  \"year\": 2017.0,\n",
      "  \"title\": \"Guardians of the Galaxy Vol. 2\",\n",
      "  \"popularity\": 142.267\n",
      "}\n",
      "🚀💥 #StarWars:  A new hero, a dark enemy, and the fate of the galaxy hangs in the balance! 🌌 Witness Anakin Skywalker's journey begin in \"The Phantom Menace\" - available now! [link to movie] #ForceAwakens #MayThe4thBeWithYou 💫 \n",
      "\n",
      "{\n",
      "  \"year\": 1999.0,\n",
      "  \"title\": \"Star Wars: Episode I - The Phantom Menace\",\n",
      "  \"overview\": \"Anakin Skywalker, a young slave strong with the Force, is discovered on Tatooine. Meanwhile, the evil Sith have returned, enacting their plot for revenge against the Jedi.\",\n",
      "  \"popularity\": 60.19\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "for o in response.objects:\n",
    "    print(o.generated)\n",
    "    print(json.dumps(o.properties, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29d4445d-918d-47c1-a553-688fffbb7164",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Each object has been transformed into a tweet by the LLM based on our prompt!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68b5ede1-0d97-44ab-8b06-29a6aea0afaf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Bonus: Multi-tenancy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16af8f66-2228-4138-aff2-24d311c5a377",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Weaviate](./img/08-mt-intro-01.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80f10456-204c-46ed-9f2b-205efbd15b30",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Weaviate](./img/08-mt-intro-03.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1136e510-b8ca-478f-af9c-a90ffdbff801",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Weaviate](./img/08-mt-intro-03.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "305bd273-6bbb-4309-b398-33c8e0c3214b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Weaviate](./img/08-mt-intro-04.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b07cc78d-c636-4db7-815d-ad456736a9b5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Weaviate](./img/08-mt-intro-05.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89924c6b-e846-4271-b1b3-e77243bba721",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![Introduction to Weaviate](./img/08-mt-intro-06.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1987929c-d3c1-41a7-a88c-9b12540d009d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Create a multi-tenant collection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "393146a8-162c-49c3-8469-ef142620aaa7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "At collection creation time, add:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a77d2145-6d3c-4985-baa6-7dc7331a7203",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "```python\n",
    "from weaviate.classes.config import Configure\n",
    "\n",
    "collection = client.collections.create(\n",
    "    name=collection_name,\n",
    "    # ...\n",
    "    multi_tenancy_config=Configure.multi_tenancy(\n",
    "        enabled=True,\n",
    "        # auto_tenant_creation=True,\n",
    "        # auto_tenant_activation=True,\n",
    "    ),\n",
    "    # ...\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f111f964-cbcf-41d1-a12d-c07d7f87ff55",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Actual collection definition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "e8e2cf9b-1805-4a76-8392-5c8aed89b210",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import weaviate\n",
    "from weaviate.classes.config import Property, DataType, Configure\n",
    "\n",
    "\n",
    "client = weaviate.connect_to_local()\n",
    "\n",
    "\n",
    "if client.collections.exists(\"SupportChatMT\"):\n",
    "    client.collections.delete(\"SupportChatMT\")\n",
    "\n",
    "\n",
    "mt_collection = client.collections.create(\n",
    "    name=\"SupportChatMT\",\n",
    "    # ========== MT Config ==========\n",
    "    multi_tenancy_config=Configure.multi_tenancy(\n",
    "        enabled=True,\n",
    "        auto_tenant_creation=True,\n",
    "        auto_tenant_activation=False\n",
    "    ),\n",
    "    # ========================================\n",
    "    properties=[\n",
    "        Property(name=\"text\", data_type=DataType.TEXT),\n",
    "        Property(name=\"dialogue_id\", data_type=DataType.INT),\n",
    "        Property(name=\"company_author\", data_type=DataType.TEXT),\n",
    "        Property(name=\"created_at\", data_type=DataType.DATE),\n",
    "    ],\n",
    "    # ================================================================================\n",
    "    # Using our Ollama integration: https://weaviate.io/developers/weaviate/model-providers/ollama\n",
    "    # Many other integrations available. See https://weaviate.io/developers/weaviate/model-providers/\n",
    "    # ================================================================================\n",
    "    vectorizer_config=[\n",
    "        Configure.NamedVectors.text2vec_ollama(\n",
    "            name=\"text\",\n",
    "            source_properties=[\"text\"],\n",
    "            # ========== Use the Dynamic Index ==========\n",
    "            vector_index_config=Configure.VectorIndex.dynamic(),\n",
    "            # ========================================\n",
    "            api_endpoint=\"http://host.docker.internal:11434\",\n",
    "            model=\"nomic-embed-text\",\n",
    "        ),\n",
    "    ],\n",
    "    generative_config=Configure.Generative.ollama(\n",
    "        api_endpoint=\"http://host.docker.internal:11434\", model=\"gemma2:2b\"\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7a43c89-a9af-4da5-8ef9-41fa48106945",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Interact with tenants"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bf18d1e-c0e7-43f2-b6e4-13352588763c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Create tenants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "6a228f8d-34c8-4760-9879-9e31d8a11688",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "new_tenants = [\"AcmeCo\", \"FancyPhones\", \"BudgetAir\", \"LightningCars\"]\n",
    "\n",
    "mt_collection.tenants.create(new_tenants)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08d1d505-d17a-4e97-8749-10dc18807aad",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### MT `tenant` == ST `collection`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "41658ddf-102e-485c-991f-4bf236e0af74",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "tenant_coll = mt_collection.with_tenant(\"AcmeCo\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "7c9bc29b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UUID('2533e902-34a9-499c-898d-ea780f66e7fc')"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tenant_coll.data.insert(\n",
    "    properties={\"text\": \"Hello, I have a problem with my phone\", \"dialogue_id\": 1},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "8b2652b6-8d5b-4659-9587-f0dc9bb0b4eb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "with client.batch.fixed_size(batch_size=100) as batch:\n",
    "    for tenant_name in new_tenants:\n",
    "        for i in range(3):\n",
    "            batch.add_object(\n",
    "                collection=\"SupportChatMT\",\n",
    "                properties={\n",
    "                    \"text\": f\"Conversation {i}\",\n",
    "                    \"dialogue_id\": i,\n",
    "                    \"company_author\": \"AcmeCo\",\n",
    "                },\n",
    "                tenant=tenant_name\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "47098f74-961f-42dc-bbea-df86464010c3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AcmeCo 4\n",
      "FancyPhones 3\n",
      "BudgetAir 3\n",
      "LightningCars 3\n"
     ]
    }
   ],
   "source": [
    "for tenant_name in new_tenants:\n",
    "    tenant_coll = mt_collection.with_tenant(tenant_name)\n",
    "    print(tenant_name, len(tenant_coll))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0461231-4781-4861-8be9-a987feef373c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Manage tenants"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7284e90-5297-4e66-b783-8bf677262b28",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "**Tenants**: data isolation & easy on/off boarding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81825abc-465c-41b3-be8d-33705e9e8474",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Isolation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "ac2791ef-2b4d-4174-a9e5-d68b1c17b3d1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "# Work with tenant data\n",
    "tenant_coll = mt_collection.with_tenant(\"AcmeCo\")\n",
    "response = tenant_coll.query.hybrid(\"water damage\", limit=2)\n",
    "print(len(response.objects))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "dc2f338c-108d-42ca-b5f4-f23979dca966",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Cannot search across multiple tenants\n",
    "# response = mt_collection.query.hybrid(\"water damage\", limit=2)  # <-- This throws an error"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71cef1a9-9e40-4a0a-9143-b86c9bdb5f30",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Off-boarding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "c8837c86-8a65-48ee-8f68-c3baa22338ae",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "mt_collection.tenants.remove(\"FancyPhones\") "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aabc6117-9bf0-4ee8-90e3-6e2b75d25572",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "This deletes all of this tenant's data!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0301928-3523-4a2e-ba9c-c760bf7f6e3c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### On-boarding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "392fd932-e039-45c7-ac65-e56629d6ca5e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "mt_collection.tenants.create(\"NewStartup1\") "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2603ff8f-704b-4f2b-b85c-15342ed252bf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "You can now add data to this tenant!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "106267cc-9d8d-4a24-b480-436ad060cb80",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Manage resources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa9e47ee-15c2-4a1f-9132-33a23d08099b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Tenants can be moved between hot (memory), warm (disk), cold (cloud) storage to manage resource used. \n",
    "\n",
    "This is done with the \"activity status\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "843712d8-781e-4d1d-81af-9ef220eb1244",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Create a tenant with a particular status:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "630db521-35d4-4881-aad6-a72127ee509a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from weaviate.classes.tenants import Tenant, TenantActivityStatus\n",
    "\n",
    "mt_collection.tenants.create(\n",
    "    Tenant(name=\"NewStartup2\", activity_status=TenantActivityStatus.INACTIVE)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c75c1ce-a573-404e-87f4-7196eb662eb7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Inactive (or offloaded) tenants are not available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "a050b3b5-2bde-4e98-89ac-51315c9c7271",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "my_tenant = mt_collection.with_tenant(\"NewStartup2\")\n",
    "# my_tenant.data.insert({\"text\": \"Your product is amazing!\"})  # Run this - you should see an error"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "953177b6-de77-4b72-b634-de576fb4a1ec",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Update a tenant status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "70cf9c41-e66e-49cf-b2f3-efd72e49ce0e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "mt_collection.tenants.update(\n",
    "    Tenant(name=\"NewStartup2\", activity_status=TenantActivityStatus.ACTIVE)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b4195f5-aa6b-4604-ac36-0483c10eca94",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Now, we can interact with it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "07061342-86c6-4a2e-ad6a-23fe9d71ced2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UUID('9663bed0-64fa-4513-a7c6-948b3107e378')"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_tenant.data.insert({\"text\": \"Your product is amazing!\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b14aa43e-538f-4eed-9e57-fef20558dbc0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Tenant status: Balance resource use & availability"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69b0c138-d8d8-4aaa-8ec2-bba0e30b3593",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "This allows us to balance resource usage & cost with availability. \n",
    "\n",
    "- Active: Available, on RAM/disk (depending on index type)\n",
    "- Inactive: Unavailable, on disk\n",
    "- Offloaded: Unavailable, on cloud storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc5e593a-3a8c-4ca4-b6fc-ecdc9070d2e4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### When you're done... "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ad47cd2-4c2b-4f06-a9a6-6f909d82dd82",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Remember to close the client connection with `client.close()` to close sockets and resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e03f4fb-8a08-41a7-a7d1-dc1ebc14fae3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "client.close()"
   ]
  }
 ],
 "metadata": {
  "editor": "DataCamp Workspace",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}