Notebooks/IBM Watson Speech to Text custom model training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a custom IBM Watson™ Speech to Text language and acoustic model\n",
    "---\n",
    "## Pre-requisites:\n",
    "- [Install Watson Libraries](#Install-Watson-libraries)\n",
    "- [Import Dependencies](#Import-Dependencies)\n",
    "- [Add Speech to Text credentials](#Add-Speech-to-Text-credentials)\n",
    "- [Setup and configure speech credentials](#Setup-and-configure-speech-credentials)\n",
    "\n",
    "## Test the Lite models:\n",
    "- [List all the available speech models](#(Optional)-List-all-the-available-speech-models)\n",
    "- [Print out the details of the Narrowband model](#(Optional)-Print-out-the-details-of-the-Narrowband-model)\n",
    "- [Transcribe using the base speech (Narrowband US-English) model](#Transcribe-using-the-base-speech-(Narrowband-US-English)-model)\n",
    "- [Print the transcript](#(Optional)-Print-the-transcription)\n",
    "\n",
    "## Follow these steps to create, add contents to, and train a custom language model for the IBM Watson™ Speech to Text service:\n",
    "1. [Create a custom language model](1.-Create-a-custom-language-model)\n",
    "1. [Add text corpus file for language training](#2.-Add-text-corpus-file-for-language-training)\n",
    "1. [Print the details of the corpora](#3.-Print-the-details-of-the-corpora)\n",
    "1. [Train the custom language model](#4.-Train-the-custom-language-model)\n",
    "1. [Check status of the custom language model](#5.-Check-status-of-the-custom-language-model)\n",
    "\n",
    "## Follow these steps to create a custom acoustic model for the IBM Watson™ Speech to Text service:\n",
    "1. [Create a custom acoustic model](#1.-Create-a-custom-acoustic-model)\n",
    "1. [Add audio file to the acoustic model](#2.-Add-audio-file-to-the-acoustic-model)\n",
    "1. [List all the audio files used for acoustic modeling](#3.-List-all-the-audio-files-used-for-acoustic-modeling)\n",
    "1. [List the details of the acoustic model](#4.-List-the-details-of-the-acoustic-model)\n",
    "1. [Train the acoustic model](#5.-Train-the-acoustic-model)\n",
    "1. [Check status of the custom acoustic model](#6.-Check-status-of-the-custom-acoustic-model)\n",
    "\n",
    "## Test the trained models:\n",
    "1. [Run transcription using the custom acoustic & language models](#1.-Run-transcription-using-the-custom-acoustic-&-language-models)\n",
    "1. [Print the transcript from the custom model](#2.-Print-the-transcript-from-the-custom-model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-requisites:\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Watson libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade \"ibm-watson>=5.3.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Speech to Text credentials\n",
    "\n",
    "> Add the credentials created in 2.1. [Create Watson Speech to Text service on IBM Cloud](https://github.com/IBM/video-summarizer-using-watson/blob/main/README.md#21-create-watson-speech-to-text-service-on-ibm-cloud)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "credentials = {\n",
    "    \n",
    "  }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup and configure speech credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson import SpeechToTextV1\n",
    "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator\n",
    "\n",
    "authenticator = IAMAuthenticator(credentials['apikey'])\n",
    "speech_to_text = SpeechToTextV1(\n",
    "    authenticator=authenticator\n",
    ")\n",
    "\n",
    "speech_to_text.set_service_url(credentials['url'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the Lite models:\n",
    "\n",
    "#### These are the pre trained models provided by IBM Watson™ Speech to Text\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) List all the available speech models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_models = speech_to_text.list_models().get_result()\n",
    "print(json.dumps(speech_models, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Print out the details of the Narrowband model\n",
    ">Note: `Narrowband` Model is good for transcribing Human to Human conversations and `Broadband` Model is good for transcribing Human to Bot conversations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_model = speech_to_text.get_model('en-US_NarrowbandModel').get_result()\n",
    "print(json.dumps(speech_model, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Transcribe using the base speech (Narrowband US English) model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'datasets/ste-training-data/audios/chunk0.wav'\n",
    "\n",
    "with open(filename, 'rb') as audio_file:\n",
    "    speech_recognition_results = speech_to_text.recognize(\n",
    "        audio=audio_file,\n",
    "        content_type='audio/wav',\n",
    "        model='en-US_NarrowbandModel',\n",
    "        timestamps=True,\n",
    "        speaker_labels=True,\n",
    "        word_alternatives_threshold=0.9,\n",
    "        smart_formatting=True\n",
    "    ).get_result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(json.dumps(speech_recognition_results, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Print the transcription"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transcript = ''\n",
    "for chunks in speech_recognition_results['results']:\n",
    "    if 'alternatives' in chunks.keys():\n",
    "        alternatives = chunks['alternatives'][0]\n",
    "        if 'transcript' in alternatives:\n",
    "            transcript = transcript + alternatives['transcript']\n",
    "print(transcript)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Follow these steps to create, add contents to, and train a custom language model for the IBM Watson™ Speech to Text service:\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Create a custom language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'STE language model'\n",
    "model_base = 'en-US_NarrowbandModel'\n",
    "model_description = 'Custom Lang Model for ST Engineering'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "language_model = speech_to_text.create_language_model(\n",
    "    model_name,\n",
    "    model_base,\n",
    "    description=model_description).get_result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Print the customization ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(language_model, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Print all the language models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "language_models = speech_to_text.list_language_models().get_result()\n",
    "print(json.dumps(language_models, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_lang_narrowband_model_id = language_model.get('customization_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Add text corpus file for language training\n",
    "\n",
    "We have used the following datasets:\n",
    "- `data/earnings-call-corpus-file.wav`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/earnings-call-corpus-file.txt','rb') as corpus_file:\n",
    "    speech_to_text.add_corpus(\n",
    "        custom_lang_narrowband_model_id,\n",
    "        'corpus-file.txt',\n",
    "        corpus_file\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Print the details of the corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpora = speech_to_text.list_corpora(custom_lang_narrowband_model_id).get_result()\n",
    "print(json.dumps(corpora, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Print all the custom words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "words = speech_to_text.list_words(custom_lang_narrowband_model_id).get_result()\n",
    "print(json.dumps(words, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Add grammar words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with open(join(dirname('.'), './.', '{localtion-path}/productionWords.abnf'),\n",
    "#                'rb') as grammar_file:\n",
    "#     speech_to_text.add_grammar(\n",
    "#         custom_lang_broadband_model_id,\n",
    "#         '{Grammar-name}',\n",
    "#         grammar_file,\n",
    "#         'application/srgs'\n",
    "#     )\n",
    "# Poll for grammar status."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Upgrade the language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_to_text.upgrade_language_model(custom_lang_narrowband_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Train the custom language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_to_text.train_language_model(custom_lang_narrowband_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Check status of the custom language model\n",
    "> The status has to become `available`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Get status of the language model - wait until it is 'available'\n",
    "language_models = speech_to_text.list_language_models().get_result()\n",
    "models = language_models[\"customizations\"]\n",
    "\n",
    "for model in models:\n",
    "    if model['customization_id'] == custom_lang_narrowband_model_id: \n",
    "        print(model['status'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Learn more about creating an IBM Watson™ Speech to Text language model here: <https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageCreate>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Follow these steps to create a custom acoustic model for the IBM Watson™ Speech to Text service:\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Create a custom acoustic model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'STE acoustic model'\n",
    "model_base = 'en-US_NarrowbandModel'\n",
    "model_description = 'Custom Acoustic Model for ST Engineering'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "acoustic_model = speech_to_text.create_acoustic_model(\n",
    "    model_name,\n",
    "    model_base,\n",
    "    description=model_description).get_result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Print the customization ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(acoustic_model, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Print all the acoustic models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acoustic_models = speech_to_text.list_acoustic_models().get_result()\n",
    "print(json.dumps(acoustic_models, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_acoustic_narrowband_model_id = acoustic_model[\"customization_id\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Add audio file to the acoustic model\n",
    "\n",
    "We have used the following datasets:\n",
    "- `data/earnings-call-2019-train1.wav`\n",
    "- `datasets/earnings-call-2019-train2.wav`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "audioFilePath = 'data/earnings-call-2019-train1.wav'\n",
    "audioFileName = audioFilePath.split('/')[3]\n",
    "\n",
    "with open(audioFilePath, 'rb') as audio_file:\n",
    "    speech_to_text.add_audio(\n",
    "        custom_acoustic_narrowband_model_id,\n",
    "        audioFileName,\n",
    "        audio_file,\n",
    "        content_type='audio/wav'\n",
    "    )\n",
    "# Poll for audio status."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. List all the audio files used for acoustic modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "audio_resources = speech_to_text.list_audio(custom_acoustic_narrowband_model_id).get_result()\n",
    "print(json.dumps(audio_resources, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. List the details of the acoustic model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acoustic_model = speech_to_text.get_acoustic_model(custom_acoustic_narrowband_model_id).get_result()\n",
    "print(json.dumps(acoustic_model, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Upgrade the acoustic model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_to_text.upgrade_acoustic_model(custom_acoustic_narrowband_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Train the acoustic model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_to_text.train_acoustic_model(custom_acoustic_narrowband_model_id) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  6. Check status of the custom acoustic model\n",
    "> The status has to become `available`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get status of the acoustic model - wait until it is 'available'\n",
    "acoustic_models = speech_to_text.list_acoustic_models().get_result()\n",
    "models = acoustic_models[\"customizations\"]\n",
    "\n",
    "for model in models:\n",
    "    if model['customization_id'] == custom_acoustic_narrowband_model_id: \n",
    "            print(model['status'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Learn more about creating an IBM Watson™ Speech to Text acoustic model here: <https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-acoustic>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the trained models:\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Run transcription using the custom acoustic & language models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'datasets/ste-training-data/audios/chunk0.wav'\n",
    "\n",
    "with open(filename ,'rb') as audio_file:\n",
    "    speech_recognition_results = speech_to_text.recognize(\n",
    "        audio=audio_file,\n",
    "        content_type='audio/wav',\n",
    "        model='en-US_NarrowbandModel',\n",
    "        customization_id=custom_lang_narrowband_model_id,\n",
    "        acoustic_customization_id=custom_acoustic_narrowband_model_id,\n",
    "        timestamps=True,\n",
    "        word_alternatives_threshold=0.9,\n",
    "        keywords=['derailment case', 'start length', 'drive side quarter', 'driver impression', 'severity'],\n",
    "        keywords_threshold=0.5).get_result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(json.dumps(speech_recognition_results, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Print the transcript from the custom model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transcript2 = ''\n",
    "for chunks in speech_recognition_results['results']:\n",
    "    if 'alternatives' in chunks.keys():\n",
    "        alternatives = chunks['alternatives'][0]\n",
    "        if 'transcript' in alternatives:\n",
    "            transcript2 = transcript2 + alternatives['transcript']\n",
    "print(transcript2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Delete the custom models:\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the custom language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_to_text.delete_language_model(custom_lang_narrowband_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the custom acoustic model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_to_text.delete_acoustic_model(custom_acoustic_narrowband_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}