notebooks/visualcla_inference.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "YOWuB66P0xyd"
      },
      "source": [
        "# VisualCLA Inference & Deployment\n",
        "\n",
        "该notebook展示了从环境准备、模型合并到推理以及Gradio部署的操作流程。\n",
        "\n",
        "⚠️以下代码需消耗 20G+的内存，请确保刷出来的机器RAM满足要求。"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "yVliGZLL1Asa"
      },
      "source": [
        "## 一、环境与代码准备工作"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "RAPy82rx1D3h"
      },
      "source": [
        "### 1.1 安装相关依赖"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QREF94o6Jbqp"
      },
      "outputs": [],
      "source": [
        "!pip install transformers==4.30.2\n",
        "!pip install peft==0.3.0\n",
        "!pip install sentencepiece\n",
        "!pip install bitsandbytes\n",
        "!pip install gradio\n",
        "!pip install mdtex2html"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "C0fZplUu0w0p"
      },
      "source": [
        "### 1.2 克隆目录和代码"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9fmoBCvSJdek"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/ymcui/Chinese-LLaMA-Alpaca"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ck6Q7fmzJeGU"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/airaria/Visual-Chinese-LLaMA-Alpaca"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Z2bGlZHt1N0T"
      },
      "source": [
        "### 1.3 安装VisualCLA代码"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GKSX-8bnOwMV"
      },
      "outputs": [],
      "source": [
        "!pip install -e Visual-Chinese-LLaMA-Alpaca/"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "plJM844v1VjW"
      },
      "source": [
        "## 二、合并模型\n",
        "\n",
        "合并模型过程分为两步：\n",
        "* 从原版LLaMA合并出Chinese-LLaMA-Alpaca-Plus-7B模型\n",
        "* 从Chinese-LLaMA-Alpaca-Plus-7B合并出VisualCLA模型"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "NAPz3OX91Z9x"
      },
      "source": [
        "### 2.1 合并Chinese-Alpaca-Plus-7B模型\n",
        "\n",
        "此处使用的是🤗模型库中提供的基模型（HF格式）\n",
        "- 基模型：`elinas/llama-7b-hf-transformers-4.29` *（use at your own risk，我们比对过SHA256和正版一致，但你应确保自己有权使用该模型）*\n",
        "- LoRA模型：'ziqingyang/chinese-llama-plus-lora-7b,ziqingyang/chinese-alpaca-plus-lora-7b'\n",
        "\n",
        "转换好的模型存放在`chinese-alpaca-plus-7b`目录。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IIq_K65JJ9BZ"
      },
      "outputs": [],
      "source": [
        "!python ./Chinese-LLaMA-Alpaca/scripts/merge_llama_with_chinese_lora_low_mem.py \\\n",
        "    --base_model 'elinas/llama-7b-hf-transformers-4.29' \\\n",
        "    --lora_model 'ziqingyang/chinese-llama-plus-lora-7b,ziqingyang/chinese-alpaca-plus-lora-7b' \\\n",
        "    --output_type huggingface \\\n",
        "    --output_dir chinese-alpaca-plus-7b"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "-jo4bkn51iWR"
      },
      "source": [
        "### 2.2 合并VisualCLA模型\n",
        "\n",
        "基于Chinese-Alpaca-Plus-7B和CLIP-ViT，合并得到最终的VisualCLA模型。\n",
        "\n",
        "转换好的模型存放在`visualcla`目录"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mWMofK9bJ_HH"
      },
      "outputs": [],
      "source": [
        "! python Visual-Chinese-LLaMA-Alpaca/scripts/merge_llama_with_visualcla_lora.py \\\n",
        "    --text_model chinese-alpaca-plus-7b \\\n",
        "    --vision_model openai/clip-vit-large-patch14 \\\n",
        "    --lora_model ziqingyang/visualcla-7b-v0.1 \\\n",
        "    --output_dir visualcla"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "zzjiXLuj3Kt_"
      },
      "source": [
        "## 三、 模型推理与部署\n",
        "\n",
        "合并完成后，用户可以选择是进行命令行的推理或启动Gradio demo进行可视化体验"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "dAcpKKuO2-9Y"
      },
      "source": [
        "### 3.1 命令行推理\n",
        "\n",
        "运行下面的命令将启动交互式命令行程序进行推理。\n",
        "\n",
        "默认读入image_file文件，用户可输入向模型提问题或输入指令。\n",
        "\n",
        "也可通过change image命令更换图片。详细使用说明可参见启动后的提示信息。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zpJp4xz83DUd"
      },
      "outputs": [],
      "source": [
        "!python Visual-Chinese-LLaMA-Alpaca/scripts/inference/inference.py \\\n",
        "    --visualcla_model visualcla \\\n",
        "    --image_file Visual-Chinese-LLaMA-Alpaca/pics/examples/food.jpg \\\n",
        "    --load_in_8bit"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "TaPw3Shg1x31"
      },
      "source": [
        "### 3.2 启动Gradio\n",
        "\n",
        "运行下面的命令启动Gradio demo。\n",
        "\n",
        "成功启动后单元格中将显示url地址。打开url地址即可访问demo。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NlWoPpz7KOgR"
      },
      "outputs": [],
      "source": [
        "!python Visual-Chinese-LLaMA-Alpaca/scripts/inference/gradio_demo.py --visualcla_model visualcla --load_in_8bit --share"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}