3_llm_pytorch_gpu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4bdf80ae-10bd-438b-a5ae-76a5c5f99a6d",
   "metadata": {},
   "source": [
    "# Inference using Pytorch on Intel GPUs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71e0aeac-58b1-4114-95f1-7d3a7a4c34f2",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook demonstrates how to run LLM inference using pytorch on Windows with Intel GPUs. It applies to Intel Core Ultra and Core 11 - 14 gen integrated GPUs (iGPUs), as well as Intel Arc Series GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97cf7db8-9529-47dd-b41d-81b22c8d5848",
   "metadata": {},
   "source": [
    "## What is an AIPC\n",
    "\n",
    "What is an AI PC you ask?\n",
    "\n",
    "Here is an [explanation](https://www.intel.com/content/www/us/en/newsroom/news/what-is-an-ai-pc.htm#gs.a55so1):\n",
    "\n",
    "”An AI PC has a CPU, a GPU and an NPU, each with specific AI acceleration capabilities. An NPU, or neural processing unit, is a specialized accelerator that handles artificial intelligence (AI) and machine learning (ML) tasks right on your PC instead of sending data to be processed in the cloud. The GPU and CPU can also process these workloads, but the NPU is especially good at low-power AI calculations. The AI PC represents a fundamental shift in how our computers operate. It is not a solution for a problem that didn’t exist before. Instead, it promises to be a huge improvement for everyday PC usages.”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4682eb3e-540b-4814-8142-c54efc32f31b",
   "metadata": {},
   "source": [
    "## Install Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f8b6d2-34af-44ad-8363-dea57660bc00",
   "metadata": {},
   "source": [
    "### Step 1: System Preparation\n",
    "\n",
    "To set up your AIPC for running with Intel iGPUs, follow these essential steps:\n",
    "\n",
    "1. Update Intel GPU Drivers: Ensure your system has the latest Intel GPU drivers, which are crucial for optimal performance and compatibility. You can download these directly from Intel's [official website](https://www.intel.com/content/www/us/en/download/785597/intel-arc-iris-xe-graphics-windows.html) . Once you have installed the official drivers, you could also install Intel ARC Control to monitor the gpu:\n",
    "\n",
    "   <img src=\"Assets/gpu_arc_control.png\">\n",
    "\n",
    "\n",
    "2. Install Visual Studio 2022 Community edition with C++: Visual Studio 2022, along with the “Desktop Development with C++” workload, is required. This prepares your environment for C++ based extensions used by the intel SYCL backend that powers accelerated Ollama. You can download VS 2022 Community edition from the official site, [here](https://visualstudio.microsoft.com/downloads/).\n",
    "\n",
    "3. Install conda-forge: conda-forge will manage your Python environments and dependencies efficiently, providing a clean, minimal base for your Python setup. Visit conda-forge's [installation site](https://conda-forge.org/download/) to install for windows.\n",
    "\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8040fd21-7782-4b97-a0eb-327816328f17",
   "metadata": {},
   "source": [
    "## Step 2: Setup the environment and install required libraries\n",
    "\n",
    "### After installation of conda-forge, open the Miniforge Prompt, and create a new python environment:\n",
    "  ```\n",
    "  conda create -n llm python=3.11 libuv\n",
    "\n",
    "  ```\n",
    "\n",
    "### Activate the new environment\n",
    "\n",
    "```\n",
    "conda activate llm\n",
    "\n",
    "```\n",
    "<img src=\"Assets/conda_llm.png\">\n",
    "\n",
    "\n",
    "\n",
    "### With the llm environment active, use pip to install ipex-llm for GPU. \n",
    "\n",
    "* pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ (for US)\n",
    "* pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/cn/ (for CN)\n",
    "\n",
    "<img src=\"Assets/llm12.png\">\n",
    "\n",
    "## Verify Installation\n",
    "You can verify if ipex-llm is successfully installed following below steps.\n",
    "\n",
    "### Open the Miniforge Prompt and activate the Python environment llm you previously created:\n",
    "```\n",
    "conda activate llm\n",
    "```\n",
    "\n",
    "\n",
    "### Set the following environment variables according to your device:\n",
    "For Intel iGPU:\n",
    "\n",
    "* set SYCL_CACHE_PERSISTENT=1\n",
    "* set BIGDL_LLM_XMX_DISABLED=1\n",
    "  \n",
    "<img src=\"Assets/llm13.png\">\n",
    "\n",
    "\n",
    "\n",
    "### Run Python Code\n",
    "Launch the Python interactive shell by typing python in the Miniforge Prompt window and then press Enter.\n",
    "Copy following code to Miniforge Prompt line by line and press Enter after copying each line.\n",
    "\n",
    "```\n",
    "import torch \n",
    "from ipex_llm.transformers import AutoModel,AutoModelForCausalLM    \n",
    "tensor_1 = torch.randn(1, 1, 40, 128).to('xpu') \n",
    "tensor_2 = torch.randn(1, 1, 128, 40).to('xpu') \n",
    "print(torch.matmul(tensor_1, tensor_2).size()) \n",
    "\n",
    "```\n",
    "\n",
    "You should see at the end:\n",
    "torch.Size([1, 1, 40, 40])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69541b1d-162b-4c8c-9311-df27db43531e",
   "metadata": {},
   "source": [
    "### Install these packages to run the below code\n",
    "```\n",
    "pip install tiktoken transformers_stream_generator einops\n",
    "\n",
    "```\n",
    "```\n",
    "conda install -c conda-forge jupyter\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41ca93a8-e995-4e0d-8361-0deb262fbe1c",
   "metadata": {},
   "source": [
    "## Code Walkthrough\n",
    "\n",
    "Now let’s play with a real LLM. We’ll be using the Qwen-1.8B-Chat model, a 1.8 billion parameter LLM for this demonstration. \n",
    "Follow the steps below to setup and run the model, and observe how it responds to a prompt “What is AI?”.\n",
    "\n",
    "Below id the code snippet using Hugging Face's Transformers library to utilize the AutoModelForCausalLM class\n",
    "\n",
    "Note: When running LLMs on Intel iGPUs with limited memory size, we recommend setting cpu_embedding=True in the from_pretrained function. This will allow the memory-intensive embedding layer to utilize the CPU instead of GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6de17b6-dd1a-4d97-be1a-d7a856411146",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from ipex_llm.transformers import AutoModelForCausalLM\n",
    "from transformers import AutoTokenizer, GenerationConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "121808e7-7832-4cc2-8743-b0bc4d956112",
   "metadata": {},
   "source": [
    "AutoModelForCausalLM is a class that automatically selects the appropriate model architecture for causal language modeling based on the pre-trained model specified, and AutoTokenizer is a class that automatically selects the appropriate tokenizer.\n",
    "We then initialize the tokenizer and the model using the from_pretrained method, which loads the pre-trained "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe7fd7d2-8f01-4171-8cbc-8094fbc971ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen-1_8B-Chat\",\n",
    "                                           trust_remote_code=True)\n",
    "\n",
    "# Load Model using ipex-llm and load it to GPU\n",
    "model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen-1_8B-Chat\",\n",
    "                                             load_in_4bit=True,\n",
    "                                             cpu_embedding=True,\n",
    "                                             trust_remote_code=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f4a7d9a-e1b6-4627-a507-1b537e533bdb",
   "metadata": {},
   "source": [
    "#### Load it to the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cebdd91b-bd33-46d9-88b4-ac11c2eb09bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model.to('xpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88c88efc-88cd-49f5-9f85-17fbd2824b53",
   "metadata": {},
   "source": [
    "We define a text prompt that the model will use as a starting point to generate text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32528937-3c95-400d-bb23-e059b45c1e67",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"What is AI?\"\n",
    "prompt = \"user: {prompt}\\n\\nassistant:\".format(prompt=question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "413c9731-a6f2-4da0-a27d-545b775d49b6",
   "metadata": {},
   "source": [
    "* We use the tokenizer to encode the text prompt into a format that the model can understand. The return_tensors='pt' argument tells the tokenizer to return PyTorch tensors.\n",
    "* We use the model's generate method to generate a sequence of text based on the input prompt. The max_length argument specifies the maximum length of the generated text. \n",
    "* The temperature argument controls the randomness of the output (lower values make the output more deterministic and higher values make it more random).\n",
    "* The num_return_sequences argument specifies the number of different sequences to generate.\n",
    "* We use the tokenizer's decode method to convert the generated sequence of tokens back into human-readable text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3f72f2e-fe84-455b-ad59-5a1391c51a4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "generation_config = GenerationConfig(use_cache=True)\n",
    "with torch.inference_mode():\n",
    "    input_ids = tokenizer.encode(prompt, return_tensors=\"pt\").to('xpu')\n",
    "\n",
    "    print('--------------------------------------Note-----------------------------------------')\n",
    "    print('| For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or |')\n",
    "    print('| Pro A60, it may take several minutes for GPU kernels to compile and initialize. |')\n",
    "    print('| Please be patient until it finishes warm-up...                                  |')\n",
    "    print('-----------------------------------------------------------------------------------')\n",
    "\n",
    "    # To achieve optimal and consistent performance, we recommend a one-time warm-up by running `model.generate(...)` an additional time before starting your actual generation tasks.\n",
    "    # If you're developing an application, you can incorporate this warm-up step into start-up or loading routine to enhance the user experience.\n",
    "    output = model.generate(input_ids,\n",
    "                            do_sample=False,\n",
    "                            max_new_tokens=32,\n",
    "                            generation_config=generation_config) # warm-up\n",
    "\n",
    "    print('Successfully finished warm-up, now start generation...')\n",
    "\n",
    "    output = model.generate(input_ids,\n",
    "                            do_sample=False,\n",
    "                            max_new_tokens=32,\n",
    "                            generation_config=generation_config).cpu()\n",
    "    output_str = tokenizer.decode(output[0], skip_special_tokens=True)\n",
    "    print(output_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec180ac3-e74a-41d9-a9b9-65478dcea556",
   "metadata": {},
   "source": [
    "## Complete code snippet using Streamlit\n",
    "\n",
    "### Install streamlit\n",
    "```\n",
    "pip install streamlit\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "666c1c8a-3355-4a1c-ae3e-06c4cb700ee7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/chat.py\n",
    "import os\n",
    "\n",
    "os.environ[\"SYCL_CACHE_PERSISTENT\"]=\"1\"\n",
    "os.environ[\"BIGDL_LLM_XMX_DISABLED\"]=\"1\"\n",
    "\n",
    "import threading\n",
    "\n",
    "import streamlit as st\n",
    "\n",
    "from ipex_llm.transformers import AutoModelForCausalLM\n",
    "from transformers import AutoTokenizer, GenerationConfig, TextIteratorStreamer\n",
    "import torch\n",
    "\n",
    "\n",
    "MODEL_CACHE = {}\n",
    "\n",
    "\n",
    "def save_model_thread(model, model_path):\n",
    "    model.save_low_bit(model_path)\n",
    "    print(f\"Model saved to {model_path}\")\n",
    "\n",
    "\n",
    "def warmup_model(model, tokenizer):\n",
    "    question = \"Hello, how are you?\"\n",
    "    tokenizer.pad_token = tokenizer.eos_token\n",
    "    if model.name_or_path.startswith(\"microsoft\"):\n",
    "        prompt = f\"<|user|>\\n{question}<|end|>\\n<|assistant|>\"\n",
    "    else:\n",
    "        prompt = \"user: {prompt}\\n\\nassistant:\".format(prompt=question)\n",
    "    dummy_input = tokenizer(prompt, return_tensors=\"pt\").to(\"xpu\")\n",
    "    generation_config = GenerationConfig(use_cache=True,\n",
    "                                        top_k=50,\n",
    "                                        top_p=0.95,\n",
    "                                        temperature=0.7, do_sample=True,\n",
    "                                        )\n",
    "    _ = model.generate(**dummy_input, generation_config=generation_config)\n",
    "    print(\"Model warmed up successfully!\")\n",
    "\n",
    "\n",
    "def load_model(model_name: str = \"Qwen/Qwen-1_8B-Chat\"):\n",
    "    if model_name in MODEL_CACHE:\n",
    "        return MODEL_CACHE[model_name]\n",
    "\n",
    "    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n",
    "    model_path = f\"./model_local_cache/{model_name}\"\n",
    "\n",
    "    if os.path.exists(model_path):\n",
    "        print(f\"Loading model from {model_path}\")\n",
    "        model = AutoModelForCausalLM.load_low_bit(\n",
    "            model_path, cpu_embedding=True, trust_remote_code=True\n",
    "        )\n",
    "    else:\n",
    "        print(f\"Loading model from {model_name}\")\n",
    "        model = AutoModelForCausalLM.from_pretrained(\n",
    "            model_name,\n",
    "            load_in_4bit=True,\n",
    "            cpu_embedding=True,\n",
    "            trust_remote_code=True\n",
    "        )\n",
    "        save_model_thread(model, model_path)\n",
    "\n",
    "    model = model.to(\"xpu\")\n",
    "\n",
    "    MODEL_CACHE[model_name] = (model, tokenizer)\n",
    "    print(\"Model loaded successfully!\")\n",
    "    return model, tokenizer\n",
    "\n",
    "\n",
    "def get_response(model, tokenizer, input_text: str):\n",
    "    question = input_text\n",
    "    tokenizer.pad_token = tokenizer.eos_token\n",
    "    if model.name_or_path.startswith(\"microsoft\"):\n",
    "        prompt = f\"<|user|>\\n{question}<|end|>\\n<|assistant|>\"\n",
    "    else:\n",
    "        prompt = \"user: {prompt}\\n\\nassistant:\".format(prompt=question)\n",
    "\n",
    "    with torch.inference_mode():\n",
    "        input_ids = tokenizer(prompt, return_tensors=\"pt\").to(\"xpu\")\n",
    "        streamer = TextIteratorStreamer(\n",
    "            tokenizer, skip_prompt=False, skip_special_tokens=True\n",
    "        )\n",
    "\n",
    "        generation_config = GenerationConfig(\n",
    "            use_cache=True, top_k=50, top_p=0.95,\n",
    "            temperature=0.7, do_sample=True,\n",
    "        )\n",
    "\n",
    "        kwargs = dict(\n",
    "            input_ids,\n",
    "            streamer=streamer,\n",
    "            max_new_tokens=256,\n",
    "            generation_config=generation_config,\n",
    "        )\n",
    "        thread = threading.Thread(target=model.generate, kwargs=kwargs)\n",
    "        thread.start()\n",
    "    return streamer\n",
    "\n",
    "\n",
    "def main():\n",
    "    if \"model\" not in st.session_state:\n",
    "        st.session_state.model = None\n",
    "    if \"tokenizer\" not in st.session_state:\n",
    "        st.session_state.tokenizer = None\n",
    "\n",
    "    st.header(\"Lets chat... 🐻‍❄️\")\n",
    "    selected_model = st.selectbox(\n",
    "        \"Please select a model\", (\"Qwen/Qwen-1_8B-Chat\", \"microsoft/Phi-3-mini-4k-instruct\")\n",
    "    )\n",
    "\n",
    "    if st.button(\"Load Model\"):\n",
    "        with st.spinner(\"Loading...\"):\n",
    "            st.session_state.model, st.session_state.tokenizer = load_model(\n",
    "                model_name=selected_model\n",
    "            )\n",
    "            if (\n",
    "                st.session_state.model is not None\n",
    "                and st.session_state.tokenizer is not None\n",
    "            ):\n",
    "                st.success(\"Model loaded successfully!\")\n",
    "                st.info(\"Warming up the model...\")\n",
    "                warmup_model(st.session_state.model, st.session_state.tokenizer)\n",
    "                st.success(\"Model warmed up and ready to use!\")\n",
    "            else:\n",
    "                st.error(\"Failed to load the model.\")\n",
    "\n",
    "    chat_container = st.container()\n",
    "    with chat_container:\n",
    "        st.subheader(\"Chat\")\n",
    "        input_text = st.text_input(\"Enter your input here...\")\n",
    "        if st.button(\"Generate\"):\n",
    "            if st.session_state.model is None or st.session_state.tokenizer is None:\n",
    "                st.warning(\"Please load the model first.\")\n",
    "            else:\n",
    "                with st.spinner(\"Running....🐎\"):\n",
    "                    streamer = get_response(\n",
    "                        st.session_state.model, st.session_state.tokenizer, input_text\n",
    "                    )\n",
    "                    st.write_stream(streamer)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "   main()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55460213-46a0-4d12-b7e2-38245a7bd9aa",
   "metadata": {},
   "source": [
    "### Sample output stream\n",
    "\n",
    "Below is the screenshot of sample output and offloaded to the iGPU\n",
    "\n",
    "<img src=\"Assets/pytorch_st.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c3a79f8-6b8e-4db8-8ac7-534b3518201e",
   "metadata": {},
   "source": [
    "* Reference: https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_windows_gpu.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "326a0eec-e519-4325-93ec-be1cdb261dfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "! streamlit run src/chat.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df414428-f466-412b-a1a1-8706ffd7adf1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}