8_llm_sycl_multimodal.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "652ea6c8-8d13-4228-853e-fad46db470f5",
   "metadata": {},
   "source": [
    "# Visual-language assistant using SYCL backend on AI PC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71e0aeac-58b1-4114-95f1-7d3a7a4c34f2",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook demonstrates how to install LLamacpp for SYCL on Windows with Intel GPUs. It applies to Intel Core Ultra and Core 11 - 14 gen integrated GPUs (iGPUs), as well as Intel Arc Series GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97cf7db8-9529-47dd-b41d-81b22c8d5848",
   "metadata": {},
   "source": [
    "## What is an AIPC\n",
    "\n",
    "What is an AI PC you ask?\n",
    "\n",
    "Here is an [explanation](https://www.intel.com/content/www/us/en/newsroom/news/what-is-an-ai-pc.htm#gs.a55so1):\n",
    "\n",
    "”An AI PC has a CPU, a GPU and an NPU, each with specific AI acceleration capabilities. An NPU, or neural processing unit, is a specialized accelerator that handles artificial intelligence (AI) and machine learning (ML) tasks right on your PC instead of sending data to be processed in the cloud. The GPU and CPU can also process these workloads, but the NPU is especially good at low-power AI calculations. The AI PC represents a fundamental shift in how our computers operate. It is not a solution for a problem that didn’t exist before. Instead, it promises to be a huge improvement for everyday PC usages.”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4682eb3e-540b-4814-8142-c54efc32f31b",
   "metadata": {},
   "source": [
    "## Install Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f8b6d2-34af-44ad-8363-dea57660bc00",
   "metadata": {},
   "source": [
    "### Step 1: System Preparation\n",
    "\n",
    "To set up your AIPC for running with Intel iGPUs, follow these essential steps:\n",
    "\n",
    "1. Update Intel GPU Drivers: Ensure your system has the latest Intel GPU drivers, which are crucial for optimal performance and compatibility. You can download these directly from Intel's [official website](https://www.intel.com/content/www/us/en/download/785597/intel-arc-iris-xe-graphics-windows.html) . Once you have installed the official drivers, you could also install Intel ARC Control to monitor the gpu:\n",
    "\n",
    "   <img src=\"Assets/gpu_arc_control.png\">\n",
    "\n",
    "\n",
    "2. Install Visual Studio 2022 Community edition with C++: Visual Studio 2022, along with the “Desktop Development with C++” workload, is required. This prepares your environment for C++ based extensions used by the intel SYCL backend that powers accelerated Ollama. You can download VS 2022 Community edition from the official site, [here](https://visualstudio.microsoft.com/downloads/).\n",
    "\n",
    "3. Install conda-forge: conda-forge will manage your Python environments and dependencies efficiently, providing a clean, minimal base for your Python setup. Visit conda-forge's [installation site](https://conda-forge.org/download/) to install for windows.\n",
    "\n",
    "4. Install [Intel® oneAPI Base Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html)\n",
    "\n",
    "   "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8040fd21-7782-4b97-a0eb-327816328f17",
   "metadata": {},
   "source": [
    "## Step 2: Install Llamacpp python for SYCL\n",
    "The llama.cpp SYCL backend is designed to support Intel GPU firstly. Based on the cross-platform feature of SYCL.\n",
    "\n",
    "### After installation of conda-forge, open the Miniforge Prompt, and create a new python environment:\n",
    "  ```\n",
    "  conda create -n llm-sycl python=3.11\n",
    "\n",
    "  ```\n",
    "\n",
    "### Activate the new environment\n",
    "```\n",
    "conda activate llm-sycl\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/llm4.png\">\n",
    "\n",
    "### With the llm-sycl environment active, enable oneAPI environment. \n",
    "Type oneapi in the windows search and then open the Intel oneAPI command prompt for Intel 64 for Visual Studio 2022 App.\n",
    "\n",
    "<img src=\"Assets/oneapi1.png\">\n",
    "\n",
    "#### Run the below command in the VS command prompt and you should see the below sycl devices displayed in the console\n",
    "There should be one or more level-zero GPU devices displayed as ext_oneapi_level_zero:gpu.\n",
    "\n",
    "```\n",
    "sycl-ls\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/oneapi2.png\">\n",
    "\n",
    "### Install build tools\n",
    "\n",
    "* Download & install [cmake for Windows](https://cmake.org/download/):\n",
    "* The new Visual Studio will install Ninja as default. (If not, please install it manually: https://ninja-build.org/)\n",
    "\n",
    "### Install llama.cpp Python\n",
    "\n",
    "  \n",
    "* On the oneAPI command line window, step into the llama.cpp main directory and run the following:\n",
    "  \n",
    "  ```\n",
    "  @call \"C:\\Program Files (x86)\\Intel\\oneAPI\\setvars.bat\" intel64 --force\n",
    "\n",
    "    Open a new terminal and perform the following steps:\n",
    "\n",
    "\n",
    "# Set the environment variables\n",
    "    set CMAKE_GENERATOR=Ninja\n",
    "    set CMAKE_C_COMPILER=cl\n",
    "    set CMAKE_CXX_COMPILER=icx\n",
    "    set CXX=icx\n",
    "    set CC=cl\n",
    "    set CMAKE_ARGS=\"-DGGML_SYCL=ON -DGGML_SYCL_F16=ON -DCMAKE_CXX_COMPILER=icx -DCMAKE_C_COMPILER=cl\"\n",
    "    Install Llamacpp-Python bindings\n",
    "    pip install llama-cpp-python -U --force --no-cache-dir –verbose  ```\n",
    "\n",
    "### Below shows a simple example to show how to run a community GGUF model with llama.cpp for SYCL\n",
    "* Download the model from huggingface and prepare the model for inference\n",
    "* Run the model as below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e65241-f87d-42b0-9b6d-41826a6583a0",
   "metadata": {},
   "source": [
    "## Pulling models from Huggingface hub\n",
    "\n",
    "The below code loads the pre-trained Llama model from huggingface repository specified by the repository ID, filename, and other parameters for the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "462980cc-f9c2-468e-9705-e8b99948d220",
   "metadata": {},
   "source": [
    "### Initialize oneAPI environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8d0962b-6000-44fe-a92b-d95c6458fcc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!@call \"C:\\\\Program Files (x86)\\\\Intel\\\\oneAPI\\\\setvars.bat\" intel64 --force"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c777d38-6990-47d9-9333-5470cd80b68b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from llama_cpp import Llama\n",
    "from llama_cpp.llama_chat_format import MoondreamChatHandler\n",
    "\n",
    "# Initialize the chat handler with a pre-trained model\n",
    "chat_handler = MoondreamChatHandler.from_pretrained(\n",
    "    repo_id=\"vikhyatk/moondream2\",  # Repository ID for the pre-trained model\n",
    "    filename=\"*mmproj*\",  # Filename pattern for the model\n",
    ")\n",
    "\n",
    "# Initialize the model with the pre-trained model and chat handler\n",
    "llm = Llama.from_pretrained(\n",
    "    repo_id=\"vikhyatk/moondream2\",  # Repository ID for the pre-trained model\n",
    "    filename=\"*text-model*\",  # Filename pattern for the text model\n",
    "    chat_handler=chat_handler,  # Chat handler for formatting\n",
    "    n_gpu_layers=-1,  # Uncomment to use GPU acceleration\n",
    "    seed=1337,  # Uncomment to set a specific seed for reproducibility\n",
    "    n_ctx=2048,  # Uncomment to increase the context window size\n",
    "    n_threads=16,  # Number of threads to use\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4785f66e-a0f1-43fb-8195-7314c41e0438",
   "metadata": {},
   "source": [
    "The below code creates a chat completion object specifies the input messages and tells the model to generate text in a streaming fashion.\n",
    "Then we iterates over the generated chunks of text to generate streaming response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2843365-7c12-412f-ba93-3dd89ec7f857",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    " # Create a chat completion request with a user message\n",
    "response = llm.create_chat_completion(\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",  # Role of the message sender\n",
    "            \"content\": [\n",
    "                {\"type\": \"text\", \"text\": \"What is unuusal int this picture?\"},  # Text content of the message\n",
    "                {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/d5fbbd1a-d484-415c-88cb-9986625b7b11\"}}  # Image URL content of the message\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    stream=True  # Stream the response\n",
    ")\n",
    "\n",
    "# Stream and print the response content\n",
    "for chunk in response:\n",
    "    delta = chunk['choices'][0]['delta']  # Extract the delta from the response chunk\n",
    "    if 'content' in delta:  # Check if the delta contains content\n",
    "        print(delta['content'], end='', flush=True)  # Print the content without a newline and flush the output buffer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f5cf673-15d0-4fdc-9e27-9b708e86bb35",
   "metadata": {},
   "source": [
    "### Streamlit Demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77723b93-b914-4728-81b8-047711a178ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install streamlit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31a1b16a-83ed-46d3-bf34-64ee50773b8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/st_visual_answering.py\n",
    "import time\n",
    "from threading import Thread\n",
    "import streamlit as st\n",
    "from llama_cpp import Llama\n",
    "from llama_cpp.llama_chat_format import MoondreamChatHandler\n",
    "import tempfile\n",
    "from PIL import Image\n",
    "import base64\n",
    "\n",
    "# Create a StreamliVisual-language assistantt app that displays the response word by word\n",
    "st.header(\"Visual-language assistant with SYCL 🐻‍❄️\")\n",
    "\n",
    "# Dropdown to select a model\n",
    "selected_model = st.selectbox(\n",
    "    \"Please select a model\", \n",
    "    (\"vikhyatk/moondream2\", \"microsoft/Phi-3-vision-128k-instruct\", \"Intel/llava-gemma-2b\"), \n",
    "    index=0\n",
    ")\n",
    "\n",
    "# File uploader for image\n",
    "img_file_buffer = st.file_uploader('Upload a PNG image', type=[\"jpg\", \"png\", \"gif\"])\n",
    "\n",
    "# Input for image URL\n",
    "# Input for image URL\n",
    "url = st.text_input(\"Enter the URL of the Image:\",value=\"Enter the URL of the Image\", key=\"url_path\")\n",
    "\n",
    "# Display the uploaded image or the image from the URL\n",
    "if img_file_buffer is not None:\n",
    "    try:\n",
    "        image = Image.open(img_file_buffer)\n",
    "        st.image(image, width=600)  # Manually Adjust the width of the image as per requirement\n",
    "    except Exception as e:\n",
    "        st.error(f\"Error loading image: {e}\")\n",
    "else:\n",
    "    st.error(\"Please provide an image URL or upload an image.\")\n",
    "\n",
    "\n",
    "# Input prompt for the question\n",
    "question = st.text_input(\"Enter the question:\", value=\"What's the content of the image?\", key=\"question\")\n",
    "\n",
    "def getfinalresponse(input_text):\n",
    "    try:\n",
    "        # Create a temporary file if an image is uploaded\n",
    "        if img_file_buffer is not None:\n",
    "            with tempfile.NamedTemporaryFile(delete=False) as tmp_file:\n",
    "                tmp_file.write(img_file_buffer.getvalue())\n",
    "                file_path = tmp_file.name\n",
    "\n",
    "            def image_to_base64_data_uri():               \n",
    "                with open(file_path, \"rb\") as img_file:\n",
    "                    base64_data = base64.b64encode(img_file.read()).decode('utf-8')\n",
    "                    return f\"data:image/jpg;base64,{base64_data}\"      \n",
    "\n",
    "        # Initialize the chat handler with a pre-trained model\n",
    "        chat_handler = MoondreamChatHandler.from_pretrained(\n",
    "            repo_id=\"vikhyatk/moondream2\",\n",
    "            filename=\"*mmproj*\",\n",
    "        )\n",
    "\n",
    "        # Initialize the Llama model with the pre-trained model and chat handler\n",
    "        llm = Llama.from_pretrained(\n",
    "            repo_id=selected_model,\n",
    "            filename=\"*text-model*\",\n",
    "            chat_handler=chat_handler,\n",
    "            n_gpu_layers=-1,  # Uncomment to use GPU acceleration\n",
    "            seed=1337,  # Uncomment to set a specific seed\n",
    "            n_ctx=2048,  # Uncomment to increase the context window\n",
    "            n_threads=16,\n",
    "        )\n",
    "\n",
    "        # Create a chat completion request with the appropriate image URL\n",
    "        if img_file_buffer is not None:\n",
    "            response = llm.create_chat_completion(\n",
    "                messages=[\n",
    "                    {\n",
    "                        \"role\": \"user\",\n",
    "                        \"content\": [\n",
    "                            {\"type\": \"text\", \"text\": question},\n",
    "                            {\"type\": \"image_url\", \"image_url\": {\"url\": image_to_base64_data_uri()}}\n",
    "                        ]\n",
    "                    }\n",
    "                ],\n",
    "                stream=True\n",
    "            )\n",
    "        else:\n",
    "            response = llm.create_chat_completion(\n",
    "                messages=[\n",
    "                    {\n",
    "                        \"role\": \"user\",\n",
    "                        \"content\": [\n",
    "                            {\"type\": \"text\", \"text\": question},\n",
    "                            {\"type\": \"image_url\", \"image_url\": {\"url\": url}}\n",
    "                        ]\n",
    "                    }\n",
    "                ],\n",
    "                stream=True\n",
    "            )\n",
    "\n",
    "        # Stream and yield the response content word by word\n",
    "        for chunk in response:\n",
    "            res = chunk['choices'][0]['delta']\n",
    "            if 'content' in res:\n",
    "                word = res['content'].split()\n",
    "                for token in word:\n",
    "                    yield token + \" \"\n",
    "    except Exception as e:\n",
    "        st.error(f\"An error occurred: {e}\")\n",
    "\n",
    "# Generate response when the button is clicked\n",
    "if st.button(\"Generate\"):\n",
    "    with st.spinner(\"Running....🐎\"):\n",
    "        if not question.strip():\n",
    "            st.error(\"Please enter a question.\")\n",
    "        elif not url.strip() and img_file_buffer is None:\n",
    "            st.error(\"Please provide an image URL or upload an image.\")\n",
    "        else:\n",
    "            st.write_stream(getfinalresponse(question))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15a8d984-5e92-4de1-bf88-f8a2e150476d",
   "metadata": {},
   "outputs": [],
   "source": [
    "! streamlit run src/st_visual_answering.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "905150b4-874d-4bf2-aec6-3dffc69a09a6",
   "metadata": {},
   "source": [
    "* Reference:https://github.com/ggerganov/llama.cpp/blob/master/docs/backend/SYCL.md\n",
    "* https://github.com/abetlen/llama-cpp-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ac73234-1851-42ad-9b6c-67ba9562db32",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}