5_llm_quantization_sycl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "652ea6c8-8d13-4228-853e-fad46db470f5",
   "metadata": {},
   "source": [
    "# Quantization using SYCL backend on AI PC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71e0aeac-58b1-4114-95f1-7d3a7a4c34f2",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook demonstrates how to quantize a model on Windows AI PC with Intel GPUs. It applies to Intel Core Ultra and Core 11 - 14 gen integrated GPUs (iGPUs), as well as Intel Arc Series GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97cf7db8-9529-47dd-b41d-81b22c8d5848",
   "metadata": {},
   "source": [
    "## What is an AIPC\n",
    "\n",
    "What is an AI PC you ask?\n",
    "\n",
    "Here is an [explanation](https://www.intel.com/content/www/us/en/newsroom/news/what-is-an-ai-pc.htm#gs.a55so1) from Intel:\n",
    "\n",
    "”An AI PC has a CPU, a GPU and an NPU, each with specific AI acceleration capabilities. An NPU, or neural processing unit, is a specialized accelerator that handles artificial intelligence (AI) and machine learning (ML) tasks right on your PC instead of sending data to be processed in the cloud. The GPU and CPU can also process these workloads, but the NPU is especially good at low-power AI calculations. The AI PC represents a fundamental shift in how our computers operate. It is not a solution for a problem that didn’t exist before. Instead, it promises to be a huge improvement for everyday PC usages.”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4682eb3e-540b-4814-8142-c54efc32f31b",
   "metadata": {},
   "source": [
    "## Install Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f8b6d2-34af-44ad-8363-dea57660bc00",
   "metadata": {},
   "source": [
    "### Step 1: System Preparation\n",
    "\n",
    "To set up your AIPC for running with Intel iGPUs, follow these essential steps:\n",
    "\n",
    "1. Update Intel GPU Drivers: Ensure your system has the latest Intel GPU drivers, which are crucial for optimal performance and compatibility. You can download these directly from Intel's [official website](https://www.intel.com/content/www/us/en/download/785597/intel-arc-iris-xe-graphics-windows.html) . Once you have installed the official drivers, you could also install Intel ARC Control to monitor the gpu:\n",
    "\n",
    "   <img src=\"Assets/gpu_arc_control.png\">\n",
    "\n",
    "\n",
    "2. Install Visual Studio 2022 Community edition with C++: Visual Studio 2022, along with the “Desktop Development with C++” workload, is required. This prepares your environment for C++ based extensions used by the intel SYCL backend that powers accelerated Ollama. You can download VS 2022 Community edition from the official site, [here](https://visualstudio.microsoft.com/downloads/).\n",
    "\n",
    "3. Install conda-forge: conda-forge will manage your Python environments and dependencies efficiently, providing a clean, minimal base for your Python setup. Visit conda-forge's [installation site](https://conda-forge.org/download/) to install for windows.\n",
    "\n",
    "4. Install [Intel® oneAPI Base Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html)\n",
    "\n",
    "   "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8040fd21-7782-4b97-a0eb-327816328f17",
   "metadata": {},
   "source": [
    "## Step 2: Install Llamacpp for SYCL\n",
    "The llama.cpp SYCL backend is designed to support Intel GPU firstly. Based on the cross-platform feature of SYCL.\n",
    "\n",
    "### After installation of conda-forge, open the Miniforge Prompt, and create a new python environment:\n",
    "  ```\n",
    "  conda create -n llm-sycl python=3.11\n",
    "\n",
    "  ```\n",
    "\n",
    "### Activate the new environment\n",
    "```\n",
    "conda activate llm-sycl\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/llm4.png\">\n",
    "\n",
    "### With the llm-sycl environment active, enable oneAPI environment. \n",
    "Type oneapi in the windows search and then open the Intel oneAPI command prompt for Intel 64 for Visual Studio 2022 App.\n",
    "\n",
    "<img src=\"Assets/oneapi1.png\">\n",
    "\n",
    "#### Run the below command in the VS command prompt and you should see the below sycl devices displayed in the console\n",
    "There should be one or more level-zero GPU devices displayed as ext_oneapi_level_zero:gpu.\n",
    "\n",
    "```\n",
    "sycl-ls\n",
    "\n",
    "```\n",
    "\n",
    "<img src=\"Assets/oneapi2.png\">\n",
    "\n",
    "### Install build tools\n",
    "\n",
    "* Download & install [cmake for Windows](https://cmake.org/download/):\n",
    "* The new Visual Studio will install Ninja as default. (If not, please install it manually: https://ninja-build.org/)\n",
    "\n",
    "### Install llama.cpp\n",
    "\n",
    "* git clone the llama.cpp repo\n",
    "  \n",
    "  ```\n",
    "  git clone https://github.com/ggerganov/llama.cpp.git\n",
    "\n",
    "  ```\n",
    "  \n",
    "* On the oneAPI command line window, step into the llama.cpp main directory and run the following:\n",
    "  \n",
    "  ```\n",
    "  @call \"C:\\Program Files (x86)\\Intel\\oneAPI\\setvars.bat\" intel64 --force\n",
    "\n",
    "    # Option 1: Use FP32 (recommended for better performance in most cases)\n",
    "  cmake -B build -G \"Ninja\" -DGGML_SYCL=ON -DCMAKE_C_COMPILER=cl -DCMAKE_CXX_COMPILER=icx  -DCMAKE_BUILD_TYPE=Release\n",
    "    \n",
    "    # Option 2: Or FP16\n",
    "  cmake -B build -G \"Ninja\" -DGGML_SYCL=ON -DCMAKE_C_COMPILER=cl -DCMAKE_CXX_COMPILER=icx  -DCMAKE_BUILD_TYPE=Release -DGGML_SYCL_F16=ON\n",
    "    \n",
    "    cmake --build build --config Release -j\n",
    "\n",
    "  ```\n",
    "\n",
    "### Below shows a simple example to show how to run a community GGUF model with llama.cpp for SYCL\n",
    "* Download the model from huggingface and prepare the model for inference\n",
    "* Run the model for example as below\n",
    "* Open the mini-forge prompt, activate the llm-sycl environment and enable oneAPI enviroment as below\n",
    "\n",
    "  ```\n",
    "  \"C:\\Program Files (x86)\\Intel\\oneAPI\\setvars.bat\" intel64  \n",
    "  ```\n",
    "* List the sycl devices as below\n",
    "\n",
    "  ```\n",
    "  build\\bin\\ls-sycl-device.exe\n",
    "\n",
    "  ```\n",
    "* Run inference\n",
    "```\n",
    "build\\bin\\llama-cli.exe -m models\\llama-2-7b.Q4_0.gguf -p \"Building a website can be done in 10 simple steps:\\nStep 1:\" -n 400 -e -ngl 33 -s 0 -sm none -mg 0\n",
    "```\n",
    "\n",
    "<img src=\"Assets/cmd1.png\">\n",
    "\n",
    "### Below is an example output\n",
    "\n",
    "<img src=\"Assets/out1.png\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01173f7a-0725-4b34-aabc-7e6582b87da4",
   "metadata": {},
   "source": [
    "## Run the inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3ad64c2-2432-4cb0-8a3d-856daad1dc18",
   "metadata": {},
   "outputs": [],
   "source": [
    "! ..\\git_llamacpp\\llama.cpp\\build\\bin\\llama-cli.exe -m Qwen1.5-4B.Q4_K_M.gguf -p \"Building a website can be done in 10 simple steps:\\nStep 1:\" -n 400 -e -ngl 25 -s 0 -sm none -mg 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b36ad00-2c2f-4005-9a20-e42c0c533aa1",
   "metadata": {},
   "source": [
    "## Quantization of the Models on AI PC\n",
    "\n",
    "* Quantization: Reduces the precision of the model's parameters (e.g., from 32-bit floating-point to 8-bit or 4-bit integers), decreasing the model size and often speeding up inference with minimal impact on accuracy.\n",
    "\n",
    "* When quantizing to 4 bits, each value is represented with only 4 bits, significantly reducing the amount of data needed to store and process information. This reduction in data size leads to several advantages, including decreased memory usage and faster processing speeds, which are particularly beneficial for deploying models on AI PCs.\n",
    "\n",
    "*  Additionally, 4-bit quantization can lead to lower power consumption, making it an attractive option for AI PCs with GPUs and NPus\n",
    "\n",
    "*  **llama-3-8b-instruct** - Llama 3 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. The Llama 3 instruction tuned models are optimized for dialogue use cases and outperform many of the available open source chat models on common industry benchmarks. More details about model can be found in [Meta blog post](https://ai.meta.com/blog/meta-llama-3/), [model website](https://llama.meta.com/llama3) and [model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct).\n",
    ">**Note**: run model with demo, you will need to accept license agreement. \n",
    ">You must be a registered user in 🤗 Hugging Face Hub. Please visit [HuggingFace model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), carefully read terms of usage and click accept button.  You will need to use an access token for the code below to run. For more information on access tokens, refer to [this section of the documentation](https://huggingface.co/docs/hub/security-tokens).\n",
    ">You can login on Hugging Face Hub in notebook environment, using following code:\n",
    "\n",
    "* **llama-2-7b-chat** - LLama 2 is the second generation of LLama models developed by Meta. Llama 2 is a collection of pre-trained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. llama-2-7b-chat is 7 billions parameters version of LLama 2 finetuned and optimized for dialogue use case. More details about model can be found in the [paper](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/), [repository](https://github.com/facebookresearch/llama) and [HuggingFace model card](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf).\n",
    ">**Note**: run model with demo, you will need to accept license agreement. \n",
    ">You must be a registered user in 🤗 Hugging Face Hub. Please visit [HuggingFace model card](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), carefully read terms of usage and click accept button.  You will need to use an access token for the code below to run. For more information on access tokens, refer to [this section of the documentation](https://huggingface.co/docs/hub/security-tokens).\n",
    ">You can login on Hugging Face Hub in notebook environment, using following code:\n",
    " \n",
    "```python\n",
    "    ## login to huggingfacehub to get access to pretrained model \n",
    "\n",
    "    from huggingface_hub import notebook_login, whoami\n",
    "\n",
    "    try:\n",
    "        whoami()\n",
    "        print('Authorization token already provided')\n",
    "    except OSError:\n",
    "        notebook_login()\n",
    "```\n",
    "\n",
    "* **phi3-mini-instruct** - The Phi-3-Mini is a 3.8B parameters, lightweight, state-of-the-art open model trained with the Phi-3 datasets that includes both synthetic data and the filtered publicly available websites data with a focus on high-quality and reasoning dense properties. More details about model can be found in [model card](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct), [Microsoft blog](https://aka.ms/phi3blog-april) and [technical report](https://aka.ms/phi3-tech-report).\n",
    "* **qwen2-1.5b-instruct/qwen2-7b-instruct** - Qwen2 is the new series of Qwen large language models.Compared with the state-of-the-art open source language models, including the previous released Qwen1.5, Qwen2 has generally surpassed most open source models and demonstrated competitiveness against proprietary models across a series of benchmarks targeting for language understanding, language generation, multilingual capability, coding, mathematics, reasoning, etc.\n",
    "For more details, please refer to [model_card](https://huggingface.co/Qwen/Qwen2-7B-Instruct), [blog](https://qwenlm.github.io/blog/qwen2/), [GitHub](https://github.com/QwenLM/Qwen2), and [Documentation](https://qwen.readthedocs.io/en/latest/).\n",
    "\n",
    "* **neural-chat-7b-v3-1** - Mistral-7b model fine-tuned using Intel Gaudi. The model fine-tuned on the open source dataset [Open-Orca/SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) and aligned with [Direct Preference Optimization (DPO) algorithm](https://arxiv.org/abs/2305.18290). More details can be found in [model card](https://huggingface.co/Intel/neural-chat-7b-v3-1) and [blog post](https://medium.com/@NeuralCompressor/the-practice-of-supervised-finetuning-and-direct-preference-optimization-on-habana-gaudi2-a1197d8a3cd3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2f5a77a-dc31-4202-b12e-94beabc90a9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login, whoami\n",
    "try:\n",
    "    whoami()\n",
    "    print('Authorization token already provided')\n",
    "except OSError:\n",
    "    notebook_login()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d48253fa-3ec0-44db-bf86-c2060e59eccc",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install ipywidgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff9b63a3-4f1d-40c5-a34b-49f755021f4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "\n",
    "model = widgets.Dropdown(\n",
    "    options=['phi3-mini-instruct', 'llama-2-7b-chat', 'qwen2-1.5b-instruct', 'llama-3-8b-instruct', 'neural-chat-7b-v3-1' ],\n",
    "    value='llama-3-8b-instruct',  # Default value\n",
    "    description=\"Select Model:\",\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cee80c1-1f86-41b7-977c-e7207e123aab",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"microsoft/Phi-3-mini-4k-instruct\"\n",
    "model_path = \"./phi3/\"\n",
    "\n",
    "if model.value == \"phi3-mini-instruct\":\n",
    "    model_id = \"microsoft/Phi-3-mini-4k-instruct\"\n",
    "    model_path = \"./phi3/\"\n",
    "    model_fp16 = \"Phi-3-mini-4k-instruct.Fp16.gguf\"\n",
    "    model_gguf = \"Phi-3-mini-4k-instruct.Q4_K_M.gguf\"\n",
    "elif model.value == \"llama-2-7b-chat\":\n",
    "    model_id = \"meta-llama/Llama-2-7b-chat-hf\"\n",
    "    model_fp16 = \"Llama-2-7b-chat-hf.Fp16.gguf\"\n",
    "    model_path = \"./llama2/\"\n",
    "    model_gguf = \"Llama-2-7b-chat-hf.Q4_K_M.gguf\"\n",
    "elif model.value == \"llama-3-8b-instruct\":\n",
    "    model_id = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n",
    "    model_fp16 = \"llama-3-8b-instruct.Fp16.gguf\"\n",
    "    model_path = \"./llama3/\"\n",
    "    model_gguf = \"llama-3-8b-instruct.Q4_K_M.gguf\"\n",
    "elif model.value == \"qwen2-1.5b-instruct\":\n",
    "    model_id = \"Qwen/Qwen1.5-4B-Chat\"\n",
    "    model_fp16 = \"Qwen1.5-4B-Chat.Fp16.gguf\"\n",
    "    model_path = \"./Qwen/\"\n",
    "    model_gguf = \"Qwen1.5-4B-Chat.Q4_K_M.gguf\"\n",
    "elif model.value == \"neural-chat-7b-v3-1\":\n",
    "    model_id = \"Intel/neural-chat-7b-v3-1\"\n",
    "    model_fp16 = \"neural-chat-7b-v3-1.Fp16.gguf\"\n",
    "    model_path = \"./Intel_neural_chat/\"\n",
    "    model_gguf = \"neural-chat-7b-v3-1.Q4_K_M.gguf\"\n",
    "else:\n",
    "    model_id = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n",
    "    model_fp16 = \"llama-3-8b-instruct.Fp16.gguf\"\n",
    "    model_path = \"./llama3/\"\n",
    "    model_gguf = \"llama-3-8b-instruct.Q4_K_M.gguf\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd1af28e-0c46-4f5c-bc43-ea10106d7175",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Selected model {model.value}\")\n",
    "print(f\"Selected model \", model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fe1fbc0-39a1-45c3-9cff-82550de83345",
   "metadata": {},
   "source": [
    "### Initialize oneAPI environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28cf0670-a537-4c08-a1dc-44b6e87e90ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "!@call \"C:\\Program Files (x86)\\Intel\\oneAPI\\setvars.bat\" intel64 --force"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0ebe9e1-037c-4800-afcb-9e2f0b512a2d",
   "metadata": {},
   "source": [
    "### Download the model from Huggingface to local folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9ad6f11-17c8-4fde-988d-63b94f1cccbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import snapshot_download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f11f272-1d1d-4288-8904-4760fe64a176",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "snapshot_download(repo_id = model_id,local_dir = model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0cd8b4f-3cdd-4ac6-a197-3ab101701491",
   "metadata": {},
   "source": [
    "### Convert the model to GGUF format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d8fbafc-4d27-41b4-821f-91fc5b958910",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "start_time = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4573ac8c-ad6f-4326-ad5b-b4d475e61bfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python ..\\git_llamacpp\\llama.cpp\\convert-hf-to-gguf.py {model_path} --outtype f16 --outfile ./converted_models/{model_fp16}\n",
    "\n",
    "end_time = time.time()\n",
    "total_time = end_time - start_time\n",
    "print(f\"Model conversion time: {total_time} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "732b4e61-4d1f-474f-858e-05279bc20367",
   "metadata": {},
   "source": [
    "### Quantize the model to 4bit (Q4_K_M) format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "172d0733-b8dc-4beb-a329-e77aafb6d9de",
   "metadata": {},
   "outputs": [],
   "source": [
    "! ..\\git_llamacpp\\llama.cpp\\build\\bin\\llama-quantize.exe ./converted_models/{model_fp16} ./quantized_models/{model_gguf} Q4_K_M"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18016571-9389-442f-9d30-cce4258c4c84",
   "metadata": {},
   "source": [
    "### Run the Inference using the quantized model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c3ae993-66b9-44ec-b4eb-74c0eff3c417",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "start_time = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d150b2b-31f0-436e-a46c-2b118f1f2d1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "! ..\\git_llamacpp\\llama.cpp\\build\\bin\\llama-cli.exe -m ./quantized_models/{model_gguf} -p \"Building a website can be done in 10 simple steps:\\nStep 1:\" -n 100 -e -ngl 33 -s 0 -sm none -mg 0\n",
    "\n",
    "end_time = time.time()\n",
    "total_time = end_time - start_time\n",
    "print(f\"Model warmup and Inference time: {total_time} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b3d9dbd-3f6d-406f-ade6-3c20bc7a92fb",
   "metadata": {},
   "source": [
    "### Upload the model to Huggingface hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec02187c-d515-4031-b15a-2e6eec1b4793",
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import login\n",
    "login()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8741dd0-8aff-4870-8153-039d53e255fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import HfApi, HfFolder, Repository, create_repo, upload_file\n",
    "#from huggingface_hub import HfApi, HfFolder, create_repo, upload_file\n",
    "import os\n",
    "\n",
    "# Authentication\n",
    "token = HfFolder.get_token()  # Make sure you have logged in using `huggingface-cli login` or set the token manually\n",
    "if token is None:\n",
    "    raise ValueError(\"Hugging Face token not found. Please login using `huggingface-cli login`.\")\n",
    "\n",
    "# Define repository details\n",
    "model_file_path = \"./quantized_models/\" + model_gguf  # Your GGUG model file name\n",
    "model_file_name = model_gguf\n",
    "repo_name = model.value  # Repository name\n",
    "organization = \"Your org name\"  # Change this to your Hugging Face username or organization\n",
    "repo_url = f\"{organization}/{repo_name}\"\n",
    "\n",
    "# Initialize HfApi to interact with Hugging Face Hub\n",
    "api = HfApi()\n",
    "\n",
    "# Check if the repository exists, if not, create it\n",
    "\n",
    "api.create_repo(repo_id=repo_name, token=token, private=True)  # Set `private=True` for a private repository\n",
    "\n",
    "# Clone the repository locally (if not already cloned)\n",
    "\n",
    "api.upload_file(\n",
    "    path_or_fileobj=model_file_path,\n",
    "    path_in_repo=model_file_name,\n",
    "    repo_id=repo_url,\n",
    "    repo_type=\"model\",\n",
    ")\n",
    "\n",
    "print(f\"Model file {model_file_name} successfully uploaded to Hugging Face at {repo_url}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4d70ebd-10d3-488a-baff-5ac2d39b41ae",
   "metadata": {},
   "source": [
    "### Download model from huggingface_hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c107346-dcfc-461e-945e-115e723a90d2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import snapshot_download\n",
    "snapshot_download(repo_id=repo_url, local_dir=\"./download_models/\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd47d1d9-414b-4cd2-b20e-4c36871f1145",
   "metadata": {},
   "source": [
    "#### Run the inference locally on AI PC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65e5cd95-18a4-4879-9d3d-05e302448ff6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! ..\\git_llamacpp\\llama.cpp\\build\\bin\\llama-cli.exe -m ./download_models/{model_gguf} -p \"Building a website can be done in 10 simple steps:\\nStep 1:\" -n 100 -e -ngl 33 -s 0 -sm none -mg 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec180ac3-e74a-41d9-a9b9-65478dcea556",
   "metadata": {},
   "source": [
    "## Example output\n",
    "\n",
    "<img src=\"Assets/output_latest.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92387fa9-2376-49a7-a94b-a29f254a0471",
   "metadata": {},
   "source": [
    "* Reference:https://github.com/ggerganov/llama.cpp/blob/master/docs/backend/SYCL.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ac73234-1851-42ad-9b6c-67ba9562db32",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}