FInetune-Mistral-7B-Instruct-v.01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This notebook has been tailored to specifically finetune a Mistral-7B-Instruct-v0.1 Model and then quantize it using AWQ algorithm\n",
    "\n",
    "The following content were used as reference to get an idea about the fine tuning process and the necessary hyperparameters in order to get the best training model\n",
    "- [Tuning-the-Finetuning](https://github.com/avisoori-databricks/Tuning-the-Finetuning)\n",
    "- [Mistral Mastery: Fine-Tuning & Fast Inference Guide](https://medium.com/@parikshitsaikia1619/mistral-mastery-fine-tuning-fast-inference-guide-62e163198b06)\n",
    "- [4-bit Transformers with Hugging Face](https://huggingface.co/blog/4bit-transformers-bitsandbytes)\n",
    "- [Transformers for Legal Language](https://huggingface.co/docs/trl/en/sft_trainer)\n",
    "- [AutoAWQ](https://github.com/casper-hansen/AutoAWQ)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the necessary prerequisite modules\n",
    "\n",
    "This notebook was run on the pytorch version 2.1 and cuda version at 12.1 \n",
    "Do bear in mind to properly set up the virtual environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Pcn2sEybFx-N",
    "outputId": "332cf8fa-e68c-4454-81fe-e5b60b40d9ef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install -U -q trl accelerate bitsandbytes peft transformers autoawq torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional\n",
    "Installing Weights and Biases to track the training progress\n",
    "You can use the below template to create a wandb project where the training run will then be assigned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mvpmb\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    }
   ],
   "source": [
    "!pip install -q wandb -U\n",
    "\n",
    "import wandb, os\n",
    "wandb.login()\n",
    "\n",
    "wandb_project = \"mistral-finetune-v2.1\"\n",
    "if len(wandb_project) > 0:\n",
    "    os.environ[\"WANDB_PROJECT\"] = wandb_project\n",
    "    os.environ[\"WANDB_RESUME\"] = \"must\"\n",
    "    os.environ[\"WANDB_RUN_ID\"] = \"f5rpfdmw\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform the necessary imports to load the model to memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "LJiKqv94Fx-O"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 173,
     "referenced_widgets": [
      "d1bc3325a8514348b28ec2e011fa5f8d",
      "1106338e991c4330b4861a6c17d02626",
      "66774908197443ad9404bea954dec1ba",
      "4182685ca9214d358ff662a0dc0f1c6e",
      "f9778293f43c42ca9a36ed53a5967ba9",
      "b494fc0d2671466fa98f336d12288900",
      "91e18d25b2f04eb380bf878d3a026593",
      "dce5fa03fb404040b61539528c880d40",
      "db371d9e30264ec4bba8bf9402be3ecc",
      "263dbaae05e74e0180c8dad4c124eff5",
      "20806fddc2164c3bb127ba38fa6d704a"
     ]
    },
    "id": "a0krjXqiFx-P",
    "outputId": "20fee676-d79e-42aa-ac4f-e87dcc6694c8"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "546946061cdc4e9683bbb0e722df4a85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#quantization config to load the model in 4 bit rather than 16 bit, reduces the memory usage and also enables qLora finetuning\n",
    "nf4_config = BitsAndBytesConfig(\n",
    "  load_in_4bit=True,\n",
    "  bnb_4bit_quant_type=\"nf4\",\n",
    "  bnb_4bit_use_double_quant=True,\n",
    "  bnb_4bit_compute_dtype=torch.bfloat16\n",
    ")\n",
    "\n",
    "# Load model\n",
    "model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.1\",  device_map='auto', quantization_config=nf4_config)\n",
    "\n",
    "# Load the tokenizer, can use use_fast=True to enable fast tokenizers\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    \"mistralai/Mistral-7B-Instruct-v0.1\")\n",
    "\n",
    "#Set up padding token, since we are doing constricted generation, we have to stop generating at the end of the JSON\n",
    "# tokenizer.pad_token = tokenizer.eos_token yielded errors so using unk_token as padding token\n",
    "tokenizer.pad_token = tokenizer.unk_token\n",
    "tokenizer.padding_side = \"right\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the dataset\n",
    "Make sure to prepare the dataset according to the appropriate jsonl format where each entry is formatted as a JSON string.  \n",
    "\n",
    "The sample format of the dataset is ```{\"prompt\": \"<s>[INST] ### Instruction : You are given a sequence of text. Generate a logical question, correct answer and three incorrect answers suitable for mcq from the input text.The output should be formatted as a json in the below format.{\\\"type\\\": \\\"object\\\", \\\"properties\\\": {\\\"Output\\\": {\\\"type\\\": \\\"object\\\", \\\"properties\\\": {\\\"question\\\": {\\\"type\\\": \\\"string\\\"}, \\\"correct_answer\\\": {\\\"type\\\": \\\"string\\\"}, \\\"incorrect_answers\\\": {\\\"type\\\": \\\"array\\\", \\\"items\\\": {\\\"type\\\": \\\"string\\\"}}}, \\\"required\\\": [\\\"question\\\", \\\"correct_answer\\\", \\\"incorrect_answers\\\"]}}, \\\"required\\\": [\\\"Output\\\"]} ### Input : a = 3\\nif a==3 :\\nb = a * 2  \\nif a < 4:\\nb = a + 2\\nif a > 2:\\nb = a * 2\\nPage 21Exercise 2 - What is the final value in b ?  [/INST]\", \"completion\": \"### Output : {\\n \\\"Output\\\": {\\n  \\\"question\\\": \\\"In the given program, what is the final value in b ?\\\",\\n  \\\"correct_answer\\\": \\\"6\\\",\\n  \\\"incorrect_answers\\\": [\\n   \\\"5\\\",\\n   \\\"4\\\",\\n   \\\"7\\\"\\n  ]\\n }\\n} </s>\"}``` Notice the use of ```<s>``` and ```</s>``` and the ```[INST]``` and ```[/INST] tokens and use of a single prompt field.\n",
    "\n",
    "Prevent splitting the dataset into training and validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "eH3FRWFlFx-P"
   },
   "outputs": [],
   "source": [
    "path_to_dataset = \"dataset_mistral_instruct1.jsonl\"\n",
    "\n",
    "dataset = load_dataset(\"json\", data_files=path_to_dataset, split='train')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the model as a PEFT model\n",
    "Can efficiently load the model as a peft model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "WsvfVldmFx-P"
   },
   "outputs": [],
   "source": [
    "\n",
    "from peft import get_peft_model\n",
    "\n",
    "# target modules selects which will be used in qlora finetuning\n",
    "target_modules = ['q_proj','k_proj','v_proj','o_proj','gate_proj','down_proj','up_proj','lm_head']\n",
    "\n",
    "#sample lora config\n",
    "lora_config = LoraConfig(\n",
    "    r=8,#or r=16\n",
    "    lora_alpha=8,\n",
    "    lora_dropout=0.05,\n",
    "    bias=\"none\",\n",
    "    target_modules = target_modules,\n",
    "    task_type=\"CAUSAL_LM\",\n",
    ")\n",
    "model = get_peft_model(model, lora_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## define training arguments to train the model\n",
    "Do not recommend chaning the optimization algorithm.\n",
    "per device train batch size and gradient accumulation steps can be lowered to 1 and 2 respectively. \n",
    "Using a lower learning rate yielded erroneous results.\n",
    "Recommends fine tuning upto one epoch as after one epoch the model looses the ability to generate responses using constricted generation.\n",
    "Have set to save the model at each epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "Gd7qntxrKCaw"
   },
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments\n",
    "from datetime import datetime\n",
    "\n",
    "output_dir = './mistral-finetune-v2.1'\n",
    "run_name = 'v2.1.1'\n",
    "\n",
    "per_device_train_batch_size = 2\n",
    "gradient_accumulation_steps = 4\n",
    "optim = 'paged_adamw_8bit'\n",
    "learning_rate = 1e-5\n",
    "max_grad_norm = 0.3\n",
    "warmup_ratio = 0.03\n",
    "lr_scheduler_type = \"linear\"\n",
    "\n",
    "training_args = TrainingArguments(\n",
    "   output_dir=output_dir,\n",
    "    save_strategy=\"epoch\",\n",
    "    num_train_epochs = 3.0,\n",
    "    logging_steps = 5,\n",
    "    per_device_train_batch_size=per_device_train_batch_size,\n",
    "    gradient_accumulation_steps=gradient_accumulation_steps,\n",
    "    optim=optim,\n",
    "    learning_rate=learning_rate,\n",
    "    bf16=True,\n",
    "    max_grad_norm=max_grad_norm,\n",
    "    warmup_ratio=warmup_ratio,\n",
    "    group_by_length=True,\n",
    "    lr_scheduler_type=lr_scheduler_type,\n",
    "    report_to=\"wandb\",           # Comment this out if you don't want to use weights & baises\n",
    "    run_name=f\"{run_name}-{datetime.now().strftime('%Y-%m-%d-%H-%M')}\"          # Name of the W&B run (optional)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the sfttrainer\n",
    "max sequence length was set to 1024 to enable autoawq quantization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "a895b403f3f9422e92574e4271d1e20f",
      "7eeffa27dc664dfe8a48577c7fe6bc9c",
      "87976280b2dd4ef2b9084a73e4b893ae",
      "696cb7051b0f4faf82e93503e90df888",
      "98c88104f2794626a6e652343dfa4bdc",
      "85bd43bb08624112a232ba6004dc22ea",
      "4dc7a48299f04635ac34c53f1e3b1024",
      "474ef59f490f4dd5a74de45b8d598ddc",
      "8ebab2597d4f4cf6aa6f07c74d8fe193",
      "4f7f10ba4527423cb184adaff5917011",
      "2cfd0bc997c745cea3fcebd3b084223c"
     ]
    },
    "id": "7VqBC2lJFx-P",
    "outputId": "03b07148-d029-4ab2-a28b-fe31c2ad3d00"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:accelerate.utils.other:Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    }
   ],
   "source": [
    "model.config.use_cache = False \n",
    "trainer = SFTTrainer(\n",
    "    model = model,\n",
    "    train_dataset=dataset,\n",
    "    tokenizer=tokenizer,\n",
    "    peft_config=lora_config,\n",
    "    max_seq_length=1024,\n",
    "    args=training_args\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Commence the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mvpmb\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.16.2"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>/workspace/wandb/run-20240129_000725-683t5t5h</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/vpmb/huggingface/runs/683t5t5h' target=\"_blank\">v2.1.1-2024-01-29-00-07</a></strong> to <a href='https://wandb.ai/vpmb/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/vpmb/huggingface' target=\"_blank\">https://wandb.ai/vpmb/huggingface</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/vpmb/huggingface/runs/683t5t5h' target=\"_blank\">https://wandb.ai/vpmb/huggingface/runs/683t5t5h</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='696' max='696' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [696/696 38:03, Epoch 2/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.129500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.358800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>2.276400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>2.248900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>2.220200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>2.130800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>35</td>\n",
       "      <td>2.063600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>40</td>\n",
       "      <td>2.001700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>45</td>\n",
       "      <td>1.839400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>1.777000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>55</td>\n",
       "      <td>1.858400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>1.640500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>65</td>\n",
       "      <td>1.570500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>70</td>\n",
       "      <td>1.434700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>75</td>\n",
       "      <td>1.405900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>80</td>\n",
       "      <td>1.232000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>85</td>\n",
       "      <td>1.207100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>90</td>\n",
       "      <td>1.112700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>95</td>\n",
       "      <td>1.026300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>0.937200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>105</td>\n",
       "      <td>1.281000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>110</td>\n",
       "      <td>1.204200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>115</td>\n",
       "      <td>1.030300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>120</td>\n",
       "      <td>1.107500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>125</td>\n",
       "      <td>0.996300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>130</td>\n",
       "      <td>0.965800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>135</td>\n",
       "      <td>0.961700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>140</td>\n",
       "      <td>0.878100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>145</td>\n",
       "      <td>0.759300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>0.701700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>155</td>\n",
       "      <td>1.247200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>160</td>\n",
       "      <td>1.095900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>165</td>\n",
       "      <td>0.953600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>170</td>\n",
       "      <td>0.997300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>175</td>\n",
       "      <td>0.942100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>180</td>\n",
       "      <td>0.887300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>185</td>\n",
       "      <td>0.786800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>190</td>\n",
       "      <td>0.730800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>195</td>\n",
       "      <td>0.694000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>0.605200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>205</td>\n",
       "      <td>1.062700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>210</td>\n",
       "      <td>0.949700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>215</td>\n",
       "      <td>0.853100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>220</td>\n",
       "      <td>0.823600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>225</td>\n",
       "      <td>0.772400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>230</td>\n",
       "      <td>0.676100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>235</td>\n",
       "      <td>0.824700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>240</td>\n",
       "      <td>1.037500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>245</td>\n",
       "      <td>0.991100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>0.907800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>255</td>\n",
       "      <td>0.849600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>260</td>\n",
       "      <td>0.806700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>265</td>\n",
       "      <td>0.812100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>270</td>\n",
       "      <td>0.742100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>275</td>\n",
       "      <td>0.674700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>280</td>\n",
       "      <td>0.625900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>285</td>\n",
       "      <td>0.920900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>290</td>\n",
       "      <td>1.042600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>295</td>\n",
       "      <td>0.923800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>0.834500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>305</td>\n",
       "      <td>0.875600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>310</td>\n",
       "      <td>0.766200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>315</td>\n",
       "      <td>0.806500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>320</td>\n",
       "      <td>0.700700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>325</td>\n",
       "      <td>0.667300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>330</td>\n",
       "      <td>0.567600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>335</td>\n",
       "      <td>0.916300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>340</td>\n",
       "      <td>0.984300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>345</td>\n",
       "      <td>0.943600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>0.809900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>355</td>\n",
       "      <td>0.861000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>360</td>\n",
       "      <td>0.799100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>365</td>\n",
       "      <td>0.723200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>370</td>\n",
       "      <td>0.704700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>375</td>\n",
       "      <td>0.667500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>380</td>\n",
       "      <td>0.614600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>385</td>\n",
       "      <td>0.914600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>390</td>\n",
       "      <td>0.925900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>395</td>\n",
       "      <td>0.909600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>0.846200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>405</td>\n",
       "      <td>0.821900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>410</td>\n",
       "      <td>0.819300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>415</td>\n",
       "      <td>0.775800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>420</td>\n",
       "      <td>0.731900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>425</td>\n",
       "      <td>0.649400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>430</td>\n",
       "      <td>0.552700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>435</td>\n",
       "      <td>0.747000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>440</td>\n",
       "      <td>0.900000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>445</td>\n",
       "      <td>0.840300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>0.773600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>455</td>\n",
       "      <td>0.714500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>460</td>\n",
       "      <td>0.601500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>465</td>\n",
       "      <td>0.526700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>470</td>\n",
       "      <td>1.014300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>475</td>\n",
       "      <td>0.948800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>480</td>\n",
       "      <td>0.825400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>485</td>\n",
       "      <td>0.805700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>490</td>\n",
       "      <td>0.742800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>495</td>\n",
       "      <td>0.735300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>0.727500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>505</td>\n",
       "      <td>0.681700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>510</td>\n",
       "      <td>0.575400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>515</td>\n",
       "      <td>0.540700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>520</td>\n",
       "      <td>1.059400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>525</td>\n",
       "      <td>1.003800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>530</td>\n",
       "      <td>0.843400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>535</td>\n",
       "      <td>0.838400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>540</td>\n",
       "      <td>0.795100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>545</td>\n",
       "      <td>0.729800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>550</td>\n",
       "      <td>0.706500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>555</td>\n",
       "      <td>0.659300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>560</td>\n",
       "      <td>0.578400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>565</td>\n",
       "      <td>0.516000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>570</td>\n",
       "      <td>1.079700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>575</td>\n",
       "      <td>0.860800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>580</td>\n",
       "      <td>0.886100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>585</td>\n",
       "      <td>0.807000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>590</td>\n",
       "      <td>0.809600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>595</td>\n",
       "      <td>0.753700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>0.685900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>605</td>\n",
       "      <td>0.664900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>610</td>\n",
       "      <td>0.549000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>615</td>\n",
       "      <td>0.483900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>620</td>\n",
       "      <td>1.076500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>625</td>\n",
       "      <td>0.898700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>630</td>\n",
       "      <td>0.900100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>635</td>\n",
       "      <td>0.804300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>640</td>\n",
       "      <td>0.820000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>645</td>\n",
       "      <td>0.744400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>650</td>\n",
       "      <td>0.722600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>655</td>\n",
       "      <td>0.636000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>660</td>\n",
       "      <td>0.598600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>665</td>\n",
       "      <td>0.483400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>670</td>\n",
       "      <td>1.007700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>675</td>\n",
       "      <td>0.766900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>680</td>\n",
       "      <td>0.794600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>685</td>\n",
       "      <td>0.704200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>690</td>\n",
       "      <td>0.678900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>695</td>\n",
       "      <td>0.595000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/peft/utils/save_and_load.py:131: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\n",
      "  warnings.warn(\"Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\")\n",
      "/usr/local/lib/python3.10/dist-packages/peft/utils/save_and_load.py:131: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\n",
      "  warnings.warn(\"Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\")\n",
      "/usr/local/lib/python3.10/dist-packages/peft/utils/save_and_load.py:131: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\n",
      "  warnings.warn(\"Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\")\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=696, training_loss=0.9421885708967844, metrics={'train_runtime': 2291.0674, 'train_samples_per_second': 2.434, 'train_steps_per_second': 0.304, 'total_flos': 9.670066722627994e+16, 'train_loss': 0.9421885708967844, 'epoch': 2.99})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the model\n",
    "Again load the model from scratch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd94dae3d1d541e89fa969575ee6e48f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/571 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0835fdc28a8d431bb4236075cc5b5f58",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors.index.json:   0%|          | 0.00/25.1k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fbb27dbae64c4c5387d8d593a7c0f6b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f50e9d87b8f4e7287b84d63e8f1d861",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00001-of-00002.safetensors:   0%|          | 0.00/9.94G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "397c8ba49bd147f0b0b9db06b3cc1d2b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00002-of-00002.safetensors:   0%|          | 0.00/4.54G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5422aee2ef8a45c0beccc0918491d86b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d210c9143cbd4439bb9611964f784a3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "generation_config.json:   0%|          | 0.00/116 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eca4ed8ddabf4d1bb37fd57a72aa5b1a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/1.47k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "240af01fd235431c90156023a8514c5f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.model:   0%|          | 0.00/493k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07e1229690b944e39ceb3f6429cc6e8c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/1.80M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d66e19e012124f69b47b2fc9801e1027",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/72.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.1\",  device_map='auto', torch_dtype=torch.bfloat16)\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    \"mistralai/Mistral-7B-Instruct-v0.1\")\n",
    "tokenizer.pad_token = tokenizer.unk_token\n",
    "tokenizer.padding_side = \"right\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the peft model\n",
    "Load the saved peft model, by merging the model and the peft layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import PeftModel\n",
    "ft_model = PeftModel.from_pretrained(model, \"mistral-finetune-v2.1/checkpoint-232\")\n",
    "ft_model.config.use_cache =False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference the model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<s> [INST]### Instruction : You are given a sequence of text. Generate a logical question, correct answer and three incorrect answers suitable for mcq from the input text.The output should be formatted as a json in the below format.\" + \"{\"type\": \"object\", \"properties\": {\"Output\": {\"type\": \"object\", \"properties\": {\"question\": {\"type\": \"string\"}, \"correct_answer\": {\"type\": \"string\"}, \"incorrect_answers\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}}}, \"required\": [\"question\", \"correct_answer\", \"incorrect_answers\"]}}, \"required\": [\"Output\"]}\"\n",
      "### Input :Object Oriented Principles we saw so far\n",
      "• Encapsulation\n",
      "It keeps the data and the code safe from external interference. It is a\n",
      "mechanism for restricting direct access to some of the object’s\n",
      "component. Binding the data with the code that manipulates it.\n",
      "• Inheritance\n",
      "Inheritance allows a class to use the properties and methods of another\n",
      "class. In other words, the derived class inherits the states and behaviors\n",
      "from the base class.\n",
      "• Polymorphism.\n",
      "Polymorphism is the ability of an object to take on many forms. The most\n",
      "common use of polymorphism in OOP occurs when a parent class\n",
      "reference is used to refer to a child class object.[/INST]\n",
      "### Output :\n",
      "[\n",
      "    {\n",
      "        \"Output\": {\n",
      "            \"question\": \"What is the main purpose of encapsulation in Object Oriented Programming?\",\n",
      "            \"correct_answer\": \"To keep the data and the code safe from external interference.\",\n",
      "            \"incorrect_answers\": [\n",
      "                \"To allow direct access to some of the object’s component.\",\n",
      "                \"To restrict the manipulation of data.\",\n",
      "                \"To allow external modification of data.\"\n",
      "            ]\n",
      "        }\n",
      "    }\n",
      "]</s>\n"
     ]
    }
   ],
   "source": [
    "eval_prompt = \"\"\"[INST]### Instruction : You are given a sequence of text. Generate a logical question, correct answer and three incorrect answers suitable for mcq from the input text.The output should be formatted as a json in the below format.\" + \"{\\\"type\\\": \\\"object\\\", \\\"properties\\\": {\\\"Output\\\": {\\\"type\\\": \\\"object\\\", \\\"properties\\\": {\\\"question\\\": {\\\"type\\\": \\\"string\\\"}, \\\"correct_answer\\\": {\\\"type\\\": \\\"string\\\"}, \\\"incorrect_answers\\\": {\\\"type\\\": \\\"array\\\", \\\"items\\\": {\\\"type\\\": \\\"string\\\"}}}, \\\"required\\\": [\\\"question\\\", \\\"correct_answer\\\", \\\"incorrect_answers\\\"]}}, \\\"required\\\": [\\\"Output\\\"]}\"\n",
    "### Input :Object Oriented Principles we saw so far\n",
    "• Encapsulation\n",
    "It keeps the data and the code safe from external interference. It is a\n",
    "mechanism for restricting direct access to some of the object’s\n",
    "component. Binding the data with the code that manipulates it.\n",
    "• Inheritance\n",
    "Inheritance allows a class to use the properties and methods of another\n",
    "class. In other words, the derived class inherits the states and behaviors\n",
    "from the base class.\n",
    "• Polymorphism.\n",
    "Polymorphism is the ability of an object to take on many forms. The most\n",
    "common use of polymorphism in OOP occurs when a parent class\n",
    "reference is used to refer to a child class object.[/INST]\n",
    "### Output :\n",
    "\"\"\"\n",
    "\n",
    "model_input = tokenizer(eval_prompt, return_tensors=\"pt\").to(\"cuda\")\n",
    "\n",
    "ft_model.eval()\n",
    "with torch.no_grad():\n",
    "    print(tokenizer.decode(ft_model.generate(**model_input, max_new_tokens=128, )[0],skip_special_tokens=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save the model by merging the qlora adapter with the base model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('./merged_model/tokenizer_config.json',\n",
       " './merged_model/special_tokens_map.json',\n",
       " './merged_model/tokenizer.model',\n",
       " './merged_model/added_tokens.json',\n",
       " './merged_model/tokenizer.json')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = ft_model.merge_and_unload()\n",
    "\n",
    "#Save the merged model in a directory in the safetensors format\n",
    "model_dir = \"./merged_model/\"\n",
    "model.save_pretrained(model_dir, safe_serialization=True)\n",
    "\n",
    "#Save the custom tokenizer in the same directory\n",
    "tokenizer.save_pretrained(model_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup autoawq for quantization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "96cdc698204846eca3bd62412340b2a7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from awq import AutoAWQForCausalLM\n",
    "from transformers import AutoTokenizer\n",
    "model_path = \"merged_model\"\n",
    "model = AutoAWQForCausalLM.from_pretrained(model_path, safetensors=True)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install autoawq\n",
    "Make sure either the cuda version is 12.1 or otherwise use this to download the wheels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting autoawq==0.1.6+cu118\n",
      "  Downloading https://github.com/casper-hansen/AutoAWQ/releases/download/v0.1.6/autoawq-0.1.6+cu118-cp310-cp310-linux_x86_64.whl (20.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.0/20.0 MB\u001b[0m \u001b[31m76.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: torch>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (2.1.0+cu118)\n",
      "Requirement already satisfied: transformers>=4.35.0 in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (4.37.1)\n",
      "Requirement already satisfied: tokenizers>=0.12.1 in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (0.15.1)\n",
      "Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (0.26.1)\n",
      "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (0.1.99)\n",
      "Requirement already satisfied: lm-eval in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (0.4.0)\n",
      "Requirement already satisfied: texttable in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (1.7.0)\n",
      "Requirement already satisfied: toml in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (0.10.2)\n",
      "Requirement already satisfied: attributedict in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (0.3.0)\n",
      "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (4.25.2)\n",
      "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (0.16.0+cu118)\n",
      "Requirement already satisfied: tabulate in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.6+cu118) (0.9.0)\n",
      "Requirement already satisfied: huggingface_hub<1.0,>=0.16.4 in /usr/local/lib/python3.10/dist-packages (from tokenizers>=0.12.1->autoawq==0.1.6+cu118) (0.20.3)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.1.6+cu118) (3.13.1)\n",
      "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.1.6+cu118) (4.4.0)\n",
      "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.1.6+cu118) (1.12)\n",
      "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.1.6+cu118) (3.0)\n",
      "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.1.6+cu118) (3.1.2)\n",
      "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.1.6+cu118) (2023.10.0)\n",
      "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->autoawq==0.1.6+cu118) (2.1.0)\n",
      "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq==0.1.6+cu118) (1.24.1)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq==0.1.6+cu118) (23.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq==0.1.6+cu118) (6.0.1)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq==0.1.6+cu118) (2023.12.25)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq==0.1.6+cu118) (2.31.0)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq==0.1.6+cu118) (0.4.2)\n",
      "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq==0.1.6+cu118) (4.66.1)\n",
      "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate->autoawq==0.1.6+cu118) (5.9.6)\n",
      "Requirement already satisfied: rootpath>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from attributedict->autoawq==0.1.6+cu118) (0.1.1)\n",
      "Requirement already satisfied: inspecta>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from attributedict->autoawq==0.1.6+cu118) (0.1.3)\n",
      "Requirement already satisfied: colour-runner>=0.0.5 in /usr/local/lib/python3.10/dist-packages (from attributedict->autoawq==0.1.6+cu118) (0.1.1)\n",
      "Requirement already satisfied: deepdiff>=3.3.0 in /usr/local/lib/python3.10/dist-packages (from attributedict->autoawq==0.1.6+cu118) (6.7.1)\n",
      "Requirement already satisfied: tox>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from attributedict->autoawq==0.1.6+cu118) (4.12.1)\n",
      "Requirement already satisfied: coverage>=4.5.2 in /usr/local/lib/python3.10/dist-packages (from attributedict->autoawq==0.1.6+cu118) (7.4.1)\n",
      "Requirement already satisfied: codecov>=2.0.15 in /usr/local/lib/python3.10/dist-packages (from attributedict->autoawq==0.1.6+cu118) (2.1.13)\n",
      "Requirement already satisfied: evaluate in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (0.4.1)\n",
      "Requirement already satisfied: datasets>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (2.16.1)\n",
      "Requirement already satisfied: jsonlines in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (4.0.0)\n",
      "Requirement already satisfied: numexpr in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (2.9.0)\n",
      "Requirement already satisfied: peft>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (0.7.1)\n",
      "Requirement already satisfied: pybind11>=2.6.2 in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (2.11.1)\n",
      "Requirement already satisfied: pytablewriter in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (1.2.0)\n",
      "Requirement already satisfied: rouge-score>=0.0.4 in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (0.1.2)\n",
      "Requirement already satisfied: sacrebleu>=1.5.0 in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (2.4.0)\n",
      "Requirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (1.4.0)\n",
      "Requirement already satisfied: sqlitedict in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (2.1.0)\n",
      "Requirement already satisfied: tqdm-multiprocess in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (0.0.11)\n",
      "Requirement already satisfied: zstandard in /usr/local/lib/python3.10/dist-packages (from lm-eval->autoawq==0.1.6+cu118) (0.22.0)\n",
      "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision->autoawq==0.1.6+cu118) (9.3.0)\n",
      "Requirement already satisfied: blessings in /usr/local/lib/python3.10/dist-packages (from colour-runner>=0.0.5->attributedict->autoawq==0.1.6+cu118) (1.7)\n",
      "Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from colour-runner>=0.0.5->attributedict->autoawq==0.1.6+cu118) (2.16.1)\n",
      "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (15.0.0)\n",
      "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (0.6)\n",
      "Requirement already satisfied: dill<0.3.8,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (0.3.7)\n",
      "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (2.2.0)\n",
      "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (3.4.1)\n",
      "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (0.70.15)\n",
      "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (3.9.2)\n",
      "Requirement already satisfied: ordered-set<4.2.0,>=4.0.2 in /usr/local/lib/python3.10/dist-packages (from deepdiff>=3.3.0->attributedict->autoawq==0.1.6+cu118) (4.1.0)\n",
      "Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.10/dist-packages (from evaluate->lm-eval->autoawq==0.1.6+cu118) (0.18.0)\n",
      "Requirement already satisfied: six>=1.11.0 in /usr/lib/python3/dist-packages (from inspecta>=0.1.0->attributedict->autoawq==0.1.6+cu118) (1.16.0)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from inspecta>=0.1.0->attributedict->autoawq==0.1.6+cu118) (2.4.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.35.0->autoawq==0.1.6+cu118) (2.1.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.35.0->autoawq==0.1.6+cu118) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.35.0->autoawq==0.1.6+cu118) (1.26.13)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.35.0->autoawq==0.1.6+cu118) (2022.12.7)\n",
      "Requirement already satisfied: coloredlogs>=10.0 in /usr/local/lib/python3.10/dist-packages (from rootpath>=0.1.0->attributedict->autoawq==0.1.6+cu118) (15.0.1)\n",
      "Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm-eval->autoawq==0.1.6+cu118) (2.1.0)\n",
      "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm-eval->autoawq==0.1.6+cu118) (3.8.1)\n",
      "Requirement already satisfied: portalocker in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm-eval->autoawq==0.1.6+cu118) (2.8.2)\n",
      "Requirement already satisfied: colorama in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm-eval->autoawq==0.1.6+cu118) (0.4.6)\n",
      "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm-eval->autoawq==0.1.6+cu118) (4.9.3)\n",
      "Requirement already satisfied: scipy>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm-eval->autoawq==0.1.6+cu118) (1.12.0)\n",
      "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm-eval->autoawq==0.1.6+cu118) (1.3.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm-eval->autoawq==0.1.6+cu118) (3.2.0)\n",
      "Requirement already satisfied: cachetools>=5.3.2 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.6+cu118) (5.3.2)\n",
      "Requirement already satisfied: chardet>=5.2 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.6+cu118) (5.2.0)\n",
      "Requirement already satisfied: platformdirs>=4.1 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.6+cu118) (4.1.0)\n",
      "Requirement already satisfied: pluggy>=1.3 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.6+cu118) (1.4.0)\n",
      "Requirement already satisfied: pyproject-api>=1.6.1 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.6+cu118) (1.6.1)\n",
      "Requirement already satisfied: tomli>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.6+cu118) (2.0.1)\n",
      "Requirement already satisfied: virtualenv>=20.25 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.6+cu118) (20.25.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=2.0.1->autoawq==0.1.6+cu118) (2.1.2)\n",
      "Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonlines->lm-eval->autoawq==0.1.6+cu118) (23.1.0)\n",
      "Requirement already satisfied: setuptools>=38.3.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm-eval->autoawq==0.1.6+cu118) (68.2.2)\n",
      "Requirement already satisfied: DataProperty<2,>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm-eval->autoawq==0.1.6+cu118) (1.0.1)\n",
      "Requirement already satisfied: mbstrdecoder<2,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm-eval->autoawq==0.1.6+cu118) (1.1.3)\n",
      "Requirement already satisfied: pathvalidate<4,>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm-eval->autoawq==0.1.6+cu118) (3.2.0)\n",
      "Requirement already satisfied: tabledata<2,>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm-eval->autoawq==0.1.6+cu118) (1.3.3)\n",
      "Requirement already satisfied: tcolorpy<1,>=0.0.5 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm-eval->autoawq==0.1.6+cu118) (0.1.4)\n",
      "Requirement already satisfied: typepy<2,>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm-eval->autoawq==0.1.6+cu118) (1.3.2)\n",
      "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=2.0.1->autoawq==0.1.6+cu118) (1.3.0)\n",
      "Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.10/dist-packages (from coloredlogs>=10.0->rootpath>=0.1.0->attributedict->autoawq==0.1.6+cu118) (10.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (1.3.1)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (1.4.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (1.9.4)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (4.0.3)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.8.0 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm-eval->autoawq==0.1.6+cu118) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2018.9 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm-eval->autoawq==0.1.6+cu118) (2023.3.post1)\n",
      "Requirement already satisfied: distlib<1,>=0.3.7 in /usr/local/lib/python3.10/dist-packages (from virtualenv>=20.25->tox>=3.0.0->attributedict->autoawq==0.1.6+cu118) (0.3.8)\n",
      "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->rouge-score>=0.0.4->lm-eval->autoawq==0.1.6+cu118) (8.1.7)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets>=2.0.0->lm-eval->autoawq==0.1.6+cu118) (2023.4)\n",
      "Installing collected packages: autoawq\n",
      "  Attempting uninstall: autoawq\n",
      "    Found existing installation: autoawq 0.1.8\n",
      "    Uninstalling autoawq-0.1.8:\n",
      "      Successfully uninstalled autoawq-0.1.8\n",
      "Successfully installed autoawq-0.1.6+cu118\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install https://github.com/casper-hansen/AutoAWQ/releases/download/v0.1.6/autoawq-0.1.6+cu118-cp310-cp310-linux_x86_64.whl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AutoAWQ quantize\n",
    "Quantize, train and save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b562f77787c4ff9b8078e1078914594",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/167 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/huggingface_hub/repocard.py:105: UserWarning: Repo card metadata block was not found. Setting CardData to empty.\n",
      "  warnings.warn(\"Repo card metadata block was not found. Setting CardData to empty.\")\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f6d7d3ae25af4c7382ebcd8c11fd9aa1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/471M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8bfabb1d8b904c4cb16369abd0728475",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating validation split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AWQ: 100%|██████████| 32/32 [15:51<00:00, 29.72s/it]\n",
      "WARNING:root:`quant_config.json` is being deprecated in the future in favor of quantization_config in config.json.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('quant/tokenizer_config.json',\n",
       " 'quant/special_tokens_map.json',\n",
       " 'quant/tokenizer.model',\n",
       " 'quant/added_tokens.json',\n",
       " 'quant/tokenizer.json')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quant_path=\"quant\"\n",
    "quant_config = { \"zero_point\": True, \"q_group_size\": 128, \"w_bit\": 4, \"version\": \"GEMM\" }\n",
    "# Quantize\n",
    "model.quantize(tokenizer, quant_config=quant_config)\n",
    "\n",
    "# Save quantized model\n",
    "model.save_quantized(quant_path)\n",
    "tokenizer.save_pretrained(quant_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the quanitzed model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Replacing layers...: 100%|██████████| 32/32 [00:03<00:00,  8.71it/s]\n",
      "Fusing layers...: 100%|██████████| 32/32 [00:05<00:00,  5.36it/s]\n"
     ]
    }
   ],
   "source": [
    "model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True)\n",
    "tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run inference on the quantized model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    }
   ],
   "source": [
    "tokens = tokenizer(\n",
    "    text=eval_prompt, \n",
    "    return_tensors='pt'\n",
    ").input_ids.cuda()\n",
    "\n",
    "\n",
    "# Generate output\n",
    "generation_output = model.generate(\n",
    "    tokens, \n",
    "    max_new_tokens=512\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<s> [INST]### Instruction : You are given a sequence of text. Generate a logical question, correct answer and three incorrect answers suitable for mcq from the input text.The output should be formatted as a json in the below format.\" + \"{\"type\": \"object\", \"properties\": {\"Output\": {\"type\": \"object\", \"properties\": {\"question\": {\"type\": \"string\"}, \"correct_answer\": {\"type\": \"string\"}, \"incorrect_answers\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}}}, \"required\": [\"question\", \"correct_answer\", \"incorrect_answers\"]}}, \"required\": [\"Output\"]}\"\\n### Input :Object Oriented Principles we saw so far\\n• Encapsulation\\nIt keeps the data and the code safe from external interference. It is a\\nmechanism for restricting direct access to some of the object’s\\ncomponent. Binding the data with the code that manipulates it.\\n• Inheritance\\nInheritance allows a class to use the properties and methods of another\\nclass. In other words, the derived class inherits the states and behaviors\\nfrom the base class.\\n• Polymorphism.\\nPolymorphism is the ability of an object to take on many forms. The most\\ncommon use of polymorphism in OOP occurs when a parent class\\nreference is used to refer to a child class object.[/INST]\\n### Output :\\n[\\n{\\n    \"Output\": {\\n        \"question\": \"What is the main purpose of encapsulation in Object Oriented Programming?\",\\n        \"correct_answer\": \"To keep the data and the code safe from external interference.\",\\n        \"incorrect_answers\": [\\n            \"To allow direct access to some of the object’s component.\",\\n            \"To bind the data with the code that manipulates it.\",\\n            \"To allow a class to use the properties and methods of another class.\"\\n        ]\\n    }\\n}\\n]</s>'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(generation_output[0],skip_special_tokens=False)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "1106338e991c4330b4861a6c17d02626": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b494fc0d2671466fa98f336d12288900",
      "placeholder": "​",
      "style": "IPY_MODEL_91e18d25b2f04eb380bf878d3a026593",
      "value": "Loading checkpoint shards:   0%"
     }
    },
    "20806fddc2164c3bb127ba38fa6d704a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "263dbaae05e74e0180c8dad4c124eff5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2cfd0bc997c745cea3fcebd3b084223c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4182685ca9214d358ff662a0dc0f1c6e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_263dbaae05e74e0180c8dad4c124eff5",
      "placeholder": "​",
      "style": "IPY_MODEL_20806fddc2164c3bb127ba38fa6d704a",
      "value": " 0/3 [00:00&lt;?, ?it/s]"
     }
    },
    "474ef59f490f4dd5a74de45b8d598ddc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4dc7a48299f04635ac34c53f1e3b1024": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4f7f10ba4527423cb184adaff5917011": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "66774908197443ad9404bea954dec1ba": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dce5fa03fb404040b61539528c880d40",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_db371d9e30264ec4bba8bf9402be3ecc",
      "value": 0
     }
    },
    "696cb7051b0f4faf82e93503e90df888": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4f7f10ba4527423cb184adaff5917011",
      "placeholder": "​",
      "style": "IPY_MODEL_2cfd0bc997c745cea3fcebd3b084223c",
      "value": " 1859/1859 [00:02&lt;00:00, 698.18 examples/s]"
     }
    },
    "7eeffa27dc664dfe8a48577c7fe6bc9c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_85bd43bb08624112a232ba6004dc22ea",
      "placeholder": "​",
      "style": "IPY_MODEL_4dc7a48299f04635ac34c53f1e3b1024",
      "value": "Map: 100%"
     }
    },
    "85bd43bb08624112a232ba6004dc22ea": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "87976280b2dd4ef2b9084a73e4b893ae": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_474ef59f490f4dd5a74de45b8d598ddc",
      "max": 1859,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_8ebab2597d4f4cf6aa6f07c74d8fe193",
      "value": 1859
     }
    },
    "8ebab2597d4f4cf6aa6f07c74d8fe193": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "91e18d25b2f04eb380bf878d3a026593": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "98c88104f2794626a6e652343dfa4bdc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a895b403f3f9422e92574e4271d1e20f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7eeffa27dc664dfe8a48577c7fe6bc9c",
       "IPY_MODEL_87976280b2dd4ef2b9084a73e4b893ae",
       "IPY_MODEL_696cb7051b0f4faf82e93503e90df888"
      ],
      "layout": "IPY_MODEL_98c88104f2794626a6e652343dfa4bdc"
     }
    },
    "b494fc0d2671466fa98f336d12288900": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d1bc3325a8514348b28ec2e011fa5f8d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_1106338e991c4330b4861a6c17d02626",
       "IPY_MODEL_66774908197443ad9404bea954dec1ba",
       "IPY_MODEL_4182685ca9214d358ff662a0dc0f1c6e"
      ],
      "layout": "IPY_MODEL_f9778293f43c42ca9a36ed53a5967ba9"
     }
    },
    "db371d9e30264ec4bba8bf9402be3ecc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "dce5fa03fb404040b61539528c880d40": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f9778293f43c42ca9a36ed53a5967ba9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}