5_biGRU_with_Glove_vectors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a biGRU neural network model with Glove vectors\n",
    "\n",
    "In this notebook, we are going to build a bidirectional Gated Recurrent Unit neural network model, which will use pre-trained Glove word embeddings instead of training the word representations from scratch.\n",
    "\n",
    "Glove vectors can be downloaded from the following page: https://nlp.stanford.edu/projects/glove/ \n",
    "\n",
    "We will use the *glove.6B.100d* Glove vectors that have been trained on 6 billion tokens from *Wikipedia 2014* and *Gigaword 5* sets. Each embedding has the dimensionality of 100.\n",
    "\n",
    "In the end, the model will be evaluated on the test set to determine the generalization error.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model's architecture\n",
    "<br>\n",
    "<div>\n",
    "<img src=\"assets/BiGRU_glove_schema.png\" width=\"900\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weight matrix will be created by *BatchIterator* class based on the Glove word vectors we want to use and vocabulary *word2index* dictionary. For each *word, index* pair from *word2index*, the weight matrix at the position equals *index* will be set with the Glove vector that represents given *word*. For words that haven't been found in the Glove mapping, the random vector will be initialized. Then the weight matrix will be passed to the embedding layer as its parameters. It is very important to set the *embedding_layer.weight.requires_grad = False* so that weights will be not updated during the training. We will also use additional features as in the previous model. After the GRU layer, extra features will be concatenated together with max-pooled, avg-pooled and last hidden size. The concatenated tensor will be next passed to the linear layer, so we have to set the input size to hidden_size * 3 + num_features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building and training the model\n",
    "\n",
    "Let's start with importing all indispensable libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from batch_iterator import BatchIterator\n",
    "from early_stopping import EarlyStopping\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch import device\n",
    "from tqdm import tqdm_notebook\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are going to load the tarining and validation sets. We will import sets with all columns except the review column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the dataset. \n",
    "train_dataset = pd.read_csv('dataset/datasets_feat_clean/train_feat_clean.csv', \n",
    "                            usecols=['clean_review', 'polarity', 'subjectivity', 'word_count', 'UPPERCASE', 'DIGITS',\n",
    "                                      'PROPN', 'VERB', 'NOUN', 'PUNCT', 'ADJ', 'label'])\n",
    "\n",
    "# Change columns order\n",
    "train_dataset = train_dataset[['clean_review', 'polarity', 'subjectivity', 'word_count', 'UPPERCASE', 'DIGITS',\n",
    "                               'PROPN', 'VERB', 'NOUN', 'PUNCT', 'ADJ', 'label']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clean_review</th>\n",
       "      <th>polarity</th>\n",
       "      <th>subjectivity</th>\n",
       "      <th>word_count</th>\n",
       "      <th>UPPERCASE</th>\n",
       "      <th>DIGITS</th>\n",
       "      <th>PROPN</th>\n",
       "      <th>VERB</th>\n",
       "      <th>NOUN</th>\n",
       "      <th>PUNCT</th>\n",
       "      <th>ADJ</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>amaze good wonderful film early ninety franchi...</td>\n",
       "      <td>0.2482</td>\n",
       "      <td>0.5293</td>\n",
       "      <td>391</td>\n",
       "      <td>0.01279</td>\n",
       "      <td>0.002558</td>\n",
       "      <td>0.10740</td>\n",
       "      <td>0.1791</td>\n",
       "      <td>0.2251</td>\n",
       "      <td>0.1381</td>\n",
       "      <td>0.10740</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>wrong end see tell chick go crazy eat old woma...</td>\n",
       "      <td>0.1763</td>\n",
       "      <td>0.5750</td>\n",
       "      <td>145</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.020690</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.2137</td>\n",
       "      <td>0.1724</td>\n",
       "      <td>0.0345</td>\n",
       "      <td>0.07587</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>guess emperor clothe see list pbs night hopefu...</td>\n",
       "      <td>0.1145</td>\n",
       "      <td>0.5557</td>\n",
       "      <td>165</td>\n",
       "      <td>0.07880</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.13330</td>\n",
       "      <td>0.2303</td>\n",
       "      <td>0.0909</td>\n",
       "      <td>0.2060</td>\n",
       "      <td>0.07275</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>earth well movie funny sweet good plot unique ...</td>\n",
       "      <td>0.3810</td>\n",
       "      <td>0.5720</td>\n",
       "      <td>55</td>\n",
       "      <td>0.01819</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.01819</td>\n",
       "      <td>0.1637</td>\n",
       "      <td>0.1637</td>\n",
       "      <td>0.1637</td>\n",
       "      <td>0.14550</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>doe eye high school student kathleen beller fi...</td>\n",
       "      <td>0.2095</td>\n",
       "      <td>0.4666</td>\n",
       "      <td>688</td>\n",
       "      <td>0.01163</td>\n",
       "      <td>0.001453</td>\n",
       "      <td>0.15550</td>\n",
       "      <td>0.2006</td>\n",
       "      <td>0.1831</td>\n",
       "      <td>0.1846</td>\n",
       "      <td>0.05524</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        clean_review  polarity  subjectivity  \\\n",
       "0  amaze good wonderful film early ninety franchi...    0.2482        0.5293   \n",
       "1  wrong end see tell chick go crazy eat old woma...    0.1763        0.5750   \n",
       "2  guess emperor clothe see list pbs night hopefu...    0.1145        0.5557   \n",
       "3  earth well movie funny sweet good plot unique ...    0.3810        0.5720   \n",
       "4  doe eye high school student kathleen beller fi...    0.2095        0.4666   \n",
       "\n",
       "   word_count  UPPERCASE    DIGITS    PROPN    VERB    NOUN   PUNCT      ADJ  \\\n",
       "0         391    0.01279  0.002558  0.10740  0.1791  0.2251  0.1381  0.10740   \n",
       "1         145    0.00000  0.020690  0.00000  0.2137  0.1724  0.0345  0.07587   \n",
       "2         165    0.07880  0.000000  0.13330  0.2303  0.0909  0.2060  0.07275   \n",
       "3          55    0.01819  0.000000  0.01819  0.1637  0.1637  0.1637  0.14550   \n",
       "4         688    0.01163  0.001453  0.15550  0.2006  0.1831  0.1846  0.05524   \n",
       "\n",
       "   label  \n",
       "0      1  \n",
       "1      0  \n",
       "2      0  \n",
       "3      1  \n",
       "4      1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Depict the first 5 rows of the training set\n",
    "train_dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we create batches of our data, we have to normalize the numerical features so that we remove the possibility that one variable is the orders of magnitude greater than other variables, which might cause that the first one dominates other features in the dataset and this is something we don't want to happen in our model.\n",
    "\n",
    "The polarity is within the range [-1.0, 1.0], and the subjectivity is within the range [0.0, 1.0], thus these both features don't require the scaling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the StandardScaler\n",
    "train_scaler = StandardScaler()\n",
    "# Scale the features\n",
    "train_dataset.iloc[:, 3:11] = train_scaler.fit_transform(train_dataset.iloc[:, 3:11])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clean_review</th>\n",
       "      <th>polarity</th>\n",
       "      <th>subjectivity</th>\n",
       "      <th>word_count</th>\n",
       "      <th>UPPERCASE</th>\n",
       "      <th>DIGITS</th>\n",
       "      <th>PROPN</th>\n",
       "      <th>VERB</th>\n",
       "      <th>NOUN</th>\n",
       "      <th>PUNCT</th>\n",
       "      <th>ADJ</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>amaze good wonderful film early ninety franchi...</td>\n",
       "      <td>0.2482</td>\n",
       "      <td>0.5293</td>\n",
       "      <td>0.925766</td>\n",
       "      <td>-0.466051</td>\n",
       "      <td>-0.029506</td>\n",
       "      <td>0.713005</td>\n",
       "      <td>-0.218791</td>\n",
       "      <td>0.986809</td>\n",
       "      <td>-0.090785</td>\n",
       "      <td>0.671761</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>wrong end see tell chick go crazy eat old woma...</td>\n",
       "      <td>0.1763</td>\n",
       "      <td>0.5750</td>\n",
       "      <td>-0.503381</td>\n",
       "      <td>-1.032507</td>\n",
       "      <td>3.306467</td>\n",
       "      <td>-1.496461</td>\n",
       "      <td>0.782012</td>\n",
       "      <td>-0.444239</td>\n",
       "      <td>-2.221886</td>\n",
       "      <td>-0.455286</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>guess emperor clothe see list pbs night hopefu...</td>\n",
       "      <td>0.1145</td>\n",
       "      <td>0.5557</td>\n",
       "      <td>-0.387190</td>\n",
       "      <td>2.457466</td>\n",
       "      <td>-0.500133</td>\n",
       "      <td>1.245828</td>\n",
       "      <td>1.262165</td>\n",
       "      <td>-2.657338</td>\n",
       "      <td>1.305950</td>\n",
       "      <td>-0.566812</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>earth well movie funny sweet good plot unique ...</td>\n",
       "      <td>0.3810</td>\n",
       "      <td>0.5720</td>\n",
       "      <td>-1.026239</td>\n",
       "      <td>-0.226890</td>\n",
       "      <td>-0.500133</td>\n",
       "      <td>-1.122251</td>\n",
       "      <td>-0.664234</td>\n",
       "      <td>-0.680484</td>\n",
       "      <td>0.435819</td>\n",
       "      <td>2.033656</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>doe eye high school student kathleen beller fi...</td>\n",
       "      <td>0.2095</td>\n",
       "      <td>0.4666</td>\n",
       "      <td>2.651199</td>\n",
       "      <td>-0.517426</td>\n",
       "      <td>-0.232807</td>\n",
       "      <td>1.702533</td>\n",
       "      <td>0.403095</td>\n",
       "      <td>-0.153684</td>\n",
       "      <td>0.865742</td>\n",
       "      <td>-1.192711</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        clean_review  polarity  subjectivity  \\\n",
       "0  amaze good wonderful film early ninety franchi...    0.2482        0.5293   \n",
       "1  wrong end see tell chick go crazy eat old woma...    0.1763        0.5750   \n",
       "2  guess emperor clothe see list pbs night hopefu...    0.1145        0.5557   \n",
       "3  earth well movie funny sweet good plot unique ...    0.3810        0.5720   \n",
       "4  doe eye high school student kathleen beller fi...    0.2095        0.4666   \n",
       "\n",
       "   word_count  UPPERCASE    DIGITS     PROPN      VERB      NOUN     PUNCT  \\\n",
       "0    0.925766  -0.466051 -0.029506  0.713005 -0.218791  0.986809 -0.090785   \n",
       "1   -0.503381  -1.032507  3.306467 -1.496461  0.782012 -0.444239 -2.221886   \n",
       "2   -0.387190   2.457466 -0.500133  1.245828  1.262165 -2.657338  1.305950   \n",
       "3   -1.026239  -0.226890 -0.500133 -1.122251 -0.664234 -0.680484  0.435819   \n",
       "4    2.651199  -0.517426 -0.232807  1.702533  0.403095 -0.153684  0.865742   \n",
       "\n",
       "        ADJ  label  \n",
       "0  0.671761      1  \n",
       "1 -0.455286      0  \n",
       "2 -0.566812      0  \n",
       "3  2.033656      1  \n",
       "4 -1.192711      1  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the dataset. Use clean_review and label columns\n",
    "val_dataset = pd.read_csv('dataset/datasets_feat_clean/val_feat_clean.csv',\n",
    "                          usecols=['clean_review', 'polarity', 'subjectivity', 'word_count', 'UPPERCASE', 'DIGITS',\n",
    "                                      'PROPN', 'VERB', 'NOUN', 'PUNCT', 'ADJ', 'label'])\n",
    "\n",
    "# Change columns order\n",
    "val_dataset = val_dataset[['clean_review', 'polarity', 'subjectivity', 'word_count', 'UPPERCASE', 'DIGITS',\n",
    "                           'PROPN', 'VERB', 'NOUN', 'PUNCT', 'ADJ', 'label']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clean_review</th>\n",
       "      <th>polarity</th>\n",
       "      <th>subjectivity</th>\n",
       "      <th>word_count</th>\n",
       "      <th>UPPERCASE</th>\n",
       "      <th>DIGITS</th>\n",
       "      <th>PROPN</th>\n",
       "      <th>VERB</th>\n",
       "      <th>NOUN</th>\n",
       "      <th>PUNCT</th>\n",
       "      <th>ADJ</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>go movie twice week sum word normally use ligh...</td>\n",
       "      <td>0.272000</td>\n",
       "      <td>0.4656</td>\n",
       "      <td>155</td>\n",
       "      <td>0.01290</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.05160</td>\n",
       "      <td>0.1807</td>\n",
       "      <td>0.1807</td>\n",
       "      <td>0.1613</td>\n",
       "      <td>0.10320</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>year big fan park work old boy time favorite.w...</td>\n",
       "      <td>-0.056670</td>\n",
       "      <td>0.3833</td>\n",
       "      <td>119</td>\n",
       "      <td>0.02520</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.10080</td>\n",
       "      <td>0.1681</td>\n",
       "      <td>0.1932</td>\n",
       "      <td>0.2185</td>\n",
       "      <td>0.08405</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>movie potential handle differently need differ...</td>\n",
       "      <td>-0.012695</td>\n",
       "      <td>0.6560</td>\n",
       "      <td>152</td>\n",
       "      <td>0.03946</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.01316</td>\n",
       "      <td>0.3157</td>\n",
       "      <td>0.1250</td>\n",
       "      <td>0.1382</td>\n",
       "      <td>0.05920</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>movie difficult review give away plot suffice ...</td>\n",
       "      <td>0.148400</td>\n",
       "      <td>0.5910</td>\n",
       "      <td>173</td>\n",
       "      <td>0.01156</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.08093</td>\n",
       "      <td>0.1849</td>\n",
       "      <td>0.2196</td>\n",
       "      <td>0.1272</td>\n",
       "      <td>0.10406</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>plot worth discussion hint corruption murder p...</td>\n",
       "      <td>0.215300</td>\n",
       "      <td>0.5790</td>\n",
       "      <td>105</td>\n",
       "      <td>0.01904</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.04760</td>\n",
       "      <td>0.1333</td>\n",
       "      <td>0.2856</td>\n",
       "      <td>0.1619</td>\n",
       "      <td>0.16190</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        clean_review  polarity  subjectivity  \\\n",
       "0  go movie twice week sum word normally use ligh...  0.272000        0.4656   \n",
       "1  year big fan park work old boy time favorite.w... -0.056670        0.3833   \n",
       "2  movie potential handle differently need differ... -0.012695        0.6560   \n",
       "3  movie difficult review give away plot suffice ...  0.148400        0.5910   \n",
       "4  plot worth discussion hint corruption murder p...  0.215300        0.5790   \n",
       "\n",
       "   word_count  UPPERCASE  DIGITS    PROPN    VERB    NOUN   PUNCT      ADJ  \\\n",
       "0         155    0.01290     0.0  0.05160  0.1807  0.1807  0.1613  0.10320   \n",
       "1         119    0.02520     0.0  0.10080  0.1681  0.1932  0.2185  0.08405   \n",
       "2         152    0.03946     0.0  0.01316  0.3157  0.1250  0.1382  0.05920   \n",
       "3         173    0.01156     0.0  0.08093  0.1849  0.2196  0.1272  0.10406   \n",
       "4         105    0.01904     0.0  0.04760  0.1333  0.2856  0.1619  0.16190   \n",
       "\n",
       "   label  \n",
       "0      1  \n",
       "1      0  \n",
       "2      0  \n",
       "3      1  \n",
       "4      0  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Depict the first 5 rows of the validation set\n",
    "val_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the StandardScaler\n",
    "val_scaler = StandardScaler()\n",
    "# Scale the features\n",
    "val_dataset.iloc[:, 3:11] = val_scaler.fit_transform(val_dataset.iloc[:, 3:11])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clean_review</th>\n",
       "      <th>polarity</th>\n",
       "      <th>subjectivity</th>\n",
       "      <th>word_count</th>\n",
       "      <th>UPPERCASE</th>\n",
       "      <th>DIGITS</th>\n",
       "      <th>PROPN</th>\n",
       "      <th>VERB</th>\n",
       "      <th>NOUN</th>\n",
       "      <th>PUNCT</th>\n",
       "      <th>ADJ</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>go movie twice week sum word normally use ligh...</td>\n",
       "      <td>0.272000</td>\n",
       "      <td>0.4656</td>\n",
       "      <td>-0.443496</td>\n",
       "      <td>-0.471182</td>\n",
       "      <td>-0.476247</td>\n",
       "      <td>-0.433351</td>\n",
       "      <td>-0.193510</td>\n",
       "      <td>-0.196521</td>\n",
       "      <td>0.377979</td>\n",
       "      <td>0.524956</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>year big fan park work old boy time favorite.w...</td>\n",
       "      <td>-0.056670</td>\n",
       "      <td>0.3833</td>\n",
       "      <td>-0.654306</td>\n",
       "      <td>0.091716</td>\n",
       "      <td>-0.476247</td>\n",
       "      <td>0.567055</td>\n",
       "      <td>-0.559484</td>\n",
       "      <td>0.142736</td>\n",
       "      <td>1.531346</td>\n",
       "      <td>-0.157714</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>movie potential handle differently need differ...</td>\n",
       "      <td>-0.012695</td>\n",
       "      <td>0.6560</td>\n",
       "      <td>-0.461063</td>\n",
       "      <td>0.744311</td>\n",
       "      <td>-0.476247</td>\n",
       "      <td>-1.214968</td>\n",
       "      <td>3.727638</td>\n",
       "      <td>-1.708251</td>\n",
       "      <td>-0.087803</td>\n",
       "      <td>-1.043581</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>movie difficult review give away plot suffice ...</td>\n",
       "      <td>0.148400</td>\n",
       "      <td>0.5910</td>\n",
       "      <td>-0.338090</td>\n",
       "      <td>-0.532506</td>\n",
       "      <td>-0.476247</td>\n",
       "      <td>0.163029</td>\n",
       "      <td>-0.071519</td>\n",
       "      <td>0.859248</td>\n",
       "      <td>-0.309605</td>\n",
       "      <td>0.555614</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>plot worth discussion hint corruption murder p...</td>\n",
       "      <td>0.215300</td>\n",
       "      <td>0.5790</td>\n",
       "      <td>-0.736288</td>\n",
       "      <td>-0.190191</td>\n",
       "      <td>-0.476247</td>\n",
       "      <td>-0.514684</td>\n",
       "      <td>-1.570269</td>\n",
       "      <td>2.650526</td>\n",
       "      <td>0.390078</td>\n",
       "      <td>2.617528</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        clean_review  polarity  subjectivity  \\\n",
       "0  go movie twice week sum word normally use ligh...  0.272000        0.4656   \n",
       "1  year big fan park work old boy time favorite.w... -0.056670        0.3833   \n",
       "2  movie potential handle differently need differ... -0.012695        0.6560   \n",
       "3  movie difficult review give away plot suffice ...  0.148400        0.5910   \n",
       "4  plot worth discussion hint corruption murder p...  0.215300        0.5790   \n",
       "\n",
       "   word_count  UPPERCASE    DIGITS     PROPN      VERB      NOUN     PUNCT  \\\n",
       "0   -0.443496  -0.471182 -0.476247 -0.433351 -0.193510 -0.196521  0.377979   \n",
       "1   -0.654306   0.091716 -0.476247  0.567055 -0.559484  0.142736  1.531346   \n",
       "2   -0.461063   0.744311 -0.476247 -1.214968  3.727638 -1.708251 -0.087803   \n",
       "3   -0.338090  -0.532506 -0.476247  0.163029 -0.071519  0.859248 -0.309605   \n",
       "4   -0.736288  -0.190191 -0.476247 -0.514684 -1.570269  2.650526  0.390078   \n",
       "\n",
       "        ADJ  label  \n",
       "0  0.524956      1  \n",
       "1 -0.157714      0  \n",
       "2 -1.043581      0  \n",
       "3  0.555614      1  \n",
       "4  2.617528      0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we will use the BatchIterator class defined in the *vocabulary* notebook to create the vocabulary, trim sequences in terms of the rare word occurrence and the length, map words to their numerical representation (word2index), furthermore BatchIterator sorts dataset examples, generates batches, performs sequence padding and enables to use it instance to iterate through all batches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use pre-trained embeddings instead of learning them during training process, thus we will create the weights matrix only for the training set.  \n",
    "\n",
    "To create the weights matrix we have to set use_pretrained_vectors=True and supply the weight file path, Glove vectors file name and the directory and the name of the file to which we will export the prepared weights matrix. To use different word embeddings, simply pass on their file to the BatchIterator class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trimmed vocabulary using as minimum count threashold: count = 3.00\n",
      "35371/130416 tokens has been retained\n",
      "Trimmed input strings vocabulary\n",
      "Trimmed input sequences lengths to the length of: 198\n",
      "Mapped words to indices\n",
      "Start creating glove_word2vector dictionary\n",
      "Extracted 30905/35375 of pre-trained word vectors.\n",
      "4470 vectors initialized to random numbers\n",
      "Weights vectors saved into glove/weights.npy\n",
      "Batches created\n"
     ]
    }
   ],
   "source": [
    "train_iterator = BatchIterator(train_dataset, batch_size=64, vocab_created=False, vocab=None, target_col=None,\n",
    "                               word2index=None, sos_token='<SOS>', eos_token='<EOS>', unk_token='<UNK>',\n",
    "                               pad_token='<PAD>', min_word_count=3, max_vocab_size=None, max_seq_len=0.9,\n",
    "                               use_pretrained_vectors=True, glove_path='glove/', glove_name='glove.6B.100d.txt',\n",
    "                               weights_file_name='glove/weights-biGRU-glove.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trimmed vocabulary using as minimum count threashold: count = 3.00\n",
      "19357/59089 tokens has been retained\n",
      "Trimmed input strings vocabulary\n",
      "Trimmed input sequences lengths to the length of: 195\n",
      "Mapped words to indices\n",
      "Batches created\n"
     ]
    }
   ],
   "source": [
    "val_iterator = BatchIterator(val_dataset, batch_size=64, vocab_created=False, vocab=None, target_col=None,\n",
    "                             word2index=train_iterator.word2index, sos_token='<SOS>', eos_token='<EOS>',\n",
    "                             unk_token='<UNK>', pad_token='<PAD>', min_word_count=3, max_vocab_size=None,\n",
    "                             max_seq_len=0.9, use_pretrained_vectors=False, glove_path='glove/',\n",
    "                             glove_name='glove.6B.100d.txt', weights_file_name='glove/weights.npy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to check out how batches that we created look like before we pass them into the model. For the record, the set of batches for input and output variables is returned as a dictionary, thus we will just look at the dictionary keys to find out how to extract particular variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['input_feat', 'input_seq', 'target', 'x_lengths'])\n"
     ]
    }
   ],
   "source": [
    "for batches in train_iterator:\n",
    "    print(batches.keys())\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see above we can distinguish the following batches: *input_feat* that comprises all additional features, *input_seq* that contains cleaned reviews, *target batch* that includes target labels and *x_lengths* batch that encompasses input sequences lengths.\n",
    "\n",
    "Notice that the *input_seq* batch has the dimensions: (batch_size, seq_len), while *input_feat* batch has the shape of (batch_size, num_features)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_seq shape:  torch.Size([64, 90])\n",
      "input_feat shape:  torch.Size([64, 10])\n",
      "target shape:  torch.Size([64])\n",
      "x_lengths shape:  torch.Size([64])\n"
     ]
    }
   ],
   "source": [
    "for batches in train_iterator:\n",
    "\n",
    "    # Unpack the dictionary of batches\n",
    "    input_seq, input_feat, target, x_lengths = batches['input_seq'], batches['input_feat'],\\\n",
    "                                               batches['target'], batches['x_lengths']\n",
    "        \n",
    "    print('input_seq shape: ', input_seq.size())\n",
    "    print('input_feat shape: ', input_feat.size())\n",
    "    print('target shape: ', target.size())\n",
    "    print('x_lengths shape: ', x_lengths.size())\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_seq shape:  torch.Size([64, 115])\n",
      "input_feat shape:  torch.Size([64, 10])\n",
      "target shape:  torch.Size([64])\n",
      "x_lengths shape:  torch.Size([64])\n"
     ]
    }
   ],
   "source": [
    "for batches in val_iterator:\n",
    "    # Unpack the dictionary of batches\n",
    "    input_seq, input_feat, target, x_lengths = batches['input_seq'], batches['input_feat'],\\\n",
    "                                               batches['target'], batches['x_lengths']\n",
    "    print('input_seq shape: ', input_seq.size())\n",
    "    print('input_feat shape: ', input_feat.size())\n",
    "    print('target shape: ', target.size())\n",
    "    print('x_lengths shape: ', x_lengths.size())\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to build a biGRU model that will use the pre-trained word embeddings and *input_feat* batch as additional features vector. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BiGRU(nn.Module):\n",
    "    \"\"\"BiDirectional GRU neural network model with pre-trained word embeddings.\n",
    "        \n",
    "    Parameters\n",
    "    ----------\n",
    "    hidden_size: int\n",
    "        Number of features in the hidden state.\n",
    "    vocab_size: int\n",
    "        The size of the vocabulary.\n",
    "    n_extra_feat: int\n",
    "        Number of additional features.\n",
    "    weight_matrix: numpy.ndarray\n",
    "        Matrix of pre-trained word embeddings.\n",
    "    output_size: int\n",
    "        Number of classes.\n",
    "    n_layers: int, optional (default=1)\n",
    "        Number of stacked recurrent layers.\n",
    "    dropout: float, optional (default=0.2)\n",
    "        Probability of an element of the tensor to be zeroed.\n",
    "    spatial_dropout: boolean, optional (default=True)\n",
    "        Whether to use the spatial dropout.\n",
    "    bidirectional: boolean, optional (default=True)\n",
    "        Whether to use the bidirectional GRU.\n",
    "\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, hidden_size, vocab_size, n_extra_feat, weights_matrix, output_size, n_layers=1, dropout=0.2,\n",
    "                 spatial_dropout=True, bidirectional=True):\n",
    "        \n",
    "        # Inherit everything from the nn.Module\n",
    "        super(BiGRU, self).__init__()\n",
    "        \n",
    "        # Initialize attributes\n",
    "        self.hidden_size = hidden_size\n",
    "        self.vocab_size = vocab_size\n",
    "        self.n_extra_feat = n_extra_feat\n",
    "        self.weights_matrix = weights_matrix\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.dropout_p = dropout\n",
    "        self.spatial_dropout = spatial_dropout\n",
    "        self.bidirectional = bidirectional\n",
    "        self.n_directions = 2 if self.bidirectional else 1\n",
    "        \n",
    "        self.vocab_size, self.embedding_dim = self.weights_matrix.shape\n",
    "        \n",
    "        # Initialize layers\n",
    "        self.embedding = nn.Embedding(self.vocab_size, self.embedding_dim)\n",
    "        \n",
    "        # Load the weights to the embedding layer\n",
    "        self.embedding.load_state_dict({'weight': torch.from_numpy(weights_matrix)})\n",
    "        self.embedding.weight.requires_grad = False\n",
    "        \n",
    "        self.dropout = nn.Dropout(self.dropout_p)\n",
    "        if self.spatial_dropout:\n",
    "            self.spatial_dropout1d = nn.Dropout2d(self.dropout_p)\n",
    "        self.gru = nn.GRU(self.embedding_dim, self.hidden_size, num_layers=self.n_layers, \n",
    "                          dropout=(0 if n_layers == 1 else self.dropout_p), batch_first=True,\n",
    "                          bidirectional=self.bidirectional)\n",
    "        # Linear layer input size is equal to hidden_size * 3 + n_extra_feat, becuase\n",
    "        # we will concatenate max_pooling ,avg_pooling, last hidden state and additional features\n",
    "        self.linear = nn.Linear(self.hidden_size * 3 + self.n_extra_feat, self.output_size)\n",
    "\n",
    "        \n",
    "    def forward(self, input_seq, input_feat, input_lengths, hidden=None):\n",
    "        \"\"\"Forward propagate through the neural network model.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        input_seq: torch.Tensor\n",
    "            Batch of input sequences.\n",
    "        input_feat: torch.Tensor\n",
    "            Batch of additional features.\n",
    "        input_lengths: torch.LongTensor\n",
    "            Batch containing sequences lengths.\n",
    "        hidden: torch.FloatTensor, optional (default=None)\n",
    "            Tensor containing initial hidden state.\n",
    "            \n",
    "        Returns\n",
    "        -------\n",
    "        torch.Tensor\n",
    "            Logarithm of softmaxed input tensor.\n",
    "        \n",
    "        \"\"\"\n",
    "        # Extract batch_size\n",
    "        self.batch_size = input_seq.size(0)\n",
    "        \n",
    "        # Embeddings shapes\n",
    "        # Input: (batch_size,  seq_length)\n",
    "        # Output: (batch_size, seq_length, embedding_dim)\n",
    "        emb_out = self.embedding(input_seq)\n",
    "        \n",
    "        if self.spatial_dropout:\n",
    "            # Convert to (batch_size, embedding_dim, seq_length)\n",
    "            emb_out = emb_out.permute(0, 2, 1)\n",
    "            emb_out = self.spatial_dropout1d(emb_out)\n",
    "            # Convert back to (batch_size, seq_length, embedding_dim)\n",
    "            emb_out = emb_out.permute(0, 2, 1)\n",
    "        else:\n",
    "            emb_out = self.dropout(emb_out)\n",
    "        \n",
    "        # Pack padded batch of sequences for RNN module\n",
    "        packed_emb = nn.utils.rnn.pack_padded_sequence(emb_out, input_lengths, batch_first=True)\n",
    "                \n",
    "        # GRU input/output shapes, if batch_first=True\n",
    "        # Input: (batch_size, seq_len, embedding_dim)\n",
    "        # Output: (batch_size, seq_len, hidden_size*num_directions)\n",
    "        # Number of directions = 2 when used bidirectional, otherwise 1\n",
    "        # shape of hidden: (n_layers x num_directions, batch_size, hidden_size)\n",
    "        # Hidden state defaults to zero if not provided\n",
    "        gru_out, hidden = self.gru(packed_emb, hidden)\n",
    "        # gru_out: tensor containing the output features h_t from the last layer of the GRU\n",
    "        # gru_out comprises all the hidden states in the last layer (\"last\" depth-wise, not time-wise)\n",
    "        # For biGRu gru_out is the concatenation of a forward GRU representation and a backward GRU representation\n",
    "        # hidden (h_n) comprises the hidden states after the last timestep\n",
    "        \n",
    "        # Extract and sum last hidden state\n",
    "        # Input hidden shape: (n_layers x num_directions, batch_size, hidden_size)\n",
    "        # Separate hidden state layers\n",
    "        hidden = hidden.view(self.n_layers, self.n_directions, self.batch_size, self.hidden_size)\n",
    "        last_hidden = hidden[-1]\n",
    "        # last hidden shape (num_directions, batch_size, hidden_size)\n",
    "        # Sum the last hidden state of forward and backward layer\n",
    "        last_hidden = torch.sum(last_hidden, dim=0)\n",
    "        # Summed last hidden shape (batch_size, hidden_size)\n",
    "        \n",
    "        # Pad a packed batch\n",
    "        # gru_out output shape: (batch_size, seq_len, hidden_size*num_directions)\n",
    "        gru_out, lengths = nn.utils.rnn.pad_packed_sequence(gru_out, batch_first=True)\n",
    "              \n",
    "        # Sum the gru_out along the num_directions\n",
    "        if self.bidirectional:\n",
    "            gru_out = gru_out[:,:,:self.hidden_size] + gru_out[:,:,self.hidden_size:]\n",
    "        \n",
    "        # Select the maximum value over each dimension of the hidden representation (max pooling)\n",
    "        # Permute the input tensor to dimensions: (batch_size, hidden, seq_len)\n",
    "        # Output dimensions: (batch_size, hidden_size)\n",
    "        max_pool = F.adaptive_max_pool1d(gru_out.permute(0,2,1), (1,)).view(self.batch_size,-1)\n",
    "        \n",
    "        # Consider the average of the representations (mean pooling)\n",
    "        # Sum along the batch axis and divide by the corresponding lengths (FloatTensor)\n",
    "        # Output shape: (batch_size, hidden_size)\n",
    "        avg_pool = torch.sum(gru_out, dim=1) / lengths.view(-1,1).type(torch.FloatTensor) \n",
    "\n",
    "        # Concatenate max_pooling, avg_pooling, hidden state and input_feat tensor\n",
    "        concat_out = torch.cat([last_hidden, max_pool, avg_pool, input_feat], dim=1)\n",
    "\n",
    "        # concat_out = self.dropout(concat_out)\n",
    "        out = self.linear(concat_out)\n",
    "        return F.log_softmax(out, dim=-1)\n",
    "    \n",
    "    \n",
    "    def add_loss_fn(self, loss_fn):\n",
    "        \"\"\"Add loss function to the model.\n",
    "        \n",
    "        \"\"\"\n",
    "        self.loss_fn = loss_fn\n",
    "        \n",
    "\n",
    "    def add_optimizer(self, optimizer):\n",
    "        \"\"\"Add optimizer to the model.\n",
    "        \n",
    "        \"\"\"\n",
    "        self.optimizer = optimizer\n",
    "        \n",
    "        \n",
    "    def add_device(self, device=torch.device('cpu')):\n",
    "        \"\"\"Specify the device.\n",
    "        \n",
    "        \"\"\"\n",
    "        self.device = device\n",
    "    \n",
    "    \n",
    "    def train_model(self, train_iterator):\n",
    "        \"\"\"Perform single training epoch.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        train_iterator: BatchIterator\n",
    "            BatchIterator class object containing training batches.\n",
    "            \n",
    "        Returns\n",
    "        -------\n",
    "        train_losses: list\n",
    "            List of the training average batch losses.\n",
    "        avg_loss: float\n",
    "            Average loss on the entire training set.\n",
    "        accuracy: float\n",
    "            Models accuracy on the entire training set.\n",
    "            \n",
    "        \"\"\"\n",
    "        self.train()\n",
    "        \n",
    "        train_losses = []\n",
    "        losses = []\n",
    "        losses_list = []\n",
    "        num_seq = 0\n",
    "        batch_correct = 0\n",
    "            \n",
    "        for i, batches in tqdm_notebook(enumerate(train_iterator, 1), total=len(train_iterator), desc='Training'):\n",
    "            input_seq, input_feat, target, x_lengths = batches['input_seq'], batches['input_feat'],\\\n",
    "                                                       batches['target'], batches['x_lengths']\n",
    "            \n",
    "            input_seq.to(self.device)\n",
    "            input_feat.to(self.device)\n",
    "            target.to(self.device)\n",
    "            x_lengths.to(self.device)\n",
    "            \n",
    "            self.optimizer.zero_grad()\n",
    "\n",
    "            pred = self.forward(input_seq, input_feat, x_lengths)\n",
    "            loss = self.loss_fn(pred, target)\n",
    "            loss.backward()\n",
    "            losses.append(loss.data.cpu().numpy())\n",
    "            self.optimizer.step()\n",
    "            \n",
    "            losses_list.append(loss.data.cpu().numpy())\n",
    "            \n",
    "            pred = torch.argmax(pred, 1)\n",
    "\n",
    "            if self.device.type == 'cpu':\n",
    "                batch_correct += (pred.cpu() == target.cpu()).sum().item()\n",
    "\n",
    "            else:\n",
    "                batch_correct += (pred == target).sum().item()\n",
    "\n",
    "            num_seq += len(input_seq)     \n",
    "    \n",
    "            if i % 100 == 0:\n",
    "                avg_train_loss = np.mean(losses)\n",
    "                train_losses.append(avg_train_loss)\n",
    "                \n",
    "                accuracy = batch_correct / num_seq\n",
    "                \n",
    "                print('Iteration: {}. Average training loss: {:.4f}. Accuracy: {:.3f}'\\\n",
    "                      .format(i, avg_train_loss, accuracy))\n",
    "                \n",
    "                losses = []\n",
    "                \n",
    "            avg_loss = np.mean(losses_list)\n",
    "            accuracy = batch_correct / num_seq\n",
    "                              \n",
    "        return train_losses, avg_loss, accuracy\n",
    "    \n",
    "    \n",
    "    def evaluate_model(self, eval_iterator, conf_mtx=False):\n",
    "        \"\"\"Perform the one evaluation epoch.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        eval_iterator: BatchIterator\n",
    "            BatchIterator class object containing evaluation batches.\n",
    "        conf_mtx: boolean, optional (default=False)\n",
    "            Whether to print the confusion matrix at each epoch.\n",
    "            \n",
    "        Returns\n",
    "        -------\n",
    "        eval_losses: list\n",
    "            List of the evaluation average batch losses.\n",
    "        avg_loss: float\n",
    "            Average loss on the entire evaluation set.\n",
    "        accuracy: float\n",
    "            Models accuracy on the entire evaluation set.\n",
    "        conf_matrix: list\n",
    "            Confusion matrix.\n",
    "            \n",
    "        \"\"\"\n",
    "        self.eval()\n",
    "        \n",
    "        eval_losses = []\n",
    "        losses = []\n",
    "        losses_list = []\n",
    "        num_seq = 0\n",
    "        batch_correct = 0\n",
    "        pred_total = torch.LongTensor()\n",
    "        target_total = torch.LongTensor()\n",
    "        \n",
    "        with torch.no_grad():\n",
    "            for i, batches in tqdm_notebook(enumerate(eval_iterator, 1), total=len(eval_iterator), desc='Evaluation'):\n",
    "                input_seq, input_feat, target, x_lengths = batches['input_seq'], batches['input_feat'],\\\n",
    "                                                           batches['target'], batches['x_lengths']\n",
    "                \n",
    "                input_seq.to(self.device)\n",
    "                input_feat.to(self.device)\n",
    "                target.to(self.device)\n",
    "                x_lengths.to(self.device)\n",
    "\n",
    "                pred = self.forward(input_seq, input_feat, x_lengths)\n",
    "                loss = self.loss_fn(pred, target)\n",
    "                losses.append(loss.data.cpu().numpy())\n",
    "                losses_list.append(loss.data.cpu().numpy())\n",
    "                \n",
    "                pred = torch.argmax(pred, 1)\n",
    "                                \n",
    "                if self.device.type == 'cpu':\n",
    "                    batch_correct += (pred.cpu() == target.cpu()).sum().item()\n",
    "                    \n",
    "                else:\n",
    "                    batch_correct += (pred == target).sum().item()\n",
    "                    \n",
    "                num_seq += len(input_seq)     \n",
    "                \n",
    "                pred_total = torch.cat([pred_total, pred], dim=0)\n",
    "                target_total = torch.cat([target_total, target], dim=0)\n",
    "                \n",
    "                if i % 100 == 0:\n",
    "                    avg_batch_eval_loss = np.mean(losses)\n",
    "                    eval_losses.append(avg_batch_eval_loss)\n",
    "                    \n",
    "                    accuracy = batch_correct / num_seq\n",
    "                    \n",
    "                    print('Iteration: {}. Average evaluation loss: {:.4f}. Accuracy: {:.2f}'\\\n",
    "                          .format(i, avg_batch_eval_loss, accuracy))\n",
    "\n",
    "                    losses = []\n",
    "                    \n",
    "            avg_loss_list = []\n",
    "                    \n",
    "            avg_loss = np.mean(losses_list)\n",
    "            accuracy = batch_correct / num_seq\n",
    "            \n",
    "            conf_matrix = confusion_matrix(target_total.view(-1), pred_total.view(-1))\n",
    "        \n",
    "        if conf_mtx:\n",
    "            print('\\tConfusion matrix: ', conf_matrix)\n",
    "            \n",
    "        return eval_losses, avg_loss, accuracy, conf_matrix\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will instantiate the model, add loss function, optimizer, and device to it and begin the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Start epoch [1/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "633d577f88f049a28a68aadcefb89877",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.7065. Accuracy: 0.503\n",
      "Iteration: 200. Average training loss: 0.6630. Accuracy: 0.554\n",
      "Iteration: 300. Average training loss: 0.6311. Accuracy: 0.587\n",
      "Iteration: 400. Average training loss: 0.5909. Accuracy: 0.614\n",
      "Iteration: 500. Average training loss: 0.5668. Accuracy: 0.634\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d1f05e6fb2f4fd99208c6a518c362fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.4823. Accuracy: 0.77\n",
      "\n",
      "Epoch [1/40]: Train accuracy: 0.642. Train loss: 0.6239. Evaluation accuracy: 0.776. Evaluation loss: 0.4794\n",
      "\n",
      "Start epoch [2/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eb160299c6fa4faf8ababb1a32312f07",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.5319. Accuracy: 0.740\n",
      "Iteration: 200. Average training loss: 0.5005. Accuracy: 0.749\n",
      "Iteration: 300. Average training loss: 0.4981. Accuracy: 0.754\n",
      "Iteration: 400. Average training loss: 0.4877. Accuracy: 0.757\n",
      "Iteration: 500. Average training loss: 0.4828. Accuracy: 0.760\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "06880f47ba104e6db2046aa3626bca1a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.4308. Accuracy: 0.80\n",
      "\n",
      "Epoch [2/40]: Train accuracy: 0.762. Train loss: 0.4971. Evaluation accuracy: 0.805. Evaluation loss: 0.4251\n",
      "\n",
      "Start epoch [3/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b6882981de74ec586e5af260fa256f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4852. Accuracy: 0.772\n",
      "Iteration: 200. Average training loss: 0.4696. Accuracy: 0.775\n",
      "Iteration: 300. Average training loss: 0.4612. Accuracy: 0.777\n",
      "Iteration: 400. Average training loss: 0.4520. Accuracy: 0.780\n",
      "Iteration: 500. Average training loss: 0.4558. Accuracy: 0.781\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ee6bbd4882b47f08295d7a8b64123c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.4073. Accuracy: 0.81\n",
      "\n",
      "Epoch [3/40]: Train accuracy: 0.783. Train loss: 0.4630. Evaluation accuracy: 0.818. Evaluation loss: 0.4017\n",
      "\n",
      "Start epoch [4/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8740d0e769aa4ed3aac254df379b389e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4565. Accuracy: 0.784\n",
      "Iteration: 200. Average training loss: 0.4406. Accuracy: 0.789\n",
      "Iteration: 300. Average training loss: 0.4465. Accuracy: 0.791\n",
      "Iteration: 400. Average training loss: 0.4349. Accuracy: 0.793\n",
      "Iteration: 500. Average training loss: 0.4416. Accuracy: 0.793\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72707d31552b46f2867a848392dbdae3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3928. Accuracy: 0.82\n",
      "\n",
      "Epoch [4/40]: Train accuracy: 0.793. Train loss: 0.4431. Evaluation accuracy: 0.825. Evaluation loss: 0.3877\n",
      "\n",
      "Start epoch [5/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5d2bbc0ca9e741a0bf1fa443dd7911f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4432. Accuracy: 0.796\n",
      "Iteration: 200. Average training loss: 0.4313. Accuracy: 0.797\n",
      "Iteration: 300. Average training loss: 0.4274. Accuracy: 0.800\n",
      "Iteration: 400. Average training loss: 0.4211. Accuracy: 0.802\n",
      "Iteration: 500. Average training loss: 0.4304. Accuracy: 0.803\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a7993277062463da0a3e7e3492a0842",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3801. Accuracy: 0.83\n",
      "\n",
      "Epoch [5/40]: Train accuracy: 0.803. Train loss: 0.4299. Evaluation accuracy: 0.832. Evaluation loss: 0.3759\n",
      "\n",
      "Start epoch [6/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eeb6aad2e4e1458d9e59e6be93e5f5ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4319. Accuracy: 0.802\n",
      "Iteration: 200. Average training loss: 0.4278. Accuracy: 0.800\n",
      "Iteration: 300. Average training loss: 0.4114. Accuracy: 0.806\n",
      "Iteration: 400. Average training loss: 0.4125. Accuracy: 0.809\n",
      "Iteration: 500. Average training loss: 0.4199. Accuracy: 0.809\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bb5d65e57ea04a81a9d53c58bec8cbb9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3728. Accuracy: 0.83\n",
      "\n",
      "Epoch [6/40]: Train accuracy: 0.809. Train loss: 0.4197. Evaluation accuracy: 0.836. Evaluation loss: 0.3692\n",
      "\n",
      "Start epoch [7/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "028cb7727dba4a98bd8104d450539a80",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4223. Accuracy: 0.805\n",
      "Iteration: 200. Average training loss: 0.4150. Accuracy: 0.807\n",
      "Iteration: 300. Average training loss: 0.4098. Accuracy: 0.808\n",
      "Iteration: 400. Average training loss: 0.4074. Accuracy: 0.810\n",
      "Iteration: 500. Average training loss: 0.4039. Accuracy: 0.812\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "098a166c47814bda8573d78e739372c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3644. Accuracy: 0.84\n",
      "\n",
      "Epoch [7/40]: Train accuracy: 0.813. Train loss: 0.4108. Evaluation accuracy: 0.842. Evaluation loss: 0.3615\n",
      "\n",
      "Start epoch [8/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "27738ea175b24ae582cb93bbace39081",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4160. Accuracy: 0.811\n",
      "Iteration: 200. Average training loss: 0.4077. Accuracy: 0.813\n",
      "Iteration: 300. Average training loss: 0.4018. Accuracy: 0.814\n",
      "Iteration: 400. Average training loss: 0.3995. Accuracy: 0.816\n",
      "Iteration: 500. Average training loss: 0.3942. Accuracy: 0.819\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7051a73ee9284f039035db9a4f3ec0bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3586. Accuracy: 0.84\n",
      "\n",
      "Epoch [8/40]: Train accuracy: 0.819. Train loss: 0.4024. Evaluation accuracy: 0.844. Evaluation loss: 0.3557\n",
      "\n",
      "Start epoch [9/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e96eb6c3e4b149e694306788bf531319",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4096. Accuracy: 0.818\n",
      "Iteration: 200. Average training loss: 0.4042. Accuracy: 0.815\n",
      "Iteration: 300. Average training loss: 0.3950. Accuracy: 0.817\n",
      "Iteration: 400. Average training loss: 0.3927. Accuracy: 0.819\n",
      "Iteration: 500. Average training loss: 0.3969. Accuracy: 0.820\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4e0dbd04a4d4d2899472424d16d1b03",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3534. Accuracy: 0.85\n",
      "\n",
      "Epoch [9/40]: Train accuracy: 0.820. Train loss: 0.3993. Evaluation accuracy: 0.847. Evaluation loss: 0.3514\n",
      "\n",
      "Start epoch [10/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba3674a6c4754f77b35730f8554fde20",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4018. Accuracy: 0.819\n",
      "Iteration: 200. Average training loss: 0.3959. Accuracy: 0.818\n",
      "Iteration: 300. Average training loss: 0.3953. Accuracy: 0.819\n",
      "Iteration: 400. Average training loss: 0.3917. Accuracy: 0.820\n",
      "Iteration: 500. Average training loss: 0.3968. Accuracy: 0.821\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7de8723e87d84d68821a8803bfa543c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3501. Accuracy: 0.85\n",
      "\n",
      "Epoch [10/40]: Train accuracy: 0.821. Train loss: 0.3957. Evaluation accuracy: 0.849. Evaluation loss: 0.3474\n",
      "\n",
      "Start epoch [11/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "308380495bb14cc18ecc577263be61bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3988. Accuracy: 0.817\n",
      "Iteration: 200. Average training loss: 0.3924. Accuracy: 0.819\n",
      "Iteration: 300. Average training loss: 0.3912. Accuracy: 0.822\n",
      "Iteration: 400. Average training loss: 0.3876. Accuracy: 0.824\n",
      "Iteration: 500. Average training loss: 0.3893. Accuracy: 0.825\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2ec19ad3ae0495aba9893e98fee5519",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3470. Accuracy: 0.85\n",
      "\n",
      "Epoch [11/40]: Train accuracy: 0.826. Train loss: 0.3906. Evaluation accuracy: 0.851. Evaluation loss: 0.3446\n",
      "\n",
      "Start epoch [12/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f98e2bc34d064f91bbf8d67e41907b6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4014. Accuracy: 0.817\n",
      "Iteration: 200. Average training loss: 0.3924. Accuracy: 0.820\n",
      "Iteration: 300. Average training loss: 0.3898. Accuracy: 0.824\n",
      "Iteration: 400. Average training loss: 0.3801. Accuracy: 0.825\n",
      "Iteration: 500. Average training loss: 0.3818. Accuracy: 0.826\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c32c90e9b764834bc6a3326f018bcf3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3436. Accuracy: 0.85\n",
      "\n",
      "Epoch [12/40]: Train accuracy: 0.827. Train loss: 0.3880. Evaluation accuracy: 0.853. Evaluation loss: 0.3411\n",
      "\n",
      "Start epoch [13/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc95e86458914c25895160d74a35785d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3948. Accuracy: 0.820\n",
      "Iteration: 200. Average training loss: 0.3868. Accuracy: 0.824\n",
      "Iteration: 300. Average training loss: 0.3805. Accuracy: 0.826\n",
      "Iteration: 400. Average training loss: 0.3804. Accuracy: 0.828\n",
      "Iteration: 500. Average training loss: 0.3853. Accuracy: 0.829\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39cb5dd5d55d4d289065a584ce520cb7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3434. Accuracy: 0.85\n",
      "\n",
      "Epoch [13/40]: Train accuracy: 0.829. Train loss: 0.3843. Evaluation accuracy: 0.852. Evaluation loss: 0.3404\n",
      "\n",
      "Start epoch [14/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8100d13c0da14098a80a08f60eab140c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3953. Accuracy: 0.823\n",
      "Iteration: 200. Average training loss: 0.3850. Accuracy: 0.824\n",
      "Iteration: 300. Average training loss: 0.3831. Accuracy: 0.827\n",
      "Iteration: 400. Average training loss: 0.3756. Accuracy: 0.829\n",
      "Iteration: 500. Average training loss: 0.3755. Accuracy: 0.830\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b6ecb21792c4edc9ef1dbbc5fafc660",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3427. Accuracy: 0.85\n",
      "\n",
      "Epoch [14/40]: Train accuracy: 0.831. Train loss: 0.3818. Evaluation accuracy: 0.852. Evaluation loss: 0.3395\n",
      "\n",
      "Start epoch [15/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6adca8bec1c149fab9908a92456bdc5a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3859. Accuracy: 0.820\n",
      "Iteration: 200. Average training loss: 0.3860. Accuracy: 0.824\n",
      "Iteration: 300. Average training loss: 0.3833. Accuracy: 0.826\n",
      "Iteration: 400. Average training loss: 0.3791. Accuracy: 0.828\n",
      "Iteration: 500. Average training loss: 0.3767. Accuracy: 0.829\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "393ab89e48974b97971a431cc6b41cf4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3402. Accuracy: 0.85\n",
      "\n",
      "Epoch [15/40]: Train accuracy: 0.829. Train loss: 0.3823. Evaluation accuracy: 0.853. Evaluation loss: 0.3373\n",
      "\n",
      "Start epoch [16/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd461daf2309475cb7f2489d2da571ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3862. Accuracy: 0.826\n",
      "Iteration: 200. Average training loss: 0.3804. Accuracy: 0.825\n",
      "Iteration: 300. Average training loss: 0.3798. Accuracy: 0.828\n",
      "Iteration: 400. Average training loss: 0.3697. Accuracy: 0.829\n",
      "Iteration: 500. Average training loss: 0.3744. Accuracy: 0.830\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "283c83cee61646128b5d29705ae26ab0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3360. Accuracy: 0.85\n",
      "\n",
      "Epoch [16/40]: Train accuracy: 0.831. Train loss: 0.3772. Evaluation accuracy: 0.855. Evaluation loss: 0.3333\n",
      "\n",
      "Start epoch [17/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e3e0072f7f646d19d48667e4ecb38ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3858. Accuracy: 0.823\n",
      "Iteration: 200. Average training loss: 0.3799. Accuracy: 0.825\n",
      "Iteration: 300. Average training loss: 0.3708. Accuracy: 0.829\n",
      "Iteration: 400. Average training loss: 0.3716. Accuracy: 0.831\n",
      "Iteration: 500. Average training loss: 0.3767. Accuracy: 0.832\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f27f548a4c74ce8b844469680c833fc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3336. Accuracy: 0.85\n",
      "\n",
      "Epoch [17/40]: Train accuracy: 0.832. Train loss: 0.3759. Evaluation accuracy: 0.855. Evaluation loss: 0.3311\n",
      "\n",
      "Start epoch [18/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39396ef45ed24facb041de5a2d3367c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3823. Accuracy: 0.831\n",
      "Iteration: 200. Average training loss: 0.3767. Accuracy: 0.830\n",
      "Iteration: 300. Average training loss: 0.3733. Accuracy: 0.831\n",
      "Iteration: 400. Average training loss: 0.3716. Accuracy: 0.832\n",
      "Iteration: 500. Average training loss: 0.3778. Accuracy: 0.834\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e5491a983be042148d939063dc6611d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3335. Accuracy: 0.86\n",
      "\n",
      "Epoch [18/40]: Train accuracy: 0.834. Train loss: 0.3754. Evaluation accuracy: 0.856. Evaluation loss: 0.3309\n",
      "\n",
      "Start epoch [19/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4596d9d8b1ce49709b69ce02b6b45637",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3814. Accuracy: 0.829\n",
      "Iteration: 200. Average training loss: 0.3791. Accuracy: 0.831\n",
      "Iteration: 300. Average training loss: 0.3687. Accuracy: 0.833\n",
      "Iteration: 400. Average training loss: 0.3659. Accuracy: 0.834\n",
      "Iteration: 500. Average training loss: 0.3685. Accuracy: 0.836\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09a84762bb664291a1286585ce2778dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3334. Accuracy: 0.86\n",
      "\n",
      "Epoch [19/40]: Train accuracy: 0.837. Train loss: 0.3717. Evaluation accuracy: 0.857. Evaluation loss: 0.3306\n",
      "\n",
      "Start epoch [20/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "920e306d6dea4e8693141015e2ec0941",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3731. Accuracy: 0.838\n",
      "Iteration: 200. Average training loss: 0.3818. Accuracy: 0.833\n",
      "Iteration: 300. Average training loss: 0.3660. Accuracy: 0.836\n",
      "Iteration: 400. Average training loss: 0.3675. Accuracy: 0.836\n",
      "Iteration: 500. Average training loss: 0.3734. Accuracy: 0.836\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d295dfbf48447efb4b5b191a7035ac5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3305. Accuracy: 0.86\n",
      "\n",
      "Epoch [20/40]: Train accuracy: 0.837. Train loss: 0.3718. Evaluation accuracy: 0.858. Evaluation loss: 0.3286\n",
      "\n",
      "Start epoch [21/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a66f24a6ffca4690a7e9f88b73af93a7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3804. Accuracy: 0.830\n",
      "Iteration: 200. Average training loss: 0.3730. Accuracy: 0.833\n",
      "Iteration: 300. Average training loss: 0.3678. Accuracy: 0.833\n",
      "Iteration: 400. Average training loss: 0.3642. Accuracy: 0.835\n",
      "Iteration: 500. Average training loss: 0.3727. Accuracy: 0.836\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77e17af1f3a146699302a43e01168b61",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3291. Accuracy: 0.86\n",
      "\n",
      "Epoch [21/40]: Train accuracy: 0.836. Train loss: 0.3710. Evaluation accuracy: 0.859. Evaluation loss: 0.3272\n",
      "\n",
      "Start epoch [22/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9f2d056fbb647e0acaaca9b0bde4b06",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3803. Accuracy: 0.830\n",
      "Iteration: 200. Average training loss: 0.3755. Accuracy: 0.830\n",
      "Iteration: 300. Average training loss: 0.3678. Accuracy: 0.833\n",
      "Iteration: 400. Average training loss: 0.3584. Accuracy: 0.835\n",
      "Iteration: 500. Average training loss: 0.3657. Accuracy: 0.836\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fe7709e69a2b4a4ea41b7149a308b495",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3279. Accuracy: 0.86\n",
      "\n",
      "Epoch [22/40]: Train accuracy: 0.837. Train loss: 0.3687. Evaluation accuracy: 0.860. Evaluation loss: 0.3260\n",
      "\n",
      "Start epoch [23/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "adf09e6ffbd14aa2ae0b6dc44840ff12",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3818. Accuracy: 0.828\n",
      "Iteration: 200. Average training loss: 0.3722. Accuracy: 0.832\n",
      "Iteration: 300. Average training loss: 0.3636. Accuracy: 0.836\n",
      "Iteration: 400. Average training loss: 0.3599. Accuracy: 0.836\n",
      "Iteration: 500. Average training loss: 0.3644. Accuracy: 0.838\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6af0a2b065de4394b755ee9c444e4c3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3250. Accuracy: 0.86\n",
      "\n",
      "Epoch [23/40]: Train accuracy: 0.838. Train loss: 0.3676. Evaluation accuracy: 0.861. Evaluation loss: 0.3234\n",
      "\n",
      "Start epoch [24/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a1e66f8aab684329bb2dc40d88a52301",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3804. Accuracy: 0.827\n",
      "Iteration: 200. Average training loss: 0.3648. Accuracy: 0.835\n",
      "Iteration: 300. Average training loss: 0.3690. Accuracy: 0.836\n",
      "Iteration: 400. Average training loss: 0.3570. Accuracy: 0.838\n",
      "Iteration: 500. Average training loss: 0.3679. Accuracy: 0.838\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6623844710b544f6a06bcc88e67499c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3246. Accuracy: 0.86\n",
      "\n",
      "Epoch [24/40]: Train accuracy: 0.839. Train loss: 0.3668. Evaluation accuracy: 0.861. Evaluation loss: 0.3224\n",
      "\n",
      "Start epoch [25/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bc17ad969a04aad8293e590fcab699c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3807. Accuracy: 0.829\n",
      "Iteration: 200. Average training loss: 0.3642. Accuracy: 0.834\n",
      "Iteration: 300. Average training loss: 0.3642. Accuracy: 0.836\n",
      "Iteration: 400. Average training loss: 0.3617. Accuracy: 0.837\n",
      "Iteration: 500. Average training loss: 0.3615. Accuracy: 0.837\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4849e518bee9420da6126ebb01670583",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3238. Accuracy: 0.86\n",
      "\n",
      "Epoch [25/40]: Train accuracy: 0.838. Train loss: 0.3657. Evaluation accuracy: 0.861. Evaluation loss: 0.3221\n",
      "\n",
      "Start epoch [26/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "447f6839efa544528f61df3a11fbf6ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3760. Accuracy: 0.834\n",
      "Iteration: 200. Average training loss: 0.3752. Accuracy: 0.835\n",
      "Iteration: 300. Average training loss: 0.3661. Accuracy: 0.837\n",
      "Iteration: 400. Average training loss: 0.3574. Accuracy: 0.838\n",
      "Iteration: 500. Average training loss: 0.3592. Accuracy: 0.838\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a7bf475d32c48f9a50cea881d36dd03",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3227. Accuracy: 0.86\n",
      "\n",
      "Epoch [26/40]: Train accuracy: 0.839. Train loss: 0.3660. Evaluation accuracy: 0.863. Evaluation loss: 0.3206\n",
      "\n",
      "Start epoch [27/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8fa5ae62a97e4bb3abd36ebafeef4df8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3768. Accuracy: 0.831\n",
      "Iteration: 200. Average training loss: 0.3706. Accuracy: 0.832\n",
      "Iteration: 300. Average training loss: 0.3622. Accuracy: 0.834\n",
      "Iteration: 400. Average training loss: 0.3611. Accuracy: 0.836\n",
      "Iteration: 500. Average training loss: 0.3582. Accuracy: 0.837\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5cf9afb3313d4d93955da0078c6f47f0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3215. Accuracy: 0.86\n",
      "\n",
      "Epoch [27/40]: Train accuracy: 0.838. Train loss: 0.3649. Evaluation accuracy: 0.861. Evaluation loss: 0.3202\n",
      "\n",
      "Start epoch [28/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d86b6400a1704f529952a9f085e7f404",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3747. Accuracy: 0.835\n",
      "Iteration: 200. Average training loss: 0.3698. Accuracy: 0.835\n",
      "Iteration: 300. Average training loss: 0.3645. Accuracy: 0.836\n",
      "Iteration: 400. Average training loss: 0.3589. Accuracy: 0.837\n",
      "Iteration: 500. Average training loss: 0.3584. Accuracy: 0.839\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fafa27131aec48d4a4af3548bfacf920",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3241. Accuracy: 0.86\n",
      "\n",
      "Epoch [28/40]: Train accuracy: 0.840. Train loss: 0.3638. Evaluation accuracy: 0.859. Evaluation loss: 0.3216\n",
      "\n",
      "Start epoch [29/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ffd36a41f7e043789dc1231a53ec5055",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3708. Accuracy: 0.836\n",
      "Iteration: 200. Average training loss: 0.3646. Accuracy: 0.838\n",
      "Iteration: 300. Average training loss: 0.3663. Accuracy: 0.839\n",
      "Iteration: 400. Average training loss: 0.3591. Accuracy: 0.840\n",
      "Iteration: 500. Average training loss: 0.3595. Accuracy: 0.841\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16aaaeb6becb4a238d9f3a882268c40b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3223. Accuracy: 0.86\n",
      "\n",
      "Epoch [29/40]: Train accuracy: 0.841. Train loss: 0.3627. Evaluation accuracy: 0.862. Evaluation loss: 0.3201\n",
      "\n",
      "Start epoch [30/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "be608713ef2d4d3795b8bb8e3eb60706",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3773. Accuracy: 0.837\n",
      "Iteration: 200. Average training loss: 0.3613. Accuracy: 0.839\n",
      "Iteration: 300. Average training loss: 0.3663. Accuracy: 0.839\n",
      "Iteration: 400. Average training loss: 0.3586. Accuracy: 0.840\n",
      "Iteration: 500. Average training loss: 0.3591. Accuracy: 0.842\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5a623a681339436b95985e82e1aa5482",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3226. Accuracy: 0.86\n",
      "\n",
      "Epoch [30/40]: Train accuracy: 0.841. Train loss: 0.3642. Evaluation accuracy: 0.861. Evaluation loss: 0.3201\n",
      "\n",
      "Start epoch [31/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d0a6af8fdcc46ee91c818d8f24ec53a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3694. Accuracy: 0.836\n",
      "Iteration: 200. Average training loss: 0.3604. Accuracy: 0.836\n",
      "Iteration: 300. Average training loss: 0.3635. Accuracy: 0.837\n",
      "Iteration: 400. Average training loss: 0.3577. Accuracy: 0.837\n",
      "Iteration: 500. Average training loss: 0.3540. Accuracy: 0.838\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1482f3b48b5d478b8db197755fb326e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3210. Accuracy: 0.86\n",
      "\n",
      "Epoch [31/40]: Train accuracy: 0.839. Train loss: 0.3609. Evaluation accuracy: 0.862. Evaluation loss: 0.3186\n",
      "\n",
      "Start epoch [32/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d34a994f6a34340bb159f5729718c2c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3751. Accuracy: 0.833\n",
      "Iteration: 200. Average training loss: 0.3692. Accuracy: 0.835\n",
      "Iteration: 300. Average training loss: 0.3637. Accuracy: 0.837\n",
      "Iteration: 400. Average training loss: 0.3562. Accuracy: 0.839\n",
      "Iteration: 500. Average training loss: 0.3555. Accuracy: 0.840\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c3b1998b2c94717bcb9d6609700c208",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3213. Accuracy: 0.86\n",
      "\n",
      "Epoch [32/40]: Train accuracy: 0.841. Train loss: 0.3629. Evaluation accuracy: 0.863. Evaluation loss: 0.3185\n",
      "\n",
      "Start epoch [33/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "74689d6858f94164b1d934dd404b842e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3745. Accuracy: 0.831\n",
      "Iteration: 200. Average training loss: 0.3595. Accuracy: 0.836\n",
      "Iteration: 300. Average training loss: 0.3620. Accuracy: 0.837\n",
      "Iteration: 400. Average training loss: 0.3613. Accuracy: 0.837\n",
      "Iteration: 500. Average training loss: 0.3558. Accuracy: 0.839\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "36ed42d8cd1d469fb31cb3d03a08d411",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3209. Accuracy: 0.86\n",
      "\n",
      "Epoch [33/40]: Train accuracy: 0.841. Train loss: 0.3616. Evaluation accuracy: 0.862. Evaluation loss: 0.3187\n",
      "\n",
      "Start epoch [34/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f02a6fa1cc8428bb961e326684309e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3672. Accuracy: 0.837\n",
      "Iteration: 200. Average training loss: 0.3647. Accuracy: 0.835\n",
      "Iteration: 300. Average training loss: 0.3591. Accuracy: 0.838\n",
      "Iteration: 400. Average training loss: 0.3565. Accuracy: 0.839\n",
      "Iteration: 500. Average training loss: 0.3591. Accuracy: 0.839\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1358f9c41c4d4b1490c69351e733872d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3211. Accuracy: 0.86\n",
      "\n",
      "Epoch [34/40]: Train accuracy: 0.840. Train loss: 0.3597. Evaluation accuracy: 0.861. Evaluation loss: 0.3183\n",
      "\n",
      "Start epoch [35/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1fffb20983374d4290df7090eee0a846",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3730. Accuracy: 0.832\n",
      "Iteration: 200. Average training loss: 0.3674. Accuracy: 0.833\n",
      "Iteration: 300. Average training loss: 0.3615. Accuracy: 0.836\n",
      "Iteration: 400. Average training loss: 0.3550. Accuracy: 0.838\n",
      "Iteration: 500. Average training loss: 0.3627. Accuracy: 0.838\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5d8541eeb7744b549c44c1c91395b20e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3191. Accuracy: 0.86\n",
      "\n",
      "Epoch [35/40]: Train accuracy: 0.839. Train loss: 0.3622. Evaluation accuracy: 0.862. Evaluation loss: 0.3166\n",
      "\n",
      "Start epoch [36/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8549e9fc484a4b2cab009f507aa4c7da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3699. Accuracy: 0.837\n",
      "Iteration: 200. Average training loss: 0.3655. Accuracy: 0.837\n",
      "Iteration: 300. Average training loss: 0.3514. Accuracy: 0.841\n",
      "Iteration: 400. Average training loss: 0.3537. Accuracy: 0.843\n",
      "Iteration: 500. Average training loss: 0.3583. Accuracy: 0.843\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "480829bc55dd4d23a6248afff7ca9d32",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3195. Accuracy: 0.86\n",
      "\n",
      "Epoch [36/40]: Train accuracy: 0.843. Train loss: 0.3586. Evaluation accuracy: 0.863. Evaluation loss: 0.3169\n",
      "\n",
      "Start epoch [37/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0eea801743ba4e5abd6d5465cbaf5d4f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3700. Accuracy: 0.838\n",
      "Iteration: 200. Average training loss: 0.3627. Accuracy: 0.839\n",
      "Iteration: 300. Average training loss: 0.3568. Accuracy: 0.841\n",
      "Iteration: 400. Average training loss: 0.3538. Accuracy: 0.842\n",
      "Iteration: 500. Average training loss: 0.3550. Accuracy: 0.843\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a9fff17f943846c28202cfd7aab56e73",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3203. Accuracy: 0.86\n",
      "\n",
      "Epoch [37/40]: Train accuracy: 0.844. Train loss: 0.3590. Evaluation accuracy: 0.862. Evaluation loss: 0.3176\n",
      "\n",
      "Start epoch [38/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "782db8dfe8934cb18f4841c00f242192",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3690. Accuracy: 0.835\n",
      "Iteration: 200. Average training loss: 0.3605. Accuracy: 0.839\n",
      "Iteration: 300. Average training loss: 0.3587. Accuracy: 0.840\n",
      "Iteration: 400. Average training loss: 0.3509. Accuracy: 0.841\n",
      "Iteration: 500. Average training loss: 0.3556. Accuracy: 0.842\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "884939c13146444cbaba27cb29a20b7a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3191. Accuracy: 0.86\n",
      "\n",
      "Epoch [38/40]: Train accuracy: 0.842. Train loss: 0.3586. Evaluation accuracy: 0.862. Evaluation loss: 0.3171\n",
      "\n",
      "Start epoch [39/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8b1c2871a9f8405c93bd8b4bf9e07efd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3690. Accuracy: 0.838\n",
      "Iteration: 200. Average training loss: 0.3595. Accuracy: 0.837\n",
      "Iteration: 300. Average training loss: 0.3580. Accuracy: 0.840\n",
      "Iteration: 400. Average training loss: 0.3527. Accuracy: 0.841\n",
      "Iteration: 500. Average training loss: 0.3526. Accuracy: 0.842\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f08fce8a01b34095940200fdcd036ab7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3193. Accuracy: 0.86\n",
      "\n",
      "Epoch [39/40]: Train accuracy: 0.843. Train loss: 0.3574. Evaluation accuracy: 0.864. Evaluation loss: 0.3170\n",
      "\n",
      "Start epoch [40/40]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "709499744802492bb0804822636b4b3a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=543, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3682. Accuracy: 0.841\n",
      "Iteration: 200. Average training loss: 0.3669. Accuracy: 0.840\n",
      "Iteration: 300. Average training loss: 0.3553. Accuracy: 0.841\n",
      "Iteration: 400. Average training loss: 0.3551. Accuracy: 0.842\n",
      "Iteration: 500. Average training loss: 0.3483. Accuracy: 0.843\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b4e4d28d31134be8ba8c62bc7f0abbb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=155, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average evaluation loss: 0.3213. Accuracy: 0.86\n",
      "\n",
      "Epoch [40/40]: Train accuracy: 0.842. Train loss: 0.3584. Evaluation accuracy: 0.862. Evaluation loss: 0.3181\n"
     ]
    }
   ],
   "source": [
    "# Initialize parameters\n",
    "hidden_size = 8\n",
    "vocab_size = len(train_iterator.word2index)\n",
    "n_extra_feat = 10\n",
    "output_size = 2\n",
    "n_layers = 1\n",
    "dropout = 0.5\n",
    "learning_rate = 0.001\n",
    "epochs = 40\n",
    "spatial_dropout = True\n",
    "\n",
    "# Load the weights matrix\n",
    "weights = np.load('glove/weights-biRGU-glove.npy')\n",
    "\n",
    "# Check whether system supports CUDA\n",
    "CUDA = torch.cuda.is_available()\n",
    "\n",
    "model = BiGRU(hidden_size, vocab_size, n_extra_feat, weights, output_size, n_layers, dropout,\n",
    "              spatial_dropout, bidirectional=True)\n",
    "\n",
    "# Move the model to GPU if possible\n",
    "if CUDA:\n",
    "    model.cuda()\n",
    "\n",
    "model.add_loss_fn(nn.NLLLoss())\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "model.add_optimizer(optimizer)\n",
    "\n",
    "device = torch.device('cuda' if CUDA else 'cpu')\n",
    "\n",
    "model.add_device(device)\n",
    "\n",
    "# Instantiate the EarlyStopping\n",
    "early_stop = EarlyStopping(wait_epochs=2)\n",
    "\n",
    "train_losses_list, train_avg_loss_list, train_accuracy_list = [], [], []\n",
    "eval_avg_loss_list, eval_accuracy_list, conf_matrix_list = [], [], []\n",
    "\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    \n",
    "    print('\\nStart epoch [{}/{}]'.format(epoch+1, epochs))\n",
    "    \n",
    "    train_losses, train_avg_loss, train_accuracy = model.train_model(train_iterator)\n",
    "    \n",
    "    train_losses_list.append(train_losses)\n",
    "    train_avg_loss_list.append(train_avg_loss)\n",
    "    train_accuracy_list.append(train_accuracy)\n",
    "    \n",
    "    _, eval_avg_loss, eval_accuracy, conf_matrix = model.evaluate_model(val_iterator)\n",
    "    \n",
    "    eval_avg_loss_list.append(eval_avg_loss)\n",
    "    eval_accuracy_list.append(eval_accuracy)\n",
    "    conf_matrix_list.append(conf_matrix)\n",
    "    \n",
    "    print('\\nEpoch [{}/{}]: Train accuracy: {:.3f}. Train loss: {:.4f}. Evaluation accuracy: {:.3f}. Evaluation loss: {:.4f}'\\\n",
    "          .format(epoch+1, epochs, train_accuracy, train_avg_loss, eval_accuracy, eval_avg_loss))\n",
    "    \n",
    "    if early_stop.stop(eval_avg_loss, model, delta=0.003):\n",
    "        break\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model's improvement during training went so slowly that we had to use 40 epochs to train the model. As we can see, for the last epochs the changes of the validation and training losses began to fluctuate near the same value, thus the performance will likely not improve if we continue training. The validation accuracy of the model with pre-trained vectors is 0.862.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add the dataset initial loss\n",
    "train_avg_loss_list.insert(0, train_losses_list[0][0])\n",
    "eval_avg_loss_list.insert(0, train_losses_list[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAFQCAYAAAC4UNf/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3wc1bn/8c+z0kpadckVS+4FsGxsjDHNBlPtUBMCoYRQAnFMzYVAgFwSiBN+l3BJKLkklNBCd0hCgBAcErDpYAPG2AYXXOWCm4qtvtL5/TEjeS2rrGRpV5a/79drX7szc2bm2SI9c86cmWPOOURERKT7CsQ7ABEREelcSvYiIiLdnJK9iIhIN6dkLyIi0s0p2YuIiHRzSvYiIiLdnJK9tMjMVpnZCc0sm2RmS2IdU3uZ2Q4zG9LC8mbf697KzP5pZhd1wnYHmZkzs8SO3nZXYGaPm9mv4rTvTvnOZN+mZC/t5px72zm3f+Q8MzvXzD40szIz2+S/vsLMzF/+uJlV+4l3m5m9bmYHRKx/m5k91XhffmIZtofxpjvnVkTEEZd/5tHqoPf8DefcEx0VUzx1x4OxpnSn70y6DiV76TBm9mPgXuB/gb5AH2A6cBSQFFH0TudcOpAHrAMeiXGoMWWeDv9b66616s7QVT6rrhLHnugO72FfpGQv0TjUzBabWZGZPWZmKQBmNtnMCv3XWcAM4Arn3AvOue3O86lz7rvOuarGG3XOVQAzgbHtDczMLjGzlyOml5vZzIjptWY21n/tzGyYmU0Dvgv8xG9heDlik2PNbIGZlZjZ8/XvtYn9Xmxm75rZ7/yyX5rZ8RHLZ5vZ7Wb2LlAODDGzLDN7xMw2mNk6M/uVmSU0s/23/Jef+TGeU/95m9mNZrYReMzMcszsFTPb7H8/r5hZfqM4LouI+R0zu8svu9LMvhFRttn4zCzBX2+Lma0ATmnhO7nJzF5oNO9eM7uvtf34y39gZl+Y2Xb/dzfOzJ4EBgAv+5/HT/yyp5vZIjMr9t/rgRHbWeV/VguAMjNL9KfX+dteEvmdtcTMTjWz+f5+3jOzgxq9368i4v1WxLL638ndZrYNuC2K76Et39lgM3vL3/e/zex+a6JlLKL8Gf77KPVjnhrxWZ0QUa6hhc12nrK51MzWAG+Y2WtmdlWjbX9mZmf6rw8wr9Vum/85fyei3Mn+57Td/y6uj+Y7kD3knNNDj2YfwCpgIdAfyAXeBX7lL5sMFPqvpwJhILGV7T0esX4a8CTwWcTy24CnmljPAcOamD8EKMY7cN0PWA2si1hWBAQabyMyjkbv9SOgn/9evwCmN/M+Lvbf77VAEDgHKAFy/eWzgTVAAZDol3kReNB/3739ff2whc9ql/fsf95h4NdAMhACegDfBlKBDODPwIsR68wGLouIuQb4AZAAXA6sB8xf3mx8eC00X0b8Dt7049vt+wYG4h3gZPrTCcAG4PAo9nM2XmvPoYABw4CBEd/PCRH7GQGUASf6n+9PgOVAUkT5+X7MIWB/YC3Qz18+CBgaxe90HLAJOMx/Lxf5206OiLkf3m/wHD+m/Rr9Tq72fwehKL6Htnxn7wN34bWcTQRKaeLvxy87Ae83eqIfax5wQDOf7W312/E/Jwf8yf/OQsCFwLsR5Ufi/R0m+2XWApf473kcsAUo8MtuACb5r3OAcfH+P7cvPOIegB5d++H/E5geMX0y8JX/ejI7k/0FwMZG677n/wOoAI725z0OVPrz64CVwEER6zT8k2m0rSaTvb9srf8P5VzgIbzkcYD/z+alprZB88n+gojpO4EHmtnnxZH/dP15HwHf81/PBmZELOsDVAGhiHnnAW+28Nk3leyrgZQW1hkLFEVMz2bXxLE8Ylmqv4++rcUHvNHod3ASzSR7f/k7wIX+6xMjfjOt7WcW8KMWfouRCelnwMyI6QDegcLkiPLfj1g+DC9pnwAEW/ndN/w+gD8Av2y0fAlwTDPrzgfOiPjM1zTx22nye2jjdzYA70AiNWL5UzSf7B8E7o7ys72N3ZP9kIjlGXgHNQP96duBR/3X5wBvN7HvW/3Xa4Af4h8M6hGbh5rxJRprI16vxqvFNLYV6GkR5/Occ0c657L9ZZG/tbv8+YPwDgQiO/mF8WppDcysfrqmmfjm4CXCo/3Xs4Fj/Mec5t9WkzZGvC4H0lsou875/718jT+byM9tIN772uA3BRfj/QPsDeA3Re/wH5Na2Odm51xl/YSZpZrZg2a22sxKgbeAbGvm9EDk+3POlfsv01uLz39fjX8HLXkGL4kDnO9PE8V++gNftbLtev0i43DO1fkx5kWUWRuxfDnwX3iJbJOZPWdmTf2WGxsI/Lg+Xj/m/v7+MbMLI5r4i4FRQM+mYojQ3PfQlObK9gO2Rcxrbl/12vLZNiXys9wO/APvABv/+Wn/9UDgsEaf13fxDlDAa4k6GVhtZnPM7Ig9iEmipGQv0egf8XoAXo22sffxamxnRLtR59wa4EfAvWYW8mevwTsIiDQYqMWrtTWlPtlP8l/PofVk75qZ3xZ5Zt5VBr7Gn03kPtbifT49nXPZ/iPTOVcA4JwrcN7VAunOubdb2GfjuH+Md7B0mHMuE++AB7wm8LZoMT68ptfGv4OW/BmYbF7/gW+xM9m3tp+1wNBmttn4va/HSyyA1xHSj3Fdc+s4555xzk3013N4p0Rasxa4PSLebOdcqnPuWTMbCDwMXAX08A9iF7Lr598Rv7WmbAByzSw1Yl7/5grT8mdbhtdqUK9vE2Uav49ngfP8ZB3CO7VTv585jT6vdOfc5QDOubnOuTPwDvBexOu3I51MyV6icaWZ5ZtZLvBT4PnGBZxzxcAvgN+b2Vlmlm5mAfM6x6U1t2Hn3Ot4/7Sn+bNeA/Y3s++ZWdDf5/8DXnDOhZvZzBzgWLym4ULgbbw+BD2AT5tZ52u8c/p7ojdwjR/n2cCBwKtNFXTObQD+BfzGzDL9z2aomR3TwvajiTEDr3Wk2P+sbm3zu4guvpl47zXfzHKAm1rZ3ma8FpbHgJXOuS+i3M8fgevN7BDzDPMTKuz+ecwETjGz4/3Wnx/jHUi811RMZra/mR1nZsl4p5Iq8A4iW/MwMN3MDvNjSjOzU8wsA++37YDN/j4uwavZdzrn3GpgHl6nvyQ/6Z7WwiqPAJf4n1fAzPJs52Wv84Fz/d/yeOCsKEJ4Fe+gaQbwvN+yAvAKMCLibzhoZoea2YF+nN81syznXA1eH4NovgPZQ0r2Eo1n8P5Br/AfTV6f7py7E7gOr6PUJrx/zg8CN9LMP2Df/+L1jE92zm3Ca+L7ob+NhXidii5vbmXn3FJgB16SxzlX6sf5rnOuuX8kjwAj/WbGF1uIrSUfAsPxOh/dDpzlnNvaQvkL8TpSLcbrOPgCXqfC5twGPOHH+J1mytyDV6vaAnyAd7DUXi3F9zDe+fTPgE+Av0axvWfwzo8/02h+s/txzv0Z77N8BtiOV/PL9df7H+AW//O43jm3BK+vyO/w3v9pwGnOuepm4kkG7vDLbsQ7WPtpa2/COTcPr4Pc//nxLsc7l45zbjHwG7yWra+B0XidWGPlu8AReKfKfoV3IL7blS8AzrmP8Pqx3I33NzWHnS0jP8Or9RfhHbQ3/s6a2l4V3u9gl+/Yb+I/Ca9pfz3eZ13fqRTge8Aq/7TTdLzvUDpZfY9OEWkDM7sYrxPVxHjHIlLPzJ4HvnTOtauFR7ov1exFRPZSfvP4UL9Zfipen5n2tlRJN6Y7IYmI7L364jWl9wAKgcudc831U5F9mJrxRUREujk144uIiHRzSvYiIiLdXLc8Z9+zZ083aNCgeIchIiISMx9//PEW51yvppZ1y2Q/aNAg5s2bF+8wREREYsbMmr2NtZrxRUREujklexERkW5OyV5ERKSb65bn7EVEpHU1NTUUFhZSWVnZemHpMlJSUsjPzycYDLZe2KdkLyKyjyosLCQjI4NBgwax62jN0lU559i6dSuFhYUMHjw46vXUjC8iso+qrKykR48eSvR7ETOjR48ebW6NUbIXEdmHKdHvfdrznSnZi4hIXGzdupWxY8cyduxY+vbtS15eXsN0dXV1VNu45JJLWLJkSYtl7r//fp5++umOCJmJEycyf/78DtlWLOmcvYiIxEWPHj0aEudtt91Geno6119//S5lnHM45wgEmq6bPvbYY63u58orr9zzYPdyqtm3orKyko9fvI/l89+OdygiIvuE5cuXM2rUKKZPn864cePYsGED06ZNY/z48RQUFDBjxoyGsvU17XA4THZ2NjfddBNjxozhiCOOYNOmTQDccsst3HPPPQ3lb7rpJiZMmMD+++/Pe++9B0BZWRnf/va3GTNmDOeddx7jx49vtQb/1FNPMXr0aEaNGsVPf/pTAMLhMN/73vca5t93330A3H333YwcOZIxY8ZwwQUXdPhn1hrV7FsRCCRw8Kc/573N32fY2EnxDkdEZJ+wePFiHnvsMR544AEA7rjjDnJzcwmHwxx77LGcddZZjBw5cpd1SkpKOOaYY7jjjju47rrrePTRR7npppt227Zzjo8++oiXXnqJGTNm8Nprr/G73/2Ovn378pe//IXPPvuMcePGtRhfYWEht9xyC/PmzSMrK4sTTjiBV155hV69erFlyxY+//xzAIqLiwG48847Wb16NUlJSQ3zYilmyd7MpgL3AgnAH51zdzRafjdwrD+ZCvR2zmX7yy4CbvGX/co590RsooakpCDbA2mUl2yJ1S5FRGLuFy8vYvH60g7d5sh+mdx6WkG71h06dCiHHnpow/Szzz7LI488QjgcZv369SxevHi3ZB8KhfjGN74BwCGHHMLbbzfdInvmmWc2lFm1ahUA77zzDjfeeCMAY8aMoaCg5bg//PBDjjvuOHr27AnA+eefz1tvvcWNN97IkiVL+NGPfsTJJ5/MSSedBEBBQQEXXHABZ5xxBt/85jfb+GnsuZg045tZAnA/8A1gJHCeme3yLTnnrnXOjXXOjQV+B/zVXzcXuBU4DJgA3GpmObGIu15VMIvwjq0452K5WxGRfVZaWlrD62XLlnHvvffyxhtvsGDBAqZOndrkpWdJSUkNrxMSEgiHw01uOzk5ebcybf3/3lz5Hj16sGDBAiZOnMh9993HD3/4QwBmzZrF9OnT+eijjxg/fjy1tbVt2t+eilXNfgKw3Dm3AsDMngPOABY3U/48vAQPMAV43Tm3zV/3dWAq8GynRhwpJZvUiu1sKKmkX3YoZrsVEYmV9tbAY6G0tJSMjAwyMzPZsGEDs2bNYurUqR26j4kTJzJz5kwmTZrE559/zuLFzaUnz+GHH84NN9zA1q1bycrK4rnnnuP6669n8+bNpKSkcPbZZzN48GCmT59ObW0thYWFHHfccUycOJGnn36a8vJyMjIyOvQ9tCRWyT4PWBsxXYhXU9+NmQ0EBgNvtLBuXhPrTQOmAQwYMGDPI44QzOhBdvE6Fq4rUbIXEYmxcePGMXLkSEaNGsWQIUM46qijOnwfV199NRdeeCEHHXQQ48aNY9SoUWRlZTVbPj8/nxkzZjB58mScc5x22mmccsopfPLJJ1x66aU45zAzfv3rXxMOhzn//PPZvn07dXV13HjjjTFN9AAWi6ZpMzsbmOKcu8yf/h4wwTl3dRNlbwTy65eZ2Q1AsnPuV/70z4By59xvmtvf+PHjXUeOZx+e+X0KF77D3ya9wrUnjuiw7YqIxNMXX3zBgQceGO8wuoRwOEw4HCYlJYVly5Zx0kknsWzZMhITu2Y/9qa+OzP72Dk3vqnysXoXhUD/iOl8YH0zZc8FIi+KLAQmN1p3dgfG1qrEtFxyE8pZtL4klrsVEZEY2bFjB8cffzzhcBjnHA8++GCXTfTtEat3MhcYbmaDgXV4Cf38xoXMbH8gB3g/YvYs4P9FdMo7Cbi5c8NtJJRDutvBosLYXy4hIiKdLzs7m48//jjeYXSamPTGd86FgavwEvcXwEzn3CIzm2Fmp0cUPQ94zkWcW/A75v0S74BhLjCjvrNezIRyCOAo276NLTuqYrprERGRPRWzNgrn3KvAq43m/bzR9G3NrPso8GinBdeakNeokG1lLFpfyjEjesUtFBERkbbS7XKjUZ/s2aHz9iIistdRso+Gn+yHZ9awaF3H3mFKRESksynZR8NP9iOza1WzFxHpQAkJCQ3D2o4dO5Y77rij9ZWaMHnyZNp7yfXs2bMbBsQBeOCBB/jTn/7Urm1FWrVqFaNGjdrj7XSE7nNdQWfyk/2wjBpWrS6ntLKGzJRgnIMSEdn7hUKhuI8PP3v2bNLT0znyyCMBmD59elzj6Qyq2UcjlA1A/5DXE7+jB4sQEZGd/vnPf/Kd73ynYXr27NmcdtppAFx++eUNQ93eeuutTa6fnp7e8PqFF17g4osvBuDll1/msMMO4+CDD+aEE07g66+/ZtWqVTzwwAPcfffdjB07lrfffpvbbruNu+66C4D58+dz+OGHc9BBB/Gtb32LoqIiwGtJuPHGG5kwYQIjRoxodtCdepWVlVxyySWMHj2agw8+mDfffBOARYsWMWHCBMaOHctBBx3EsmXLKCsr45RTTmHMmDGMGjWK559/vn0fZAQl+2gkBCEpg77BCgAWKdmLiHSIioqKXZrxn3/+eU488UQ++OADysrKAHj++ec555xzALj99tuZN28eCxYsYM6cOSxYsCDqfU2cOJEPPviATz/9lHPPPZc777yTQYMGMX36dK699lrmz5/PpEm7DmV+4YUX8utf/5oFCxYwevRofvGLXzQsC4fDfPTRR9xzzz27zG/K/fffD8Dnn3/Os88+y0UXXURlZSUPPPAAP/rRj5g/fz7z5s0jPz+f1157jX79+vHZZ5+xcOHCDhkHQM340QrlkFq7nd4ZySxap/P2ItLN/PMm2Ph5x26z72j4Rsvn4Jtrxp86dSovv/wyZ511Fv/4xz+48847AZg5cyYPPfQQ4XCYDRs2sHjxYg466KCowiksLOScc85hw4YNVFdXM3jw4BbLl5SUUFxczDHHHAPARRddxNlnn92wvKmhcpvzzjvvcPXV3h3iDzjgAAYOHMjSpUs54ogjuP322yksLOTMM89k+PDhjB49muuvv54bb7yRU089dbcDkPZQzT5aoWyoKGJUXpZq9iIineycc85h5syZvPHGGxx66KFkZGSwcuVK7rrrLv7zn/+wYMECTjnllCaHujWzhteRy6+++mquuuoqPv/8cx588MEm122LpobKbU5z49Ccf/75vPTSS4RCIaZMmcIbb7zBiBEj+Pjjjxk9ejQ333wzM2bM2KM4QTX76IVyoKKIgv6ZzF6yiYrqWkJJCfGOSkSkY7RSA4+1yZMnc+mll/Lwww83NOGXlpaSlpZGVlYWX3/9Nf/85z+ZPHnybuv26dOHL774gv3335+//e1vDSPMlZSUkJfnDZr6xBNPNJTPyMigtHT3SlxWVhY5OTm8/fbbTJo0iSeffLKhlt9WRx99NE8//TTHHXccS5cuZc2aNey///6sWLGCIUOGcM0117BixQoWLFjAAQccQG5uLhdccAHp6ek8/vjj7dpnJCX7aIVyYNNiCvplUefgy42lHDwgp/X1RESkWfXn7OtNnTqVO+64g4SEBE499VQef/zxhsQ8ZswYDj74YAoKCloc6vaOO+7g1FNPpX///owaNYodO3YAcNttt3H22WeTl5fH4YcfzsqVKwE47bTTOOuss/j73//O7373u1229cQTTzB9+nTKy8sZMmQIjz32WLve5xVXXMH06dMZPXo0iYmJPP744yQnJ/P888/z1FNPEQwG6du3Lz//+c+ZO3cuN9xwA4FAgGAwyB/+8Id27TNSTIa4jbWOHuIWgJf/C758hbWXLmDSnW/yq2+O4oLDB3bsPkREYkhD3O692jrErc7ZR8tvxs/PTiErFNTNdUREZK+hZB+t1FyoC2PVOyjol6lOeiIistdQso+Wfxe9+h75X27YTk1tXXxjEhERiYKSfbQikn1Bv0yqa+tY9vWO+MYkIrKHumO/re6uPd+Zkn20dkn2WQA6by8ie7WUlBS2bt2qhL8Xcc6xdetWUlJS2rSeLr2LVkSyHzw4jdSkBBatL+XsltcSEemy8vPzKSwsZPPmzfEORdogJSWF/Pz8Nq2jZB+tiGSfEDAO3C9TNXsR2asFg8FWbxkr3YOa8aOV4o18R4U34tGofpksXl9KXZ2av0REpGtTso9WMAWCqQ3JvqBfFmXVtazaWhbnwERERFqmZN8WoRyoKAagIC8TgIW63l5ERLo4Jfu28O+iBzC8dwZJCQENdysiIl1ezJK9mU01syVmttzMbmqmzHfMbLGZLTKzZyLm15rZfP/xUqxi3k1Esk9KDDCib7rupCciIl1eTHrjm1kCcD9wIlAIzDWzl5xziyPKDAduBo5yzhWZWe+ITVQ458YSb6Fs2LK8YXJUvyxeW7QR59wu4yeLiIh0JbGq2U8AljvnVjjnqoHngDMalfkBcL9zrgjAObcpRrFFL6JmD1DQL5Pi8hrWl1TGMSgREZGWxSrZ5wFrI6YL/XmRRgAjzOxdM/vAzKZGLEsxs3n+/G92drDNqk/2/t2mCvK8O+kt1Hl7ERHpwmKV7Jtq4258gXoiMByYDJwH/NHM/IvbGeCP0Xs+cI+ZDd1tB2bT/AOCeZ12N6hQDtRWQU0FAAf2zSRgqJOeiIh0abFK9oVA/4jpfGB9E2X+7pyrcc6tBJbgJX+cc+v95xXAbODgxjtwzj3knBvvnBvfq1evjn8HsMtd9ABCSQkM7aVOeiIi0rXFKtnPBYab2WAzSwLOBRr3qn8ROBbAzHriNeuvMLMcM0uOmH8UsJh4aJTsAUblZbFQt80VEZEuLCbJ3jkXBq4CZgFfADOdc4vMbIaZne4XmwVsNbPFwJvADc65rcCBwDwz+8yff0dkL/6YaiLZF/TL5OvSKjZvr4pLSCIiIq2J2UA4zrlXgVcbzft5xGsHXOc/Isu8B4yORYytajLZ7xzudvL+vZtaS0REJK50B722aCLZj+zn3TZX5+1FRKSrUrJviyaSfVYoyIDcVA13KyIiXZaSfVsEUyEhaZdkDzAqL5OF61SzFxGRrknJvi3M/BvrbNtldkG/LNZsK6ekoiZOgYmIiDRPyb6tGt0yF7we+QCLdd5eRES6ICX7tgrlNoxpXy+yR76IiEhXo2TfVk3U7HtlJNMnM1k98kVEpEtSsm+rJpI9eLV7DYgjIiJdkZJ9W4Wym0z2o/pl8tXmHVRU18YhKBERkeYp2bdVKAdqyqFm1zHsC/KyqHPwxUY15YuISNeiZN9W9TfWqWzcSU930hMRka5Jyb6tmriLHkBedojs1CCL1SNfRES6GCX7tmom2ZsZg3umsXpreRyCEhERaZ6SfVs1k+wB+ueksrZIyV5ERLoWJfu2aiHZ5+eE2FBcSbi2LsZBiYiINE/Jvq1aTPaphOscX2+vinFQIiIizVOyb6vkDLCEppvxc0MArN2mpnwREek6lOzbqmHku6Zr9gCFRRWxjkpERKRZSvbt0Uyy75edghkUqpOeiIh0IUr27dFMsk9OTKBPRgprt6lmLyIiXYeSfXs0k+zB65Gvmr2IiHQlSvbt0WqyV81eRES6DiX79gjlQEVxk4v656aysVTX2ouISNcRs2RvZlPNbImZLTezm5op8x0zW2xmi8zsmYj5F5nZMv9xUaxiblYoB6pKobZmt0X5OSFq6xwbSiqbWFFERCT2EmOxEzNLAO4HTgQKgblm9pJzbnFEmeHAzcBRzrkiM+vtz88FbgXGAw742F+36Xb0WGgY+a4E0nrusqj+8ru1ReX0z02NdWQiIiK7iVXNfgKw3Dm3wjlXDTwHnNGozA+A++uTuHNukz9/CvC6c26bv+x1YGqM4m5afbIv37bbov661l5ERLqYWCX7PGBtxHShPy/SCGCEmb1rZh+Y2dQ2rBtbLdwyt29WCgGDQt1FT0REuoiYNOMD1sQ812g6ERgOTAbygbfNbFSU62Jm04BpAAMGDNiTWFvXQrJPSgzQNzNFNXsREekyYlWzLwT6R0znA+ubKPN351yNc24lsAQv+UezLs65h5xz451z43v16tWhwe8mlO09N3f5XW6qkr2IiHQZsUr2c4HhZjbYzJKAc4GXGpV5ETgWwMx64jXrrwBmASeZWY6Z5QAn+fPiJzXXe27hWnuNay8iIl1FTJrxnXNhM7sKL0knAI865xaZ2QxgnnPuJXYm9cVALXCDc24rgJn9Eu+AAWCGc273nnGxlJwFWAvJPpWNpeuoDteRlKhbGYiISHzF6pw9zrlXgVcbzft5xGsHXOc/Gq/7KPBoZ8cYtUDAa8pvJtn3zwnhHGwoqWBgj7QYByciIrIrVTvbq8Vb5vrX2mtAHBER6QKU7Nurlfvjg4a6FRGRrkHJvr1aSPb7ZaWQEDD1yBcRkS5Byb69Wkj2iQkB9stKUY98ERHpEpTs26uFZA8a6lZERLoOJfv2CuV4A+HU1Ta5uH9Oqs7Zi4hIl6Bk316hHMB5Cb8J+TmpfF1aRWVN0wcDIiIisaJk314t3B8fdvbIX1+spnwREYkvJfv2akj2xU0urh/LXuftRUQk3pTs2yvKmr165IuISLwp2bdXK8m+T2YKwQRday8iIvGnZN9erST7hIDRL1uX34mISPwp2bdXSstj2oM/1O02NeOLiEh8Kdm3V0IiJGe2nOyzU1WzFxGRuFOy3xMtDHML0D83xJYdutZeRETiS8l+T4RyoGJbs4vrh7rVnfRERCSelOz3RBT3xwdYq6Z8ERGJIyX7PdFKsteNdUREpCtQst8TrST7XunJJCUEKFSPfBERiSMl+z1Rn+zr6ppcHAgYeRrqVkRE4kzJfk+EcsDVQfX2Zot449qrZi8iIvGjZL8nQrnec4ud9FLVQU9EROJKyX5PtHLLXPBq9tvKqimrCscoKBERkV3FLNmb2VQzW2Jmy83spiaWX2xmm81svv+4LGJZbcT8l2IVc6uiSPb1PfLXaVx7ERGJk8RY7MTMEoD7gROBQmCumb3knFvcqOjzzrmrmthEhXNubGfH2WZR1uwB1m4rZ/SD+lIAACAASURBVESfjFhEJSIisotY1ewnAMudcyucc9XAc8AZMdp352lDslePfBERiZdYJfs8YG3EdKE/r7Fvm9kCM3vBzPpHzE8xs3lm9oGZfbNTI22LUOsj3/VKTyY5MaAe+SIiEjexSvbWxDzXaPplYJBz7iDg38ATEcsGOOfGA+cD95jZ0N12YDbNPyCYt3nz5o6Ku2WJyRBMg4riZouYmT/UrWr2IiISH7FK9oVAZE09H1gfWcA5t9U5V+VPPgwcErFsvf+8ApgNHNx4B865h5xz451z43v16tWx0beklbvogXf5XWGxavYiIhIfsUr2c4HhZjbYzJKAc4FdetWb2X4Rk6cDX/jzc8ws2X/dEzgKaNyxL36iSPb9c3UXPRERiZ+Y9MZ3zoXN7CpgFpAAPOqcW2RmM4B5zrmXgGvM7HQgDGwDLvZXPxB40Mzq8A5O7miiF3/8tDKmPXg1++LyGrZX1pCREoxRYCIiIp6YJHsA59yrwKuN5v084vXNwM1NrPceMLrTA2yvUA5sWdpikcge+Qfup2QvIiKxpTvo7alomvFzNNStiIjEj5L9nqpP9q7xxQU7Rd5YR0REJNaU7PdUKAdqq6Gm+USem5ZEKJigmr2IiMSFkv2eiuIuembm98hXzV5ERGJPyX5PRZHsQUPdiohI/CjZ76mok71q9iIiEh9K9nuqPtmXb2uxWP+cVLZXhimpqIlBUCIiIjsp2e+pNtTsQT3yRUQk9pTs91QbztmDrrUXEZHYizrZm9l1ZjbWf324ma0xsxVmdkTnhbcXCIYgITmq++MDOm8vIiIx15aa/bXASv/1/wC/BW4H7unooPYqZlHdRS8rFCQ9OVE1exERibm23Bs/yzlXYmYZwBjgBOdcrZn9ppNi23tEkezrx7VXzV5ERGKtLcl+rZkdCRQAb/mJPhOo7ZzQ9iKhHKgobrVYfk6qkr2IiMRcW5L9DcALQDXwbX/eqcBHHR3UXic1F7atbLVYfk6I97/agnMOM4tBYCIiIm1I9v4Qtf0azf6z/9i3RTGmPXjJvqy6luLyGnLSkmIQmIiISNt64480sz7+63Qz+wXe+PMaoD2Kc/YA/XN1+Z2IiMReW3rjPwNk+6/vAo4GjgAe7Oig9jqhHAhXQE3LSbzhxjo6by8iIjHUlnP2g5xzS8w72fwtvI56Fey8HG/f1XBjnWLvuvtm7LyxjpK9iIjETltq9lX+ZXcTgLXOuS1AFZDSKZHtTaK8i15WKEhmiq61FxGR2GpLzf4Z4A0gA/g/f944VLOPOtmDP9St7o8vIiIx1Jbe+Nea2UlAjXPuTX92Hd6d9fZtbUr2IVZuKevkgERERHZqS80e59y/zGyAfz/8dc65eZ0U196lDcm+f24qby/TtfYiIhI7bbn0bj8zmwMsA/4KLDezOWbW+Nr75tafamZLzGy5md3UxPKLzWyzmc33H5dFLLvIzJb5j4uijTlm2lizr6ipZWtZdScHJSIi4mlLB70/AJ8Buc65/YAcYD7wQGsrmlkCcD/wDWAkcJ6ZjWyi6PPOubH+44/+urnArcBheJ0DbzWznDbE3fmS0iGQGF3NXkPdiohIjLUl2U8EfuycKwPwn38CHBnFuhOA5c65Fc65auA54Iwo9zsFeN05t805VwS8DkxtQ9ydL8qR7wDyNdStiIjEWFuSfRFerTzS/kDrI8BAHrA2YrrQn9fYt81sgZm9YGb927hufEWb7P2a/dptqtmLiEhstKWD3p3Av83sEWA1MBC4BPhZFOs21RPNNZp+GXjWOVdlZtOBJ4DjolwXM5sGTAMYMGBAFCF1sCiTfXpyIjmpQdXsRUQkZqKu2TvnHgbOAXoCp/nP3wPyo1i9EOgfMZ0PrG+0/a3OuSp/8mHgkGjX9dd/yDk33jk3vlevXlGE1MGiTPbg1e7X6Fp7ERGJkbY04+Oce8M5d5lz7mTn3GXAHOC/o1h1LjDczAabWRJwLvBSZAEz2y9i8nTgC//1LOAkM8vxO+ad5M/rWtqQ7Mf2z+bDFdvYVFrZyUGJiIi0Mdk3o9WLxZ1zYeAqvCT9BTDTObfIzGaY2el+sWvMbJGZfQZcA1zsr7sN+CXeAcNcYIY/r2tpQ7K/dOJgwnV1PPKubj4oIiKdr0031WnGbufPmyzk3KvAq43m/Tzi9c14Q+Y2te6jwKN7EGPnC+VA9Q4IV0Niy2PVD+qZxsmj9+PpD9ZwxeRhZIU0SrCIiHSeVpO9mR3XwuKWs9q+pP7GOpXFkN671eLTjxnKKws28NQHq7ny2GGdHJyIiOzLoqnZP9LK8jUdEcheL/IuelEk+1F5WRw9ohePvbuSSycOJiWY0MkBiojIvqrVc/bOucGtPWIRaJcXyvaeozxvD3DF5KFs2VHNn+etbb2wiIhIO3VEBz2BNt0fv95hg3M5eEA2D761gnBtXScFJiIi+zol+47SjmRvZlwxeRiFRRX84/MNnRSYiIjs65TsO0o7kj3A8Qf0ZnjvdP4w+yuci+rCBhERkTZRsu8oyVlggTYn+0DAmH7MUL7cuJ03l2zqpOBERGRfpmTfUQIBSMluc7IHOH1sP/KyQ/xh9ledEJiIiOzrlOw7UmYebFna5tWCCQF+MGkwc1cVMXdV17s5oIiI7N2U7DvSsONg9ftQWdrmVc85dAC5aUmq3YuISIdTsu9Iw6dAXQ2seLPNq4aSErjkyEG88eUmvtjQ9oMFERGR5ijZd6T+h0FKFiz9V7tWv/CIQaQlJfDAHNXuRUSk4yjZd6SERBh2AiybBXVtv0lOVmqQ7x4+kJc/W8+arRrvXkREOoaSfUcbPgXKNsOGT9u1+qUTB5MYCPDw2ys6ODAREdlXKdl3tGEneNfbL53VrtX7ZKZw5rg8Zs5by+btVR0cnIiI7IuU7DtaWg/IP7TdyR7gh8cMpbq2jsfeXdmBgYmIyL5Kyb4zjJgCG+bD9o3tWn1wzzROHrUfT76/mtLKmg4OTkRE9jVK9p1h+BTveVn7euUDXD55KNurwjzz4ZoOCkpERPZVSvadoU8BZObvUVP+qLwsJg3vyYNzvmJ9cUUHBiciIvsaJfvOYAYjToKv3oRw+zvZ3XpaATW1jsuf+pjKmtoODFBERPYlSvadZcRUqCmDVe+0exPDeqdz9zlj+aywhP/+20INgSsiIu2iZN9ZBk2CxJQ9Om8PcOLIPvzXCcP5yyeFPPHeqo6JTURE9ilK9p0lKRUGHwNLX4M9rJFfc9xwThzZh1/+4wve/2prBwUoIiL7ipglezObamZLzGy5md3UQrmzzMyZ2Xh/epCZVZjZfP/xQKxi3mMjToKiVbBl2R5tJhAwfvudMQzqkcqVz3zCOnXYExGRNohJsjezBOB+4BvASOA8MxvZRLkM4Brgw0aLvnLOjfUf0zs94I5Sfwne0tf2eFMZKUEeunA8NeE6fvjkPHXYExGRqMWqZj8BWO6cW+GcqwaeA85ootwvgTuByhjF1bmy+0Pvgj0+b19vaK907j1vLIvWl3LzXz9Xhz0REYlKrJJ9HrA2YrrQn9fAzA4G+jvnXmli/cFm9qmZzTGzSZ0YZ8cbcRKseR8qijtkc8cd0IfrThjB3z5dxyPv6Ha6IiLSulgle2tiXkO11MwCwN3Aj5sotwEY4Jw7GLgOeMbMMnfbgdk0M5tnZvM2b97cQWF3gBFToS4MX73RYZu88thhTCnow//880veXb6lw7YrIiLdU6ySfSHQP2I6H1gfMZ0BjAJmm9kq4HDgJTMb75yrcs5tBXDOfQx8BYxovAPn3EPOufHOufG9evXqpLfRDvmHQiinw5ryweuw95vvjGVIzzSueuYT1m4r77Bti4hI9xOrZD8XGG5mg80sCTgXeKl+oXOuxDnX0zk3yDk3CPgAON05N8/Mevkd/DCzIcBwYO8Z7D2QAMNO9JJ9Xcd1qktPTuShC8cTrnNMe/JjKqrVYU9ERJoWk2TvnAsDVwGzgC+Amc65RWY2w8xOb2X1o4EFZvYZ8AIw3Tm3rXMj7mAjpkD5Vlj3SYdudnDPNO4772C+3FjKDS98Rm2dOuyJiMjuEmO1I+fcq8Crjeb9vJmykyNe/wX4S6cG19mGHgeW4F2C1//QDt30sfv35idTDuDXr31JWVWY+847mIyUYIfuQ0RE9m66g14spOZC/8NgWftHwWvJ5ZOH8sszCnhr2RbO/P17rN5a1in7ERGRvZOSfayMmAIbP4eSdZ2y+e8dMYgnvz+BTdurOOP+d3nvK/XSFxERj5J9rIzw76bXgb3yGztyWE/+fuVR9ExP5sJHPuKpD1Z32r5ERGTvoWQfK70OgOwBsLRzmvLrDeqZxl+vOJJJw3tyy4sL+dmLC6mprevUfYqISNemZB8rZt698lfOgZrOHcgmMyXIHy86lGlHD+HJD1Zz0aMfUVxe3an7FBGRrkvJPpZGTIWaclj1TqfvKiFg/PTkA7nr7DHMW1XEGfe/y/JN2zt9vyIi0vUo2cfSoIkQTO30pvxIZx2Sz7PTDqOsKsy37n+PN5dsitm+RUSka1Cyj6VgCgw+xkv2MRyx7pCBufz9qon0z03l+4/P5RcvL2JHVThm+xcRkfhSso+1EVOgZA1s/jKmu83LDvHC5Ufw3cMG8Ph7qzjpt3P49+KvYxqDiIjEh5J9rA0/yXte+lrMd52alMivvjmaF6YfSXpKIpf9aR5XPP0xm0orYx6LiIjEjpJ9rGXlQd/RsODPUFsTlxAOGZjDK1dP4oYp+/PvLzZx/G/n8NQHq6nTvfVFRLolJft4OPonsGkRvP2buIWQlBjgymOHMeu/jmZ0Xha3vLiQsx98n6Vfq8e+iEh3o2QfDyNPh9Hfgbf+F9Z/GtdQBvdM4+nLDuOus8ewYvMOTrnvbX77ryVU1mjIXBGR7kLJPl5OvhPSesPfpkNNfM+ZmxlnHZLPv687htMO6sd9byzn5HvfZrYu0xMR6RaU7OMllANn/M7rlf/GL+MdDQA90pP57TljefLSCdQ5x8WPzeXCRz9iyUY17YuI7M2U7ONp2Akw/vvw/v2w6t14R9Ng0vBe/OvaY/jZqSOZv6aIb9z7Fjf/9XM2b6+Kd2giItIO5mJ4c5dYGT9+vJs3b168w4hO1Q544CjvJjuXvwvJGfGOaBdFZdXc98Yynnx/NcmJAa44dhiXThxMSjAh3qGJiEgEM/vYOTe+qWWq2cdbcjp88wEoXgP/uiXe0ewmJy2JW08r4F/XHs2Rw3ryv7OWcPxv5vD3+evojgeKIiLdkZJ9VzDwCDjyavj4cVj2eryjadKQXuk8fOF4nv3B4WSnBvnRc/P55u/fY96qbfEOTUREWqFk31Uc+9/Q60D4+1VQ3nUT6BFDe/DyVRO56+wxbCyp4KwH3mfan+apE5+ISBemZN9VBFPgzAehfAu8ekO8o2lRIOBdqvfm9ZO57sQRvP/VVqbe+xbXPj+f1VvL4h2eiIg0omTflew3Bo65ERa+AIv+Fu9oWpWalMg1xw/nrZ8cy7Sjh/DPhRs4/jdz+OnfPmdjie63LyLSVag3fldTG4ZHToSiVXDFB5DRJ94RRW1TaSX/9+Zynv1oDWbGhYcP5PLJQ+mRnhzv0EREur0u0RvfzKaa2RIzW25mN7VQ7iwzc2Y2PmLezf56S8xsSmwijpOERPjWg1BTDi9fE9Nx7/dU78wUZpwxijd+PJnTx/Tj0XdXcvSdb/Lbfy2htDI+g/6IiEiMavZmlgAsBU4ECoG5wHnOucWNymUA/wCSgKucc/PMbCTwLDAB6Af8GxjhnGv25u17dc2+3vu/h1k3w+n/B+O+F+9o2mX5pu3c/foy/vH5BrJCQU4bsx+j+mVR0C+LEX3TSU7UtfoiIh2lpZp9YoximAAsd86t8AN6DjgDWNyo3C+BO4HrI+adATznnKsCVprZcn9773d61PF02HRY8iq8ej2kZHmD5+xlhvXO4P7vjuPydSXc+59lvPjpep76YA0AiQFjWO90CvplUdAvk4J+mYzsl0lGSjDOUYuIdD+xSvZ5wNqI6ULgsMgCZnYw0N8594qZXd9o3Q8arZvXWYF2GYEAnP0EPHsOzLwQpv4PHH55vKNql1F5WTx84Xjq6hxrtpWzaH0pi9aXsGh9KXOWbuYvnxQ2lB3YI5XReVmMG5DDIQNzGNkvk2CC+pGKiOyJWCV7a2Jew/kDMwsAdwMXt3XdiG1MA6YBDBgwoF1BdjlpPeCil+Evl8FrN3l32TvpVxDYO5u/AwFjUM80BvVM45SD9muYv6m0cpcDgE/XFPPKgg0ApAQDHJSXzbiBXvIfNyBbHf5ERNooVsm+EOgfMZ0PrI+YzgBGAbPNDKAv8JKZnR7FugA45x4CHgLvnH1HBh9XwRB8508w67/hg99DyVo482FvfjfROzOF3pkpHHtA74Z5G0oq+GR1MR+vLuKTNUU88s4KHpjjfa2DeqQybmAOhw3O5fgD+9BTyV9EpEWx6qCXiNdB73hgHV4HvfOdc4uaKT8buN7voFcAPMPODnr/AYZ3+w56TXn/9zDrp5B/KJz3LKT1jHdEMVNZU8vn60q85O8fAGzZUU3AYPygXKYW9GXKqL7kZXefgyARkbaIewc951zYzK4CZgEJwKPOuUVmNgOY55x7qYV1F5nZTLzOfGHgypYSfbd2xBWQlQd/neZdi//dF6DH0HhHFRMpwQQOHZTLoYNyAXDO8cWG7cxatJFZizYy45XFzHhlMaPzspg6qi9TCvowrHfXGkFQRCRedFOdvdGaD+HZc8EMznsO+k+Id0Rxt2pLGbMWbeS1RRv5dE0xAEN7pTGloC/HH9iHA/fLIDUpVmetRERir6WavZL93mrrV/DUt2H7Bvj2H+HA0+IdUZexsaSS1xd7if+DFduorXOYQX5OiBG9MxjWJ50RvTMY0SeDob3TdBAgIt2Ckn13VbYFnjkH1n0MU/6fd2meNXXxwr6rqKyaD1duZenXO1j69XaWb9rBis1lVNfWAexyEDCibwZj+2czbkAOvTLU6U9E9i5K9t1ZdTn89Qfw5StQcCacdo93Ex5pVri2jlVby1n29XaWbfIOApZ9vYMVW3ZQU+v9PQzskcohA3I4ZJB3yd/w3hkkBHQgJSJdl5J9d1dXB+/eDW/cDtn94axHIe+QeEe116msqWXReq/Hf/1jy45qADKSExk7IJtD/Ov99++bQa/0ZEwtKSLSRSjZ7yvWfAh/udQ7j3/CbXD4ld6d+KRdnPPu+BeZ/Jd8vb1hbKKM5ESG9E5naM80hvZOZ4j/PLBHqu77LyIxp2S/LynfBi9d7TXrDz8JvvmHfep6/M62vbKGBYUlLPt6Oyu2lPHVZq8PwIaSyoYyAYP+uakM6ZlGcmIC1bV1VIVrqQ7XURWuo9p/VDU8ahnYI5UpI/sydVRfhvVOV4uBiLSZkv2+xjmY+0fvrnuhHPj2wzD46HhH1a2VVYVZ6Sf/rzZ7zys3lxGuqyM5MYGkxADJiQGSEgMkJQRIDiaQlFA/bSxYV9JwyeCQXmneTYIK+nJQfpYSv4hERcl+X7VhAbxwiXeZ3jE/gaN/Agm6zKyrauqSwX5ZKZzkJ/5DB+WQqEGBRKQZSvb7sqod8OoN8NkzMOBI75r8rO4/aODerqismv98uYnXFm7krWWbqQ7XkZuWxPiBOfRITyY3LUhuWjI90pLIbfRICaq/gMi+SMle4LPn4JXrICEIE6bBIRcr6e8lyqrCzFm6mdcWbuTLjaVsK6tmW1k1dc386aYlJZAVCpIcTCDZP32QnJhAcjDidWKA5GCAzJQgY/tnc8igHHpnpMT2jYlIh1KyF8+W5fCv/4als8ACcMDJcOgPvPP5Oi+8V6mrc5RW1rDVT/xbd1RTVL7zdUlFjdcxsKa2oRNgVbiOqpqdr6vDdRRX1FAd9m4wNLBHKuMH5nLooBzGD8plaK809RcQ2Yso2cuuilbBvEfhkyehYhv03B8OvQzGnAspmfGOTmKoOlzHovUlzFtVxNxV25i3uohtZd69BXLTkjhkYA6HDsphbP8cUoIBauscdc67LLHOQZ1z1DmHa3gNKYkB0pITSU1KaHhOTUrUTYlEOpmSvTStphIW/dXrub/uYwimwZhzvNp+n5Hxjk7iwDnHii1lzFu1jbmripi3ahurtpZ3yLZTggHSkhJJTU4gLSmRvlkpDOqRxpBeaQzqkcbgnmn0yw7poECknZTspXXrPvGS/sK/QLjS68x31DUwYqqa+Pdxm7ZXsmh9Kc45zIyAGQGDgBnmP9fPA6gK11FWFaa8upay6jAV1bWUVdVSXh2mrDpMeVUt26vCrC+uYOWWMsqrd45YnZQYYGBuKoN7pjU8+malkJ2aRFYoSHYoSGYoqAMCkSYo2Uv0yrfBp0/B3IeheA30GQWTroOR34SAenlLx3LOsXl7FSu2lLFqSxkrt5Q1vF69tbxhwKLGMlISyU4Nkh1KIjvVOwDITAmS5p86SE/2WhDSkxMbWhPSkxNJTUokMyWRrNSg7nIo3Y6SvbRdbRgWvgBv/wa2LIUew2DitXDQOV6PfpFOVlvnWF9cwabtVZRUVFNcXkNJRU3Es9cRsbiihpLyGkorayirqqWiprb1jQOhYALZqUGvxSA1SE5qkj/tPacnJ1Jb56iprSNc56gJ11FT5wjX1lFTW0dNrSNcV0dtHWSmJJLpbyc7lNSwzaxQkKzUIBnJiersKJ1OyV7ar64OvngJ3r4LNn4OWf3hqB/BwRdAMBTv6ER2U1vnvFMGVbXsqApTXh1mR1W44VRCaWWYUv9gobh858FCUXl1w+vmWhQAgglGYiBAYoKRlBDAzNhRVUNlTfPrJATMa1EIeQcA3oFBElmhnfMil2X5y7NDQVKTEqI+UHDOUVZdy5btVWzeUdXwnBgIMDovi/37ZpCUqBszdVdK9rLnnINlr3tJf+2HkNYbjrwKxn8fkjPiHZ1Ih3HOUVFTy47KMAkBIzHBu8VxYoKRGLBmE29lTS0lFTtbH+pbHhqmK6opqQg3zCv1n0sqaqht7qYJQGLAGk5VZEccCGSFgtTU1rFlRxWbt1exZUc1m7dXtdiykZQQ4MD9Mhidn8VB+dkclJ/FsF7pLd6ZcXtlDeuKKyjcVkFhUTmFRRVsLasmKxRsuJFT/c2deqQnkZuWTHYoSKCD+1VUh+tYW1ROTqq3L9mdkr10HOdg1Tte0l8xG1KyvVr+mPOg76h4Ryey16mvjZf4rQrFFdUNBwINpyzqDwwaDhq8ZYkBo1dGMr0ykumZnkyv9IjXEc+VNbUsKCxhQWExCwpLWLiuhO1VYcA7nVHQL5PR+VnkZYfYWFJJYVEFhcVeYi8ur9kl3pRggB5pyZRW1rC9MtzkewoY/kFAMn2zUuiXHSI/J0S/7BT6ZYXolx2ib1YKwSYOMorLq70xJjbVjzXhDTa1elt5w0HR4J5pHDwgm3EDcjh4QDb798nQraRRspfOUjgP3r0HlvwT6sJeZ74x58LosyGjb7yjE5Fm1NU5Vm4t4/PCEj4rLObzwhIWri+hsqaOlGCA/JxU8nNC/iN1l+ceaUkNrRvV4TqKyr0bOW0rq2ZrWVXDjZ22llWzdUcVG0oqWV/stQZEChj0yfQOBPpkJnsdNTeX7VIuKSHA4J7e5ZlDe6UzuGcam7ZX8cmaIj5dU8SWHV7Z1KQExuRnM25g/QFATkPtv77fRU2tdyOpmlpvutqf5xwE/dabYKIRTAjsnE4wElpozelqlOylc5Vt9a7X/+xZ73p9C8CQY73a/gGnQFJqvCMUkVaEa+sorQyTkxrslORWUV3L+pIK1hdXsK7Ify72DgQ2llbSMz2Job3SvUdvL7nn56Q2e5mlc4612yr4ZE2Rn/yLWbyhtKH2nxIMUFPrWjxFEg0z76AjJZhAbloSOaneuBSR41Pk+KcyctKSSE9OZEdVuKETaUutNLedXsARQ3vsUXy7xqpkL7GyZZl3H/4FM6FkDSSlw8gzvF78gyZBQE1tItI5KqprWVBYzCdriikur26opQcTza+p1z+sYbhpgJr6qy382n5VRAtAfStARXUt28qqd2nJKCqvpqY2uhya6o9ZEfmYPnko4wbkdNj77xLJ3symAvcCCcAfnXN3NFo+HbgSqAV2ANOcc4vNbBDwBbDEL/qBc256S/tSsu8C6upgzXte4l/8d6gqhVAuDDnGq/UPPRayB8Q7ShGRdnPOsb0qTFGZd9qiqKyaHVVh0pMTd1566V+KGYurIOKe7M0sAVgKnAgUAnOB85xziyPKZDrnSv3XpwNXOOem+sn+Fedc1L2/lOy7mJoK77z+sn/BV2/Cjo3e/NyhXtIfciwMngQpWfGNU0RkL9ZSsk+MUQwTgOXOuRV+QM8BZwANyb4+0fvSgO53fmFfFQzBqDO9h3OweQmseNNL/POf9W7TawHIO8RL/EOOgbzxENSQqyIiHSFWyT4PWBsxXQgc1riQmV0JXAckAcdFLBpsZp8CpcAtzrm3OzFW6Uxm0PsA73H45RCuhsK5O5P/23fBW3dCQjLkj4eBR8GgoyD/UEhKi3f0IiJ7pVg1458NTHHOXeZPfw+Y4Jy7upny5/vlLzKzZCDdObfVzA4BXgQKGrUEYGbTgGkAAwYMOGT16tWd+I6k01QUwer3YfW73mPDZ+DqIJAI/cZ5iX/gUdD/MA3HKyISoSucsz8CuM05N8WfvhnAOfc/zZQPAEXOud1O4prZbOB651yzJ+V1zr4bqSz17ti3+l1Y9S6s/8S7pt8C0KcA+oz2n/1Heu94RywiEhdd4Zz9XGC4mQ0G1gHnAudHFrD/3965R9lV1Xf885s7j8zkNQmZPEgCBAwi2oA0KLZIFbAivrt0FWq7wJZ2SaVWXNVC7cIlXdZXl+1aVGt9YB+CShUVLSqIWsUHGJBoIiEkJJCEkASSkBeTzOPXP377cs9c79y59565ec0qrAAAEiFJREFUk8md72etvc4+r+/d5/7uPb999t7nt82Wu/vDafXVwMNpex+w292HzOxkYDnwyASVWxxtps2C5a+IBHDkYDT7b/5xLDfeBatvLh0/vS85/hfA/NMj33ea+v+FEFOaCXH27j5oZlcB3yFevbvR3dea2fXAKne/DbjKzC4EBoA9wGXp9POA681skHgt723uvnsiyi0mIZ3T4eSXRSpy8EnYsTaT1sSgv8H+2N/WHs5/ydmRlp4Nc5bF+AEhhJgCKKiOaE2Gh2D3I+H4t6+O0L7b7oeBg7G/57jk/FfG8vizNAZACHFMMxma8YWYWNoKMG95pOe/MbYND8HOB6P5f+uqWK7/djrB4tjuOdDREy0IHd2ZfE+E/e1IqWtmKXXOGLle6Dhqly2EEJWQsxdTh7ZCzMy38AWw8q2x7Zm9Ec9/6yp44pdweD8cOQAHdsLAoUhHDkWLgI8+X/kICl0lxz9rMcxdBnNPjnTcKdGF0DWjedcphBBlyNmLqU13LzzngkjVcIehIzFAcOAQHD6QKgb7Y1lxfR/s3QLrvwMHd47Um7GgVAGYuwwWngEnnKOuBCFEU5CzF6IWzKC9KxJz6z//8H7YvQl2b4yxBLsfifWN34MHtqfPaINFZ8BJ58KJ58KJL1EIYSHEuCBnL8RE0DUTFq2IVM7hA9GV8OiPYfPdcM+/w09uCOe/cEU4/5POhRNeEi0RQghRJxqNL8RkY+CZUiyBzXdHfugwMYjw1Gj2n7MM5pxUyveeoFgCQkxxNBpfiGOJjm5Ydl4kgIF+2LYKNv0oXiXcsznyxdcIATCYdXypEjBrEbR1QKE94gw8mwpl65lU6MgsO9KyEPn2aaGp+QmEOCaRsxdistMxrdSUX8Q9ggnt2RR9/3s2RSVg9ybYcGe8TdCMiSN75kHv0mhJ6D0Bek+E2cX1pdFdIYSYdMjZC3EsYgYz+iItfVHlY4aHYx6BamloEIYHRuaHBtKyeNxAtC7s2wp7H4s3DHb8Ot4yKEYpLNI9J940mN4XgYum96WU8j3z0vo8mNYLbW3N/66EEHL2QrQsbW3Q1knMGN0E3OHgrlQBeLRUETi4Ew4+FaGLD+6C/r2Vz7c26J4bjr/nOOiZm5ZlafYSmHsKtDfpOoSYAsjZCyEawyxmGZwxP8IOj8bQABx6Khz/wSdT2hXbnk274ckNcOieWPehss8qRECivufGxEZ9p0X+uOUamChEDcjZCyGaS6EDZi6MVAvu0P90qSKw51HYtS7SznWw7vZSZcDaYkBi32kxfmDarBS9MC2nzUr5WaV9nTM0CZKYcsjZCyEmF2YRT6C7N57my8ckDB6GpzamCsBDpYrAph9FBMNaaGuHQmfprYNCZ7y5UOhM29vjrYieeTEuYvr8UitGMT+9LyoPeSoOw0PRqnFw58iWj/auqBzNmJ/GQMxXN4bIhZy9EOLYor0LFpweqZzh4VLI4v59KXTxvkj9aXnkYHQtDB1JAxOPpPXitpQfOBRvOWxJXQuV3m5onxZOv6M75kRo74xthc4oZyGtt6eKRf/TGae+a3TdSnTPDcc/c0EsZ8yHjulxvntaDlfOt3eXKg7ZSkRHd6NWEMcYcvZCiNahrS1CDE+bDeMZaXhoMBzzgR3xFH5gV1rujO0Dz0RFYbAfBo/EZEqHnkrbDkcaOhJdCdPnR4vFCedERWHG/BikWHxzoWde6Bx4IvQP7ID9O2JZTI/9NLYNHS6V0doAi5aGbB6L4ypN5NQ1O1N5WBBvU1hb6bwRy+LnpG0+HGl4KOWHytZTGtF60lFqPcm2pBQ649pnHR9pxoKI8dBqDDwTk2/1Px0DV+edGgNTJwA5eyGEGItCezjFmQsm7jNnL66+vxj9tJZuhOGhaE3IVhj2P5HJ74iQzf17My0DMLLVoGxphXDIZpG3trTelvalCsfwUGo9OTKy9aQaVkjjPBalCsDiCOo0a3G0mAwNhO5w5vXQ7PrQQFSwBg5GS05x5spK+eHB0rwXha5Mvtgq01WqkFhbqTJUrFQ9W7FKeR9Ozjw59Gf2lpbZyhnAJTfDaa8e237jgJy9EEIci9QzVqCtMPGVlWq4Z7pQUuvHgZ2wfzvs2wb7Hk9pW4zL2Pi9aC2pl/buiPrY2RNdHsV895xSvq09WmOGDpdaZgb7o1yHnkqtModje7GyU2y1IJMvbjeLAaHdvRFLYv5pqbWpt7StuFx0xnh/s6N/FRP2SUIIIQSEQyyk5n1SCOaZCytPFFWkf19UAIYHUojnjpHhnwvl652t2RXQIHL2QgghJj/T0uuToiEUq1IIIYRoceTshRBCiBZHzl4IIYRoceTshRBCiBZnwpy9mV1kZg+Z2QYzu6bC/reZ2a/M7AEzu9vMTs/suzad95CZvXKiyiyEEEK0AhPi7M2sAHwceBVwOnBp1pknbnb333L3M4GPAB9L554OXAI8H7gI+ETSE0IIIUQNTNST/YuADe7+iLsfAb4IvD57gLvvy6ymgM+Qjvuiux92903AhqQnhBBCiBqYqPfsFwNbMutbgReXH2RmbwfeBXQC52fO/VnZuWPEkRRCCCFEkYl6sq8U1/E3pnpy94+7+ynA3wJ/X8+5ZvYXZrbKzFbt2rUrV2GFEEKIVmKinP1WYGlmfQnweJXjvwi8oZ5z3f1T7r7S3Vf29fXlLK4QQgjROph7jXMp5/kQs3ZgPXABsA34OfBH7r42c8xyd3845V8LvM/dV5rZ84GbiX7644G7gOXuPlTl83YBj47zZcwDnpxCes3QnGp6zdCc7HrN0Jzses3QnGp6zdCc7HrN0DzR3Ss+7U5In727D5rZVcB3gAJwo7uvNbPrgVXufhtwlZldCAwAe4DL0rlrzewW4NfAIPD2ao4+nTPuj/ZmtsrdV04VvWZoTjW9ZmhOdr1maE52vWZoTjW9ZmhOdr1maY7GhE2E4+63A7eXbbsuk//rKud+APhA80onhBBCtC6KoCeEEEK0OHL2tfOpKabXDM2pptcMzcmu1wzNya7XDM2pptcMzcmu1yzNikzIAD0hhBBCHD30ZC+EEEK0OHL2YzDWBD4N6C01s++b2YNmttbMRh2YWKduwcx+YWbfHAetXjP7spmtS+V8SU69q9O1rjGzL5jZtAY0bjSznWa2JrNtrpndaWYPp+WcnHofTdf8SzP7qpn15i1jZt/fmJmb2by8emb2V+k3udbMPpJHz8zONLOfpQmoVplZzaGoR/stN2qXKnoN22Ws/1u9dqmm14hdqlxzHrtMM7N7zWx10nx/2r7MzO5JdvmSmXXm1LspXe+a9NvqyKOX2X+DmR0Yh+s1M/uAma1P3+87xkHzAjO730oTtj2nVs10/oj7dKM2aQh3VxolEa8JbgROJkL4rgZOz6m5CDgr5WcS8QdyaSatdxHxCL45Dlr/CVyR8p1Abw6txcAmoDut3wJc3oDOecBZwJrMto8A16T8NcCHc+r9PtCe8h+uR280zbR9KfHa6aPAvJxlfDnwXaArrc/PqXcH8KqUvxj4Qd7fcqN2qaLXsF2q/d8asUuVMjZklyp6eexiwIyU7wDuAc5J/71L0vZPAlfm1Ls47TPgC3n10vpK4L+BA+NwvW8F/gtoa+C/MprmeuB5aftfAv9Rq2Y6Z8R9ulGbNJL0ZF+dMSfwqRd33+7u96f8fuBBcsb6N7MlwKuBz+TRSVqzCKfw2VTGI+6+N6dsO9BtEVyph+rREyvi7j8Edpdtfj1RMSEt30CNVNJz9zvcfTCt/oyI1pi3jAD/DLyHCmGeG9C7EviQux9Ox+zMqefArJSfTR22qfJbbsguo+nlscsY/7e67VJFryG7VNHLYxd39+KTcUdKTsw38uW0vR67VNRz99vTPgfupUa7jKZnMZvpRwmb1EyV670SuN7dh9Nx9fxXRtNs2C7l92kzMxq0SSPI2Ven0gQ+4zYJj5mdBLyQqDXm4V+IP8hwTh2IVoxdwOdSc9NnzGx6o2Luvg34J+AxYDvwtLvfMQ7lBFjg7tvT52wH5o+TLsCfAt/KK2JmrwO2ufvq/EUC4FTgpanp7//M7Oyceu8EPmpmWwg7XduISNlvObddqvw3GrZLVnM87FJWxtx2KdPLZZfUXPwAsBO4k2ih3JupNNV1LyvXc/d7Mvs6gD8Bvp1T7yrgtuJvpx5G0TsF+MPUDfItM1s+DppXALeb2Vbimj9Uh2T5ffo4ctikXuTsq1PTJDwNCZvNAL4CvNNHTu9br85rgJ3uft94lIt4Cj8L+Dd3fyFwkGiKbbR8c4gnvWVEuOPpZvbH41HQZmFm7yWiNd6UU6cHeC9w3VjH1kE7MIdoUnw3cEt6QmiUK4Gr3X0pcDWpRacexuu3PJZeHrtkNZNGLrtUKGMuu1TQy2UXdx9y9zOJp+0XAc+rdFijemb2gszuTwA/dPcf5dA7D3gzcEOtGjWUrwvo94hQ92ngxnHQvBq42N2XAJ8DPlaL1ij36ab5l0rI2Ven3gl8aiLVhL8C3OTut+aU+13gdWa2mehmON/MPp9DbyuwNVNz/zLh/BvlQmCTu+9y9wHgVuB3cuhl2WFmiwDSsuZmutEws8uA1wBvSc2TeTiFqOSsTvZZAtxvZgtzaG4Fbk3NjPcSTwk1D/qrwGWETQD+h3AMNTPKb7lhu4z238hjlwqauewyShkbtssoernsUiR1wf2AqIT0pq40aPBeltG7KJX9fUAf0Redp3wvB54DbEg26TGzDTnLt5X4XgG+CqzIWcZXAWdk7o1fovZ72W/cp4kn/dw2qRU5++r8HFieRkx2ApcAt+URTLX9zwIPuntNtcJquPu17r7E3U9K5fueuzf85OzuTwBbzOy5adMFxLwEjfIYcI6Z9aRrv4DolxwPbiPNoZCWX88jZmYXEdMrv87dD+UsG+7+K3ef7+4nJftsJQZjPZFD9mvEjQIzO5UYQJlnIo3Hgd9L+fOBh2s9scpvuSG7jKaXxy6VNPPYpco1N2SXKnp57NJn6Y0FM+smKtwPAt8H3pQOq8culfTWmdkVwCuBS4v94jn07nP3hRmbHHL3mka6j1Y+MjYhvsv1Ocv4IDA72RfgFdR4LxvlPv0WGrRJQ3iTRv61SiJGnK4n+rzeOw565xJNNb8EHkjp4nEq68sYn9H4ZwKrUhm/BszJqfd+4s+3hhhp29WAxheIPv8B4ub8Z0Sf113EjfAuYG5OvQ3EGI2iXT6Zt4xl+zdT32j8SmXsBD6fvsv7gfNz6p0L3Ee8aXIP8Nt5f8uN2qWKXsN2qeX/Vo9dqpSxIbtU0ctjlxXAL5LmGuC6tP1kYiDdBqK1oKb/YRW9QeK+WCz3dXn0yo6pZzT+aOXrBf4X+BXwU+KpPK/mG5PeauJp/+RaNTPaL6M0Gr8hmzSSFEFPCCGEaHHUjC+EEEK0OHL2QgghRIsjZy+EEEK0OHL2QgghRIsjZy+EEEK0OHL2QoijhsWMc3XNHCaEqB85eyHEs5jZZjN7xswOZNK/Hu1yCSHy0T72IUKIKcZr3f27R7sQQojxQ0/2QogxMbPLzezHZnaDmT1tZuvM7ILM/uPN7DYz221mG8zszzP7Cmb2d2a20cz2m9l9Zpadc+JCM3vYzPaY2cdzTuwjhKiAnuyFELXyYmJipHnAHwC3mtkyd99NhOJdS8xseBpwp5k94u53EZOkXEop9PQKIBvj/jXA2cQ84fcB36CO6VKFEGOjcLlCiGdJs3LNI+KeF3k3EU//H4HFnm4aZnYvMSXpD4j48r3uvj/t+yCwyN0vN7OHgPe4+29M8mFmDrzU3e9O67cA97t7PfOECyHGQM34Qohy3uDuvZn06bR9m498OniUeJI/HthddPSZfYtTfikxYcpoZGebOwTMyFd8IUQ5cvZCiFpZXNaffgIxFevjwFwzm1m2b1vKbyHmkBdCHCXk7IUQtTIfeIeZdZjZm4HnAbe7+xbgJ8AHzWyama0gps+9KZ33GeAfzGy5BSvM7LijcgVCTFE0QE8IUc43zGwos34n8HViXvXlwJPADuBN7v5UOuZS4JPEU/4e4H3ufmfa9zGgC7iDGA+wjpgXXAgxQWiAnhBiTMzscuAKdz/3aJdFCFE/asYXQgghWhw5eyGEEKLFUTO+EEII0eLoyV4IIYRoceTshRBCiBZHzl4IIYRoceTshRBCiBZHzl4IIYRoceTshRBCiBbn/wEm/xCqBhIeuAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the training and the validation learning curve\n",
    "plt.figure(figsize=(8,5))\n",
    "plt.plot(train_avg_loss_list, label='Training loss')\n",
    "plt.plot(eval_avg_loss_list, label='Evaluation loss')\n",
    "plt.xlabel('Epoch', size=12)\n",
    "plt.ylabel('Loss', size=12)\n",
    "plt.title('biGRU with pre-trained vectors learning curves')\n",
    "plt.xticks(ticks=range(0, 41, 2))\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAFZCAYAAAB0RP9xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5xU1d3H8c+XBRQEpVsARWwx+kSDRo0l9oY90dg1PhpMNDGWGBNrNJJYElvUKLE3jC1WfLBEjFixYBRNFCtVRYoCKiz7e/64d3GcbBmZnbJzv29f83Lm3HLOLLu/OfM7556riMDMzLKhQ6UbYGZm5eOgb2aWIQ76ZmYZ4qBvZpYhDvpmZhnioG9mliEdK90AM7Nqs3DG20XNZe/UZ7Daqi1tzT19M7MMcU/fzCxfw6JKt6BkHPTNzPJFQ6VbUDIO+mZm+Roc9M3MMiNquKfvgVwzswxxT9/MLJ/TO2ZmGVLD6R0HfTOzfJ6yaWaWITXc0/dArplZhrinb2aWr4YHct3TtzYlqYuk+yTNkXR7Eec5UNJDbdm2SpD0oKRDK90O+3oiGop6VDMH/YySdICk5yXNlTQtDU6bt8Gp9waWB3pHxD5LepKIuDkidmiD9nyFpK0khaS78srXS8vHFHie30q6qbX9ImLniLh+CZtrldLQUNyjijnoZ5Ck44GLgN+TBOiVgcuBPdrg9KsAb0REfRucq1Q+AjaV1Dun7FDgjbaqQAn/fVnV8S9lxkhaDjgLODoi7oqIeRGxMCLui4gT032WknSRpKnp4yJJS6XbtpI0WdIJkj5MvyUclm47Ezgd2Df9BnF4fo9Y0qC0R90xff0jSW9L+lTSO5IOzCkfm3PcppLGpWmjcZI2zdk2RtLvJD2ZnuchSX1a+DEsAO4G9kuPrwN+CNyc97O6WNIkSZ9IekHSFmn5TsDJOe/z5Zx2DJf0JDAfGJyWHZFu/4ukO3LOf66kRyVV7drrmRUNxT2qmIN+9nwXWBr4ewv7nAJsAqwPrAdsBJyas30FYDmgP3A4cJmknhFxBsm3h79FRLeIuLqlhkhaBrgE2DkiugObAuOb2K8X8EC6b2/gAuCBvJ76AcBhQD+gM/DLluoGbgAOSZ/vCEwApubtM47kZ9ALuAW4XdLSEfF/ee9zvZxjDgaGAd2B9/LOdwLwrfQDbQuSn92hEVHUDTusBBoWFfeoYg762dMbmNFK+uVA4KyI+DAiPgLOJAlmjRam2xdGxChgLrDWEranAVhXUpeImBYRE5rYZxfgzYi4MSLqI2Ik8G9gt5x9ro2INyLiM+A2kmDdrIh4CuglaS2S4H9DE/vcFBEfp3X+CViK1t/ndRExIT1mYd755gMHkXxo3QT8PCImt3I+qwT39K2GfAz0aUyvNGMlvtpLfS8tW3yOvA+N+UC3r9uQiJgH7Av8BJgm6QFJ3yigPY1t6p/zevoStOdG4GfA1jTxzSdNYb2eppRmk3y7aSltBDCppY0R8RzwNiCSDyerRh7ItRryNPA5sGcL+0wlGZBttDL/nfoo1Dyga87rFXI3RsToiNgeWJGk9/7XAtrT2KYpS9imRjcCRwGj0l74Ymn65SSSXH/PiOgBzCEJ1gDNpWRaTNVIOprkG8NU4FdL3nSzJeOgnzERMYdksPUySXtK6iqpk6SdJZ2X7jYSOFVS33RA9HSSdMSSGA98T9LK6SDybxo3SFpe0u5pbv8LkjRRUwnRUcCa6TTTjpL2Bb4J3L+EbQIgIt4BtiQZw8jXHagnmenTUdLpwLI52z8ABn2dGTqS1gTOJknxHAz8SlKLaSirEKd3rJZExAXA8SSDsx+RpCR+RjKjBZLA9DzwL+AV4MW0bEnqehj4W3quF/hqoO5AMrg5FZhJEoCPauIcHwO7pvt+TNJD3jUiZixJm/LOPTYimvoWMxp4kGQa53sk345yUzeNF559LOnF1upJ02k3AedGxMsR8SbJDKAbG2dGWRWp4fSOPHHAzOyrPn95VFGBcen1hlbtNFz39M3MMsQLrpmZ5avyvHwxHPTNzPJVeV6+GA76Zmb53NM3M8uQKl9KoRjtJeh7ipGZFapqZ85Ug/YS9PliwqOVboJVkaXW2RaAFXqsXeGWWDWZPvv1tjlRmdI76QqvzwNTImJXSasCt5Is8vcicHBELEiv5bgB2IDkWpV9I+Ld9By/IVm8bxFwTESMbqlOT9k0M8tXvouzfgHkflKdC1wYEWsAs0iCOen/Z0XE6sCF6X5I+ibJEuHrADsBl6cfJM1y0Dczy1eGZRgkDSBZQfaq9LWAbYDGey5cz5drZO2Rvibdvm26/x7ArRHxRbqsyESSpdCb5aBvZpavyJ6+pGFKbkfa+BjWRC0XkSwp0vgp0RuYnbOC7WS+XEm2P+kyIOn2Oen+i8ubOKZJ7Sanb2bWXkTECGBEc9sl7Qp8GBEvSNqqsbipU7WyraVjmuSgb2aWr/QXZ20G7C5pKMmd7JYl6fn3kNQx7c0P4MslzScDA4HJ6eJ9y5EsUthY3ij3mCY5vWNmlidiUVGP1s8fv4mIARExiGQg9h8RcSDwGLB3utuhwD3p83vT16Tb/5HeZvNeYL/0vtarAmsAz7VUt3v6Zmb5KrcMw0nArZLOBl4CGu8zfTXJMtwTSXr4+wFExARJtwGvkdz/4eho5VPHQd/MrIIiYgwwJn3+Nk3MvomIz4F9mjl+ODC80Poc9M3M8nntHTOzDPEqm2ZmGeKevplZhtRwT99TNs3MMsQ9fTOzfE7vmJllSA2ndxz0zczyOeibmWVIDad3PJBrZpYh7umbmeVzesfMLENqOL3joG9mlq+Ge/rO6ZuZZYh7+mZm+ZzeMTPLkBpO7zjom5nlc9A3M8uQiEq3oGQ8kGtmliHu6ZuZ5XN6x8wsQxz0zcwyxFM2zcwypIZ7+h7INTPLEPf0zczy1fCUTQd9M7N8NZzecdA3M8tXw0HfOX0zswxxT9/MLJ+nbJqZZUc0eCDXzCw7ajin76BvZpavhtM7Hsg1M8sQ9/TNzPI5p29mliHO6ZuZZYiDvplZhtTw2jseyDUzyxD39M3M8jm9Y2aWIZ69Y2aWIb44y8zMaoF7+mZm+ZzeMTPLjvBArplZhrinb2aWIR7INTOzWuCevplZPqd3zMwyxAO5ZmYZ4p6+mVmGeCDXzMxqgXv6VWLRogb2/9U59OvVg0tPOYqRo8Zw0/2PMWn6Rzx+3Xn0XLYbAONefYNfnHMF/fv1AWDbTdbnJz8cCsBOR55K1y5LU9ehA3V1Hbj1/F9X7P1Y21lt9UFcee0Fi1+vsspAzvvDn3nyiWc574LfsswyXZk0aQpH/fhE5n46j06dOnH+Rb9lvfXXpSEaOO3Xv+epseMq+A7aIad3rNRufuAxVh2wAvPmfw7A+t9Yje9t+D8cftqF/7XvkLVX59JTjmryPFefdeziDwirDW9NfJfttvg+AB06dGD862N48P5HuOr6izjztPN5+slx7H/Q9znqmMM5b/glHHToPgBsvdke9OnTi5vvGMFOW+9D1PCNQdpaLV+RW9b0jqSlyllfezF9xiz++cKrfH+7zRaXrT14IP379a5gq6wabbHlJrz7ziQmT5rKaquvytNPJj34xx97il132x6ANddajScefwaAGTNm8smcT1j/2+tWrM3tUkMU96hiZQn6kjaS9ArwZvp6PUl/Lkfd7cF519zB8YfsRQepoP1f/s877H3ccH76u0uZ+P7ULzdIHHnmn9n3l3/gjofGlqi1Vkl7/mAod9/5AAD/fv1Ndhy6DQC77bkjK/VfEYAJr/6bnYZuQ11dHSuv0p9vrb8OKw1YoWJttupSrvTOJcCuwN0AEfGypK3LVHdVe/z5V+i1XDe+udrKjHv1jVb3X3vwQEZf+Tu6dlmaJ154lWPPvZL7LzsTgBt+fwL9evXg49mfcuSZlzCo//JsuM4apX4LViadOnVih523YfiZScrvuJ+dwtnnnsLxvzqKhx78BwsWLgRg5E13scZaqzF6zO1MnjSV558dT339oko2vf2p8t56McoV9DtExHv6ak+2xd9CScOAYQBXXnklh262WgmbVznj//0WY8a9wtgXJ/DFwnrmzf+M31x0LX849rAm9+/Wtcvi51tssC7DR9zKrE/m0nPZbvTr1QOA3j26s83G6/Hqm+866NeQbbbfgldefo0ZH30MwMQ332G/7x8BwODVBrHdDlsCsGjRIs44+ZzFx903+hbeeeu98je4PavhKZvlCvqTJG0EhKQ64OdAi93aiBgBjGh8+cWER0vcxMr4xUF78ouD9gSSmTnX3/NIswEfYMasOfTusSySeOXNd2mIoEf3ZZj/+RdEBMt0WZr5n3/B0y+/zpH7DC3X27Ay2OsHuyxO7QD06dOLGTNmIonjTvwJN1z7NwC6dFkaScyf/xnf22pT6hct4o3/vFWpZrdP7ukX7ackKZ6VgQ+AR9Iya8bNDzzGtX9/mI9nf8Lexw1n8yHrcObRB/Hw0y9x2+gnqOvQgaU6d+K84/8XScyc/SnHnnslAIsaGth5iw3ZfMg6FX4X1la6dFma7229KSced8bisj333oXDjjgAgFH3PczIm+4CoE/fXoy88yoaGhqYPu1Dfn7kSRVpc3sWJQ76kpYG/gksRRKH74iIMyTdDGwILASeA46MiIVK0iQXA0OB+cCPIuLF9FyHAqempz47Iq5vse52Mo2rZnv6tmSWWmdbAFbosXaFW2LVZPrs1wEKmxHRgk+P3a2owNj9ovtabEMaxJeJiLmSOgFjgV8AvYAH091uAf4ZEX+RNJQkQzIU2Bi4OCI2ltQLeJ7kgyKAF4ANImJWc3WXpacv6a9pg74iIoaVo34zs6+lxD39SHrbc9OXndJHRMSoxn0kPQcMSF/uAdyQHveMpB6SVgS2Ah6OiJnpMQ8DOwEjm6u7XOmdR3KeLw3sBUwqU91mZl9PGS7OSsc3XwBWBy6LiGdztnUCDibp/QP056sxc3Ja1lx5s8oS9CPib7mvJd0IPFyOus3MvrYie/q5sw9TI9LJKYtFxCJgfUk9gL9LWjciXk03X06S2nmi8ZRNVBMtlDerUsswrAqsUqG6zcxaVmTQz5t92Nq+syWNIUnLvCrpDKAvcGTObpOBgTmvBwBT0/Kt8srHtFRfua7InSVpZvqYTdLLP7kcdZuZVRtJfdMePpK6ANsB/5Z0BLAjsH/EVy4WuBc4RIlNgDkRMQ0YDewgqaeknsAOaVmzSt7TT0ep1wOmpEUN0U6mDJlZNpUhRK0IXJ/m9TsAt0XE/ZLqgfeAp9OLWe+KiLOAUSQzdyaSTNk8LG3nTEm/AxqXUT2rcVC3OSUP+hERkv4eERuUui4zszZR+tk7/wK+3UR5kzE57Sgf3cy2a4BrCq27XKtsPidpSJnqMjMrTg2vslnSnr6kjhFRD2wO/FjSW8A8khHniAh/EJiZlVGp0zvPAUOAPUtcj5lZmyn1MgyVVOqgL4CI8GpPZtZ+OOgvsb6Sjm9uY0Rc0Nw2M7OKqd2VlUse9OuAbrTBAkhmZuXi9M6Sm5bOMTUzsypQlpy+mVm74p7+Etu2xOc3M2t7zukvmdYuBzYzq0a1nNMv1xW5ZmZWBSq1tLKZWfVyesfMLDtqOb3joG9mls89fTOz7IgaDvoeyDUzyxD39M3M8tVwT99B38wsTy2ndxz0zczyOeibmWVHLff0PZBrZpYh7umbmeWp5Z6+g76ZWR4HfTOzLInavRXIEuX0Ja0iaUBbN8bMzEqroKAv6SZJ302fHwL8B3hD0o9K2DYzs4qIhuIe1azQnv4OwAvp8xOA7YFNgJNL0Sgzs0qKBhX1qGaF5vQ7R8QCSSsBfSPiCQBJK5auaWZmlVHtvfViFBr0X5Z0IjAIeAAg/QD4pETtMjOrmPBALkcA3wF6AKemZZsBI0vRKDMzK42CevoR8Sbww7yy24HbS9EoM7NKymR6J52l06qIuKHtmmNmVnnVPhhbjJZ6+j8u4PgAHPTNrKZE7d4it/mgHxFblLMhZmbVopZ7+gVfkSupp6T9JR2fvl4hncFjZmbtRKFX5G4BvAEcDpyZFn8DuKJE7TIzqxhfnAUXAwdGxEOSZqVlzwAblaZZZmaVk8mcfp5VI+Kh9Hnjj2MB0Kntm2RmVlnV3lsvRqE5/X9L2i6vbBvg1TZuj5mZlVChPf1fAvdIugfoIukyYK/0YWZWU2p5GYZCr8h9UtK3gYNJ5uVPA74bEe+VsnFmZpWQySty80XEJOD3knpGxKxWDzAza6caarinX+iUzeUkXStpPjBD0vz0dY8St8/MrOwiVNSjmhU6kHsNyQqbGwM90/8vm5abmVk7UWh6ZxtgpYj4LH39Srog25TSNMvMrHI8ZRMmAivnlQ0A3mzb5piZVV5EcY9qVujSyqOBhyRdD0wCBgKHADeWtnlmZuVXyz39r7O08vvA1jmvJwFbtnmLzMwqrJZn73hpZTOzDCl4nr6ZWVZU+7TLYhQ6T38lSbdJ+kDSotxHqRtoZlZutTyQW+jsnSvSfXcB5pIsqfwAcFSJ2mVmVjENoaIe1azQ9M5mwCoRMVdSRMQLkg4DxgJXlq55ZmbWlgoN+otI1s8HmCOpLzCHZK6+mVlNqeWcfqFBfxywM3AP8DBwCzAfeLFE7TIzq5hqz8sXo9CgfzBf5v+PAU4CugEXlKJRTVlqnW3LVZW1I9Nnv17pJlgNqva8fDEKXU9/Zs7z+cAZJWuRmVmFZTK9I+n0Qk4QEWe1XXOa17Fz/3JUY+1E/YJkrb+FM96ucEusmnTqM7jSTah6LfX01yjg+BrOfJlZVmUyvRMRB5ezIWZm1aLUvVlJA0luPbsC0ACMiIiLc7b/Ejgf6BsRMyQJuBgYSjKJ5kcR8WK676HAqemhZ0fE9S3V7WUYzMzylKGnXw+cEBEvSuoOvCDp4Yh4Lf1A2J5kkctGO5NkX9YguYnVX4CNJfUiGWPdkOSz6gVJ97Z0S9tCr8g1M8uMUt8uMSKmNfbUI+JT4HWgceDyQuBXfPULxx7ADZF4BughaUVgR+DhiJiZBvqHgZ1aqttB38ysgiQNAr4NPCtpd2BKRLyct1t/kuXsG01Oy5orb5bTO2ZmeRqKPF7SMGBYTtGIiBjRxH7dgDuBY0lSPqcAOzR1yibKooXyZhUc9CVtDewHLB8Re0oaAnSPiMcLPYeZWXsQTcbSr3F8EuD/K8jnktSJJODfHBF3SfofYFXg5WTclgHAi5I2IunBD8w5fAAwNS3fKq98TEv1Frq08lHA1SRfIxrvnrUAGF7I8WZm7UlDFPdoTTob52rg9Yi4ACAiXomIfhExKCIGkQT0IRExHbgXOESJTYA5ETGN5Fa2O0jqKaknybeE0S3VXWhP/wRgu4h4W9IJadnrwNoFHm9mZl/ajGR5m1ckjU/LTo6IUc3sP4pkuuZEkimbh0GyWoKk35GsjwZwVu4KCk0pNOh3B95Lnzd+jnXky5U3zcxqRkOR6Z3WRMRYms7H5+4zKOd5AEc3s981wDWF1l3o7J2xwC/zyo4GnM83s5oTqKhHNSu0p/9z4H5JPwa6S5pA0ssfWrKWmZlVSLGzd6pZoatsTpG0AfBdYGWSAd2nI8L3yDWzmlPtvfViFDxlMyIagCfTh5mZtUMFBX1J79DMhP+I8FqmZlZTMp/eAY7Ie70iSZ5/ZNs2x8ys8jIf9CPi0fwySY+SzB29qK0bZWZWSc7pN+0zwKkdM6s5DbUb8wvO6effOrErsAvwUJu3yMzMSqbQnn7+rRPnAZcB17Vpa8zMqkCpr8itpFaDvqQ6koX5b4uIz0vfJDOzyqrlm3+3ugxDegHWnx3wzSwrGop8VLNC1955QJKXXDAza+cKzel3AO6SNJZkCYbF334i4n9L0TAzs0ppUIZz+qk3gfNL2RAzs2pRyzn9FoO+pP0jYmREnFauBpmZVVq15+WL0VpO/8qytMLMrIo0qLhHNWst6Fd5883M7OtoLadfJ2lrWgj+EfGPtm2SmVllZfnirKVI7tje3E8g8Po7ZlZjMjuQC8zzevlmljXVnpcvRqEXZ5mZWQ1oradfw593ZmZNq+Upmy0G/YjoXq6GmJlViyzn9M3MMqeWc/oO+mZmeWo5veOBXDOzDHFP38wsTy339B30zczyhHP6ZmbZ4Z6+mVmG1HLQ90CumVmGuKdvZpbHF2eZmWWIL84yM8sQ5/TNzKwmuKdvZpanlnv6DvpmZnk8kGtmliEeyDUzy5BaTu94INfMLEPc0zczy+OcvplZhjTUcNh30Dczy1PLOX0HfTOzPLXbz/dArplZprinb2aWx+kdM7MM8cVZZmYZUsuzd5zTNzPLEPf0q8zEN57h07lzWbSogfr6ejb57lAAjj7qMI466jDq6+t58MFH+fVvhtOpUyf+cvm5bLDBt2hoCI4//nQe/+fTFX4H1pYWLVrEvocfQ7++fbj8/DM57Q8XMuHfbxIRDBrYn+GnnEDXrl2YOv0DTvv9hcycPYfllu3OOaefyAr9+jJ1+gcce/LZi3+fDth7d/bda5dKv62qV7v9fAf9qrTd9vvw8cezFr/eastN2X23Hfn2kO1YsGABffv2BuCIww8A4NtDtqNv397cf99NbPLdoUTU8q9sttx0+z0MHrQyc+fNB+CkY4bRbZllADjvkhHccud9HHHwD/njpVex+07bssfQ7Xn2hfFcdMV1nHP6ifTt3YubrvgTnTt3Zv78z9jz4J+w9eab0C/9HbKm1fJArtM77cCRRx7CeedfxoIFCwD46KOPAVh77TX5x2NjF5fNmf0JG26wXsXaaW1r+ocf8c+nnuMHu+24uKwx4EcEn3/xBUoHHN9653023nB9ADYash6PPZF84+vUqROdO3cGYMHChTS4Q1CQBqKoRzUrW9BX4iBJp6evV5a0Ubnqby8iggdHjeTZZx7kiMMPBGCNNQaz+eYb8dTY+/jHI3csDuz/+tdr7L7bjtTV1TFo0ECGDPkfBgxcqZLNtzZ07sVXcvxRhyN99c/01OEXsOVuB/DOe5M5YO/dAVhrjcE8POZJAB55/Cnmzf+M2XM+AWDaBx+x1yE/Zbu9DuHwA/dxL78AUeSjmpWzp3858F1g//T1p8BlZay/XfjeVnuy0cY7setuB/HTn/6ILTbfmI4d6+jRYzk23Xw3Tvr12Yy85QoArr3uVqZMnsazzzzIBX86k6effp76+voKvwNrC2OefJZePXuwzjfW+K9tZ59yPI/dcxODBw3k/x79JwC/PPoInn/pFfb+0dE8P/4Vlu/bm7q6OgBWXL4vf7/hL4z629Xc8+AjzJg567/OadlRzpz+xhExRNJLABExS1Ln5naWNAwYBnDllVeWqYmVN23aB0CSrrnnngf5znfWZ8rkadx994MAjHt+PA0NDfTp04sZM2Zywom/XXzsE4/fw8SJ71Si2dbGXvrXa4wZ+wxPPD2OLxYsZN68+Zx05nmce8avAKirq2Onbb/HtbfcyV677EC/vr25+A+nATB//mc8MmYs3bst85Vz9uvbm9VXXYUXX36VHbbeouzvqT1xTr9tLJRUR/rtR1JfWvjZRsSIiNgwIjYcNmxYudpYUV27dqFb+ofatWsXtt9uSyZM+A/33DuarbfeDEhSPZ07d2bGjJl06bI0Xbt2AWC7bbegvr6e119/s2Ltt7Zz3E8P49G7b+KhO6/n/DN/zUYbrMc5p5/I+5OnAkkacMyTz7LqKgMAmDV7Dg0NyZ/TX2/8G3vtsgOQjAt8/sUXAMz55FNeeuU1Bq08oALvqH2p5Zx+OXv6lwB/B/pJGg7sDZxaxvqr3vLL9+WO268GoGPHOm699W5GPzSGTp06cdVf/8T4lx5lwYKF/O/hxwLQr18fRj1wCw0NDUydMp1DDzumks23EosITj77T8ybN5+IYK3VV+W0E38GwLiX/sVFV1yHJDZYb11OPeEoAN5+dxLnX/pXJBER/Gj/77PmaqtW8m20C9Udtoujck7vk/QNYFtAwKMR8XqBh0bHzv1L1zBrd+oXTAFg4Yy3K9wSqyad+gyGJL4U5ReD9isqMF787q1Vu5BDOWfvrAa8ExGXAa8C20vqUa76zcyqhaRrJH0o6dW88p9L+o+kCZLOyyn/jaSJ6bYdc8p3SssmSvp1IXWXM6d/J7BI0urAVcCqwC1lrN/MrCBR5H8FuA7YKbdA0tbAHsC3ImId4I9p+TeB/YB10mMul1SXjpFeBuwMfBPYP923ReXM6TdERL2k7wMXR8SfG2fymJlVk1LP3omIf0oalFf8U+CciPgi3efDtHwP4Na0/B1JE4HGa5wmRsTbAJJuTfd9raW6yz17Z3/gEOD+tKxTGes3MytIhWbvrAlsIelZSY9L+k5a3h+YlLPf5LSsufIWlTPoH0ZycdbwiHhH0qrATWWs38ysLCQNk/R8zqOQeecdgZ7AJsCJwG2SRNMD09FCeauVlEVEvAYck/P6HeCcctVvZlaoYuc0RsQIYMTXPGwycFckUyqfk9QA9EnLB+bsNwCYmj5vrrxZJQ/6kl6hhZ9hRHyr1G0wM/s6KnSB1d3ANsAYSWsCnYEZwL3ALZIuAFYC1gCeI+npr5FmTaaQDPYe0Fol5ejp71qGOszM2kypB3IljQS2AvpImgycAVwDXJNO41wAHJr2+idIuo1kgLYeODoiFqXn+RkwGqgDromICa3VXfKgHxHvlboOM7O2VOC0yyU/f8T+zWw6qJn9hwPDmygfBYz6OnWX8+KsTSSNkzRX0gJJiyR9Uq76zcysvPP0LyXJOd0ObEgydXP1MtZvZlaQWl5ls6y3S4yIiZLq0nzUtZKeKmf9ZmaFKHV6p5LKGfTnp+vnj0/XlJgGLNPKMWZmZVfLPf1yXpx1cFrfz4B5JPNLf1DG+s3MCtIQUdSjmpVjnv7KEfF+ziyez4EzS12vmZn9t3L09O9ufCLpzjLUZ2ZWlFq+MXo5cvq560MMLkN9ZmZFqfZbHhajHEE/mnluZlaVPHunOOulF2EJ6JJzQZaAiIhly9AGMzOjPMsw1JW6DjOztlTLUzbLenGWmVl74Jy+mVmGOKdvZpYhtZzeKecVuWZmVmHu6ZuZ5dU5UpYAAAhLSURBVIkqX0qhGA76ZmZ5PJBrZpYhtZzTd9A3M8tTy7N3PJBrZpYh7umbmeVxTt/MLEM8e8fMLENqeSDXOX0zswxxT9/MLE8tz95x0Dczy+OBXDOzDPFArplZhtRyT98DuWZmGeKevplZHg/kmpllSINz+mZm2VG7Id9B38zsv3gg18zMaoJ7+mZmeWq5p++gb2aWxxdnmZllSC339J3TNzPLEPf0zczy+OIsM7MMcU7fzCxDajmn76BvZpanlnv6Hsg1M8sQ9/TNzPI4vWNmliGevWNmliG1vLSyc/pmZhninr6ZWR6nd8zMMqSW0zsO+mZmedzTNzPLkFru6Xsg18wsQ9zTNzPL4/ROFahfMKXSTbAq1KnP4Eo3wWpQLad32kvQV6UbUC0kDYuIEZVuh1UX/160rVru6Tun3/4Mq3QDrCr596INRTQU9ahmDvpmZhnSXtI7ZmZl41U2rZo4b2tN8e9FG6rlm6iolt+cmdmSGNBr3aIC4+SZr1bt5BPn9M3MKkDScZImSHpV0khJS0taVdKzkt6U9DdJndN9l0pfT0y3D1rSeh30q4Ck3pLGp4/pkqbkvO5c6fZZ+UlalPM7ML6lP3JJgyS9Wr7W1b6IKOrRGkn9gWOADSNiXaAO2A84F7gwItYAZgGHp4ccDsyKiNWBC9P9lohz+lUgIj4G1geQ9FtgbkT8MXcfSSJJx1X3fDBrK59FxPqVbkRWlenirI5AF0kLga7ANGAb4IB0+/XAb4G/AHukzwHuAC6VpFiC/Lx7+lVM0urpV78rgBeBgZJm52zfT9JV6fPlJd0l6XlJz0napFLtttJIe/RPSHoxfWzaxD7rpP/+4yX9S9IaaflBOeVXSqor/ztoP6LI/yQNS/8WGx9fuY4iIqYAfwTeJwn2c4AXgNkRUZ/uNhnonz7vD0xKj61P9++9JO/NPf3q903gsIj4iaSW/r0uAc6LiGfSVMD9wLplaJ+VRhdJ49Pn70TEXsCHwPYR8XkazEcCG+Yd9xPg4oi4OU0N1klaG9gX2CwiFkq6HDgQuKE8b6X9KXaCS3p1dLMzqiT1JOm9rwrMBm4Hdm7qVI2HtLDta3HQr35vRcS4AvbbDlgryQIB0FNSl4j4rHRNsxJqKr3TieRr/frAImDNJo57GjhF0gDgroh4U9K2wAbAuPT3owvJB4hVznYkH+YfAUi6C9gU6CGpY9qbHwBMTfefDAwEJqedv+WAmUtSsYN+9ZuX87yBr37iL53zXMBGEbGgLK2ySjgO+ABYjyQ1+3n+DhFxi6RngV2A0ZKOIPnduD4iflPOxrZnZbg4631gE0ldgc+AbYHngceAvYFbgUOBe9L9701fP51u/8eS5PPBOf12JR3EnSVpDUkdgL1yNj8CHN34Iu0NWm1ZDpiW/h4cTDLj4yskDQbejohLSALFt4BHgb0l9Uv36SVplfI1u/0p9eydiHiWZED2ReAVklg8AjgJOF7SRJKc/dXpIVcDvdPy44FfL+l7c0+//TkJ+D+SnsJrwFJp+dHAXyQdRvLv+hg5HwJWEy4H7pS0D8m/77wm9tkXOCidETIdOCsiZko6FXgo7SwsJPndeK9M7W53yjF7JyLOAM7IK34b2KiJfT8H9mmLen1FrplZnp7dVi8qMM6aO9FX5JqZWeU5vWNmlserbJqZZUgtp70d9M3M8tTyPXKd0zczyxAHfas66Roz0bjshKQHJR26BOdZWdJcrzNjX1exa+9UMwd9W2KS3pX0WRpYP5B0raRubV1PROwcEdcX2J7tco57PyK6RcSitm6T1baGiKIe1cxB34q1W0R0A4YA3wFOzd2ohH/PrF0p9RW5leQ/RmsT6VKxDwLrShojabikJ4H5wGBJy0m6WtK09CYxZzemXSTVSfqjpBmS3iZZN2ax9HxH5Lz+saTXJX0q6TVJQyTdCKwM3Jd+8/hVE2milSTdK2mmkjsQ/TjnnL+VdJukG9LzTpC0Yc72k9J2fyrpP+kiZlajnN4xa4WkgcBQ4KW06GBgGNCd5HL/64F6YHXg28AOQGMg/zGwa1q+IcmCUs3Vsw/JzSQOAZYFdgc+joiDSZam2C1N6ZzXxOEjSVYrXCmt4/d5wXt3koWuepCsW3NpWudawM+A70REd2BH4N3Wfypm1cdB34p1t5Ibu4wFHgd+n5ZfFxET0iVie5GsFX5sRMyLiA9Jbvm2X7rvD4GLImJSRMwE/tBCfUeQ3DdgXCQmRkSra8ikH0qbAydFxOcRMR64iuTDqdHYiBiVjgHcSLKaJSTLGC8FfFNSp4h4NyLeaq1Oa79qOb3jefpWrD0j4pHcgnTN9kk5RauQrAU/LWe9/w45+6yUt39LQXwgsCQBdyVgZkR8mldP7k1Ipuc8nw8sna5tPlHSsSTfMNaRNBo4PiKmYjWp2gN3MdzTt1LJ/auZBHwB9ImIHulj2YhYJ90+jSSYN1q5hfNOAlYroM58U4Fekrrn1TOlhWO+PHHELRGxOckHWFDEjamt+kWRj2rmnr6VXERMk/QQ8CdJpwFzSW4TNyAiHgduA46RdD/JcsEtrRV+FXCBpLEka5GvBixMUzwfAIObacMkSU8Bf5D0S5K7Th0OHNRa+9Ocfn/gSZIbl3yGO0w1rX7BlKpdJbNY/sW1cjkE6ExyD4BZJDeQWDHd9ldgNPAySSC/q7mTRMTtwHDgFuBT4G6SMQNIxgJOlTQ7Dez59gcGkfT6/w6cEREPF9D2pYBzgBkkKaB+wMkFHGdWdbyevplZhrinb2aWIQ76ZmYZ4qBvZpYhDvpmZhnioG9mliEO+mZmGeKgb2aWIQ76ZmYZ4qBvZpYh/w/Xqy5zyORvjAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Confusion matrix\n",
    "plt.figure(figsize=(6,6))\n",
    "ax = sns.heatmap(conf_matrix, fmt='d', annot=True, linewidths=1, square=True)\n",
    "ax.set_xlabel('Predictions', size=12)\n",
    "ax.set_ylabel('True labels', size=12) \n",
    "ax.set_title('Confusion Matrix', size=12); \n",
    "ax.xaxis.set_ticklabels(['True', 'False'])\n",
    "ax.yaxis.set_ticklabels(['True', 'False'])\n",
    "ax.set_ylim(2,0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The generalization error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the dataset. Use clean_review and label columns\n",
    "test_dataset = pd.read_csv('dataset/datasets_feat_clean/test_feat_clean.csv',\n",
    "                            usecols=['clean_review', 'polarity', 'subjectivity', 'word_count', 'UPPERCASE', 'DIGITS',\n",
    "                                     'PROPN', 'VERB', 'NOUN', 'PUNCT', 'ADJ', 'label'])\n",
    "\n",
    "# Change columns order\n",
    "test_dataset = test_dataset[['clean_review', 'polarity', 'subjectivity', 'word_count', 'UPPERCASE', 'DIGITS',\n",
    "                             'PROPN', 'VERB', 'NOUN', 'PUNCT', 'ADJ', 'label']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clean_review</th>\n",
       "      <th>polarity</th>\n",
       "      <th>subjectivity</th>\n",
       "      <th>word_count</th>\n",
       "      <th>UPPERCASE</th>\n",
       "      <th>DIGITS</th>\n",
       "      <th>PROPN</th>\n",
       "      <th>VERB</th>\n",
       "      <th>NOUN</th>\n",
       "      <th>PUNCT</th>\n",
       "      <th>ADJ</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>hear film till pop cable tv understand geena o...</td>\n",
       "      <td>-0.01698</td>\n",
       "      <td>0.4990</td>\n",
       "      <td>226</td>\n",
       "      <td>0.01770</td>\n",
       "      <td>0.004425</td>\n",
       "      <td>0.08405</td>\n",
       "      <td>0.1991</td>\n",
       "      <td>0.1903</td>\n",
       "      <td>0.0929</td>\n",
       "      <td>0.05310</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>know film show local tv kid remember watch see...</td>\n",
       "      <td>0.03116</td>\n",
       "      <td>0.4822</td>\n",
       "      <td>418</td>\n",
       "      <td>0.09810</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.14830</td>\n",
       "      <td>0.1459</td>\n",
       "      <td>0.1698</td>\n",
       "      <td>0.2057</td>\n",
       "      <td>0.06940</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>admire kiss mouth frankness  pubic hair cut m...</td>\n",
       "      <td>-0.05817</td>\n",
       "      <td>0.5215</td>\n",
       "      <td>298</td>\n",
       "      <td>0.01342</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.03693</td>\n",
       "      <td>0.1544</td>\n",
       "      <td>0.2214</td>\n",
       "      <td>0.1913</td>\n",
       "      <td>0.12760</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>see cure kid love year late get hold copy acci...</td>\n",
       "      <td>0.08920</td>\n",
       "      <td>0.5264</td>\n",
       "      <td>757</td>\n",
       "      <td>0.03568</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.04755</td>\n",
       "      <td>0.1770</td>\n",
       "      <td>0.1783</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.07135</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>movie true fact see documentary day early movi...</td>\n",
       "      <td>0.12930</td>\n",
       "      <td>0.4688</td>\n",
       "      <td>186</td>\n",
       "      <td>0.01613</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.02689</td>\n",
       "      <td>0.1774</td>\n",
       "      <td>0.1720</td>\n",
       "      <td>0.0914</td>\n",
       "      <td>0.08060</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        clean_review  polarity  subjectivity  \\\n",
       "0  hear film till pop cable tv understand geena o...  -0.01698        0.4990   \n",
       "1  know film show local tv kid remember watch see...   0.03116        0.4822   \n",
       "2  admire kiss mouth frankness  pubic hair cut m...  -0.05817        0.5215   \n",
       "3  see cure kid love year late get hold copy acci...   0.08920        0.5264   \n",
       "4  movie true fact see documentary day early movi...   0.12930        0.4688   \n",
       "\n",
       "   word_count  UPPERCASE    DIGITS    PROPN    VERB    NOUN   PUNCT      ADJ  \\\n",
       "0         226    0.01770  0.004425  0.08405  0.1991  0.1903  0.0929  0.05310   \n",
       "1         418    0.09810  0.000000  0.14830  0.1459  0.1698  0.2057  0.06940   \n",
       "2         298    0.01342  0.000000  0.03693  0.1544  0.2214  0.1913  0.12760   \n",
       "3         757    0.03568  0.000000  0.04755  0.1770  0.1783  0.1625  0.07135   \n",
       "4         186    0.01613  0.000000  0.02689  0.1774  0.1720  0.0914  0.08060   \n",
       "\n",
       "   label  \n",
       "0      1  \n",
       "1      0  \n",
       "2      0  \n",
       "3      1  \n",
       "4      0  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the StandardScaler\n",
    "test_scaler = StandardScaler()\n",
    "# Scale the features\n",
    "test_dataset.iloc[:, 3:11] = test_scaler.fit_transform(test_dataset.iloc[:, 3:11])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trimmed vocabulary using as minimum count threashold: count = 3.00\n",
      "13480/39104 tokens has been retained\n",
      "Trimmed input strings vocabulary\n",
      "Trimmed input sequences lengths to the length of: 187\n",
      "Mapped words to indices\n",
      "Batches created\n"
     ]
    }
   ],
   "source": [
    "test_iterator = BatchIterator(test_dataset, batch_size=256, vocab_created=False, vocab=None, target_col=None,\n",
    "                              word2index=train_iterator.word2index, sos_token='<SOS>', eos_token='<EOS>',\n",
    "                              unk_token='<UNK>', pad_token='<PAD>', min_word_count=3, max_vocab_size=None,\n",
    "                              max_seq_len=0.9, use_pretrained_vectors=False, glove_path='glove/',\n",
    "                              glove_name='glove.6B.100d.txt', weights_file_name='glove/weights.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "adfb8557637f4939ab23fe65a56e6a83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=20, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, test_avg_loss, test_accuracy, test_conf_matrix = model.evaluate_model(test_iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.862. Test error: 0.319\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: {:.3f}. Test error: {:.3f}'.format(test_accuracy, test_avg_loss))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAFZCAYAAAB0RP9xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3debxVVf3/8dcbLoOKTEIqiDKI81fLyEytHEhFcagcMEO/hvLtF6ZmppmlaZFNlllaUmipqTkmKgrOheY8o6bgxBQyK+AA3M/vj70vHQ6Xew/33DPcs99PHvvB2WtP61wun7POZ629tiICMzPLhnaVroCZmZWPg76ZWYY46JuZZYiDvplZhjjom5lliIO+mVmG1FW6AmZm1WbF/NeLGsveoddAtVZdWptb+mZmGeKWvplZvvpVla5ByTjom5nli/pK16BkHPTNzPLVO+ibmWVG1HBL3x25ZmYZ4pa+mVk+p3fMzDKkhtM7DvpmZvk8ZNPMLENquKXvjlwzswxxS9/MLF8Nd+S6pW+tStIGkm6XtETSjUWc51hJk1uzbpUg6S5Jx1e6HrZ+IuqLWqqZg35GSfqKpCclLZU0Jw1Oe7XCqY8ANgU2iYgjW3qSiPhrROzfCvVZg6S9JYWkW/LKd0nLHyzwPD+UdE1z+0XEsIj4Swura5VSX1/cUsUc9DNI0unAxcBPSAL0lsBlwGGtcPqtgFcjYmUrnKtU5gF7SNokp+x44NXWuoAS/v9lVce/lBkjqRtwATAmIm6JiGURsSIibo+I76T7dJJ0saTZ6XKxpE7ptr0lzZT0bUnvpN8STki3nQ+cCxydfoMYld8iltQ/bVHXpev/K+l1Se9JekPSsTnlU3KO20PSE2na6AlJe+Rse1DSjyQ9nJ5nsqReTfwYPgL+DoxIj28PHAX8Ne9n9RtJMyS9K+kpSZ9Nyw8EvpfzPp/LqcdYSQ8Dy4GBadmJ6fbfS7op5/w/k3SfpKqdez2zor64pYo56GfPZ4DOwK1N7HMOsDvwcWAXYDfg+znbNwO6AX2BUcClknpExHkk3x7+FhFdImJ8UxWRtBFwCTAsIjYG9gCebWS/nsCd6b6bAL8C7sxrqX8FOAH4GNAROKOpawNXAcelrw8ApgKz8/Z5guRn0BO4FrhRUueIuDvvfe6Sc8xIYDSwMfBW3vm+DeycfqB9luRnd3xEFPXADiuB+lXFLVXMQT97NgHmN5N+ORa4ICLeiYh5wPkkwazBinT7ioiYCCwFtm1hfeqBnSRtEBFzImJqI/scDLwWEVdHxMqIuA54BTgkZ58rI+LViHgfuIEkWK9TRDwC9JS0LUnwv6qRfa6JiAXpNS8COtH8+/xzRExNj1mRd77lwFdJPrSuAb4ZETObOZ9Vglv6VkMWAL0a0ivr0Ic1W6lvpWWrz5H3obEc6LK+FYmIZcDRwNeBOZLulLRdAfVpqFPfnPX/tKA+VwMnA/vQyDefNIX1cppSWkzy7aaptBHAjKY2RsTjwOuASD6crBq5I9dqyL+AD4DDm9hnNkmHbIMtWTv1UahlwIY565vlboyISRHxBWBzktb7HwuoT0OdZrWwTg2uBr4BTExb4aul6ZezSHL9PSKiO7CEJFgDrCsl02SqRtIYkm8Ms4EzW151s5Zx0M+YiFhC0tl6qaTDJW0oqYOkYZJ+nu52HfB9Sb3TDtFzSdIRLfEs8DlJW6adyGc3bJC0qaRD09z+hyRposYSohOBbdJhpnWSjgZ2AO5oYZ0AiIg3gM+T9GHk2xhYSTLSp07SuUDXnO1zgf7rM0JH0jbAj0lSPCOBMyU1mYayCnF6x2pJRPwKOJ2kc3YeSUriZJIRLZAEpieB54EXgKfTspZc6x7gb+m5nmLNQN2OpHNzNrCQJAB/o5FzLACGp/suIGkhD4+I+S2pU965p0REY99iJgF3kQzjfIvk21Fu6qbhxrMFkp5u7jppOu0a4GcR8VxEvEYyAujqhpFRVkVqOL0jDxwwM1vTB89NLCowdt7loKodhuuWvplZhnjCNTOzfFWely+Gg76ZWb4qz8sXw+kdM7N8JR69I6mfpAfS+0CmSjo1Le8p6R5Jr6V/90jLJekSSdMkPS9p15xzHZ/u/5oKmNHVQd/MLF/pp2FYCXw7IrYnmfJkjKQdgO8C90XEYOC+dB1gGDA4XUYDv4fVU5ScB3yaZLqU8xo+KNalraR3PMTIzApVtSNnGkTEHGBO+vo9SS+T3GF+GLB3uttfgAdJbhI8DLgqnafpUUndJW2e7ntPRCwEkHQPcCDJvTaNaitBnw+nP1rpKlgV6TRodwC6dRlU4ZpYNVmydHrrnKiMHbmS+gOfAB4DNk0/EIiIOZI+lu7WlzXvE5mZlq2rfJ3aTNA3MyubIjtyJY0mScM0GBcR4xrZrwtwM3BaRLzbxCzbjW2IJsrXyUHfzCxfkS39NMCvFeRzSepAEvD/GhENT3KbK2nztJW/OfBOWj4T6Jdz+BYkd7LP5L/poIbyB5u6rjtyzczylXgahvTBOeOBl9NpURpMIHmKG+nft+WUH5eO4tkdWJKmgSYB+0vqkXbg7p+WrZNb+mZm5bcnyaR7L0hqeHDQ94CfAjdIGgW8DTQ8Z3oicBAwjWTq8BMAImKhpB+RPPAHkudcLGzqwg76Zmb5SnxzVkRMYd2jjPZrZP8AxqzjXFcAVxR6bQd9M7M8EdX9yMNiOOibmeXzNAxmZlYL3NI3M8vnWTbNzDKkhtM7DvpmZvnc0jczy5Aabum7I9fMLEPc0jczy+f0jplZhtRwesdB38wsn4O+mVmG1HB6xx25ZmYZ4pa+mVk+p3fMzDKkhtM7DvpmZvlquKXvnL6ZWYa4pW9mls/pHTOzDKnh9I6DvplZPgd9M7MMiah0DUrGHblmZhnilr6ZWT6nd8zMMsRB38wsQzxk08wsQ2q4pe+OXDOzDHFL38wsXw0P2XTQNzPLV8PpHQd9M7N8NRz0ndM3M8sQt/TNzPJ5yKaZWXZEvTtyzcyyo4Zz+g76Zmb5aji9445cM7MMcUvfzCyfc/pmZhninL6ZWYY46JuZZUgNz73jjlwzswxxS9/MLJ/TO2ZmGeLRO2ZmGeKbs8zMrBa4pW9mls/pHTOz7Ah35JqZZYhb+mZmGeKOXDMzqwVu6ZuZ5XN6x8wsQ9yRa2aWIW7pm5lliDtyzcysFrilX2H/mbeAcy4ax/xFS2gn8eUD9+Grh+/PkveW8p0LL2P2O/Pp87Fe/PLsMXTdeCPefW8Z5178J2bMeYdOHTtw/mknMrj/Frwxcw5n/vSy1eedOecdvjHyS4w8/IAKvjtrDZ06deSuSdfTsVNH6urac9vf7+bCsb/hrsnX06XLRgD07r0JTz35PMce83VOOfUkjjz6UADq6urYdttBDOr/KRYtWlLJt9G21HB6R9E2HhYQH05/tNJ1KIl5Cxczb+Fidti6P8uWv8+IU87j4nNP5bZ7ptBt440YddRwxt9wB+8uXca3vnY0F42/ng07d+L/HftF3pgxm7GXXc2fLjxrjXOuWlXP0ONO46+/Opc+m/aq0DsrrU6DdgegW5dBFa5JeWy00YYsW7acuro6Jt3zN84680c8+cSzq7df/ddLufOOe7n+ulvXOO7AYfsy5uSvccjBXy13lStiydLpACr2PEvP/nJRgbHLhTcXXYdSKWt6R1Kncl6vLejdszs7bN0fgI023IABW/bhnfmLeODRpzl06F4AHDp0L+7/19MAvP72bD798R0BGNCvD7PnzmNBXgvuseem0m+z3jUb8LNo2bLlAHToUEeHDnXkNta6dNmIz33uM9x5xz1rHXfEkYdw0423l62eNaM+iluqWFmCvqTdJL0AvJau7yLpt+W4dlsya+48Xpn+Fv+z3SAWLn6X3j27A8kHw8Il7wKwzYB+3PfwkwC88O/pzHlnAXPnL1zjPHc/9BjD9t69vJW3kmrXrh3/fOR2pr3xOA/c/zBPPfnc6m3DD9mfhx56hPfeW7rGMRts0JmhQz/HhNvuLnd1rYqVq6V/CTAcWAAQEc8B+5Tp2m3C8vc/4PSxv+XM0cfSZcMN1rnfqKOG8+7SZRx58g+4bsK9bDdoK9q3b796+4oVK3nwsWfYf6/dylFtK5P6+no+u8ch7LDtnuw6ZBe232Gb1dvW1ZofdtB+PProU87lt0QNt/TL1ZHbLiLektZIc61q6gBJo4HRAJdffjnH77dzCatXWStWruT0sb/l4L33YOieQwDo2b0r8xYupnfP7sxbuJie3boC0GXDDfjR6ScBEBEMO+EM+m7We/W5pjz5PNsP2opNenQr/xuxkluy5D2m/PNRhg79HC+/9Co9enbnk5/cmWOP+fpa+37piOFO7bSUh2wWbYak3YCQ1F7SacCrTR0QEeMiYkhEDBk9enR5alkBEcF5F49nQL8+HPelA1eX7737J5hw7xQAJtw7hX123xWAd5cuY8WKlQDcPOkhdt1pmzW+Gdz10KMM+7xTO7Vkk1496dZtYwA6d+7E3vvsyauvTgfgi18cxt13P8CHH360xjFdu3Zhrz13Y+Kd95a9vjWhxC19SVdIekfSi3nl35T0b0lTJf08p/xsSdPSbQfklB+Ylk2T9N1C3lq5Wvr/jyTFsyUwF7g3Lcu8Z156jTvuf4TB/bfgyJN/AMApxx/BqCOHc8aFl3Lr5H+wWe9NuOh7YwB4Y8YczrloHO3atWPQln04/9RRq8/1/gcf8q9nXuQH3/zfSrwVK5HNNu3NH8b9gnbt29OuXTtuveVOJt39AJC05n990eVrHTP8kAO4//4pLF/+frmrWxOi9CmaPwO/A65qKJC0D3AYsHNEfCjpY2n5DsAIYEegD3CvpIb83qXAF4CZwBOSJkTES01d2EM2rU3K2pBNK0xrDdl877RDigqMG198e7N1kNQfuCMidkrXbwDGRcS9efudDRARF6brk4Afppt/GBEHNLbfupSlpS/pj8BaP8SIqN28jZm1XUW29HP7JFPjImJcM4dtA3xW0ljgA+CMiHgC6AvktnpnpmUAM/LKP91c3cqV3sn95OoMfJE1K2tmVj2KnGUzDfDNBfl8dUAPYHfgU8ANkgbS+DeXoPE+2WY/rcoS9CPib7nrkq4G1r6TxMysGlRm2OVM4JZIcu6PS6oHeqXl/XL22wKYnb5eV/k6VWrCtQHAVhW6tplZ0yozTv/vwL4AaUdtR2A+MAEYIamTpAHAYOBx4AlgsKQBkjqSdPZOaO4i5crpL+K/XzvaAQuBgoYXmZnVGknXAXsDvSTNBM4DrgCuSIdxfgQcn7b6p6advC8BK4ExEbEqPc/JwCSgPXBFRExt7tolD/pK7sjaBZiVFtVHGxkyZGbZVOoQFRHHrGNTozPjRcRYYGwj5ROBietz7ZIH/YgISbdGxCdLfS0zs1ZR5VMpFKNcOf3HJe1apmuZmRXHc++0jKS6iFgJ7AWcJGk6sIxkCFJEhD8IzMzKqNTpnceBXYHDS3wdM7NWU4ZpGCqm1EFfABExvcTXMTNrPQ76LdZb0unr2hgRvyrx9c3M1l/tzqxc8qDfHuhCK0yAZGZWLk7vtNyciLigxNcwM7MClSWnb2bWpril32L7lfj8Zmatzzn9lomIhaU8v5lZKdRyTr9Ss2yamVkFlOshKmZmbYfTO2Zm2VHL6R0HfTOzfG7pm5llR9Rw0HdHrplZhrilb2aWr4Zb+g76ZmZ5ajm946BvZpbPQd/MLDtquaXvjlwzswxxS9/MLE8tt/Qd9M3M8jjom5llSdTuo0BalNOXtJWkLVq7MmZmVloFBX1J10j6TPr6OODfwKuS/reEdTMzq4ioL26pZoW29PcHnkpffxv4ArA78L1SVMrMrJKiXkUt1azQnH7HiPhIUh+gd0T8E0DS5qWrmplZZVR7a70YhQb95yR9B+gP3AmQfgC8W6J6mZlVTLgjlxOBTwHdge+nZXsC15WiUmZmVhoFtfQj4jXgqLyyG4EbS1EpM7NKymR6Jx2l06yIuKr1qmNmVnnV3hlbjKZa+icVcHwADvpmVlOidh+Ru+6gHxGfLWdFzMyqRS239Au+I1dSD0nHSDo9Xd8sHcFjZmZtRKF35H4WeBUYBZyfFm8H/KFE9TIzqxjfnAW/AY6NiMmSFqVljwK7laZaZmaVk8mcfp4BETE5fd3w4/gI6ND6VTIzq6xqb60Xo9Cc/iuShuaV7Qu82Mr1MTOzEiq0pX8GcJuk24ANJF0KfDFdzMxqSi1Pw1DoHbkPS/oEMJJkXP4c4DMR8VYpK2dmVgmZvCM3X0TMAH4iqUdELGr2ADOzNqq+hlv6hQ7Z7CbpSknLgfmSlqfr3UtcPzOzsotQUUs1K7Qj9wqSGTY/DfRI/+6alpuZWRtRaHpnX6BPRLyfrr+QTsg2qzTVMjOrHA/ZhGnAlnllWwCvtW51zMwqL6K4pZoVOrXyJGCypL8AM4B+wHHA1aWtnplZ+dVyS399plZ+G9gnZ30G8PlWr5GZWYXV8ugdT61sZpYhBY/TNzPLimofdlmMQsfp95F0g6S5klblLqWuoJlZudVyR26ho3f+kO57MLCUZErlO4FvlKheZmYVUx8qaqlmhaZ39gS2ioilkiIinpJ0AjAFuLx01TMzs9ZUaNBfRTJ/PsASSb2BJSRj9c3Makot5/QLDfpPAMOA24B7gGuB5cDTJaqXmVnFVHtevhiFBv2R/Df/fwpwFtAF+FUpKtWYToN2L9elrA1ZsnR6patgNaja8/LFKHQ+/YU5r5cD55WsRmZmFZbJ9I6kcws5QURc0HrVWbe6jn3LcRlrI1Z+lMz1t2L+6xWuiVWTDr0GVroKVa+plv7gAo6v4cyXmWVVJtM7ETGynBUxM6sWtdyaLfTmLDOzzCj1zVmSrpD0jqQXc8p+IekVSc9LujX3yYSSzpY0TdK/JR2QU35gWjZN0ncLeW8O+mZmecrwuMQ/Awfmld0D7BQROwOvAmcDSNoBGAHsmB5zmaT2ktoDl5IMp98BOCbdt0kO+mZmZRYR/wAW5pVNjoiV6eqj/Pfm18OA6yPiw4h4g+ShVruly7SIeD0iPgKuT/dtkoO+mVme+iIXSaMlPZmzjF7PKnwNuCt93Zfk+SUNZqZl6ypvUsFTK0vah+QrxqYRcbikXYGNI+KhQs9hZtYWBMWN3omIccC4lhwr6RxgJfDXhqLGLkHjjfZm+6ALnVr5G8B4kk+VhqdnfQSMLeR4M7O2pD6KW1pK0vHAcODYiNWTQcwkeURtgy2A2U2UN6nQ9M63gaER8WOSby8ALwPbF3i8mZk1QdKBJFPcHJrOfNBgAjBCUidJA0juoXqcZE60wZIGSOpIkomZ0Nx1Ck3vbAy8lb5u+PSp478zb5qZ1Yz6ItM7zZF0HbA30EvSTJKpbc4GOgH3SAJ4NCK+HhFTJd0AvESS9hkTEavS85wMTALaA1dExNTmrl1o0J8CnAH8LKdsDOB8vpnVnGJz+s2eP+KYRorHN7H/WBpJp0fERGDi+ly70KD/TeAOSScBG0uaStLKP2h9LmZm1hbUN79Lm1XoLJuzJH0S+AywJUmH7r8avmKYmdWSUrf0K6ngIZsRUQ88nC5mZtYGFRT0Jb3BOsZ/RoTnMjWzmpL59A5wYt765iR5/utatzpmZpWX+aAfEffll0m6j6TX+OLWrpSZWSU5p9+49wGndsys5tTXbswvOKef/+jEDYGDgcmtXiMzMyuZQlv6+Y9OXEYyj/OfW7U2ZmZVoNR35FZSs0E/naj/HuCGiPig9FUyM6usTD8uMb0B67cO+GaWFcXOp1/NCp1l805JnnLBzKyNKzSn3w64RdIUkikYVn/7iYivlaJiZmaVUq8M5/RTrwG/KGVFzMyqRS3n9JsM+pKOiYjrIuIH5aqQmVmlVXtevhjN5fQvL0stzMyqSL2KW6pZc0G/yqtvZmbro7mcfntJ+9BE8I+I+1u3SmZmlZXlm7M6kTzCa10/gcDz75hZjclsRy6wzPPlm1nWVHtevhiF3pxlZmY1oLmWfg1/3pmZNa6Wh2w2GfQjYuNyVcTMrFpkOadvZpY5tZzTd9A3M8tTy+kdd+SamWWIW/pmZnlquaXvoG9mliec0zczyw639M3MMqSWg747cs3MMsQtfTOzPL45y8wsQ3xzlplZhjinb2ZmNcEtfTOzPLXc0nfQNzPL445cM7MMcUeumVmG1HJ6xx25ZmYZ4pa+mVke5/TNzDKkvobDvoO+mVmeWs7pO+ibmeWp3Xa+O3LNzDLFLX0zszxO75iZZYhvzjIzy5BaHr3jnL6ZWYa4pV9lOnXqxIP330zHTp2oq2vPLbfcyfkXXET//v249prL6NGjB888+wLH/+8prFixgn79+nDl+N/QrXtX2rdvxznnXMhdd99f6bdhRZozdx7f+9Evmb9wEe0kjjhsGCOPOpxJ9/+Ty8Zfw+tvzeC6P17MTttvA8Adk+7nymtvXn38q9Pf4MYrfku/vptz3De+s7p87rz5DN9/H7572tfL/p7aktpt54Mi2sTbi7qOfStdh7LZaKMNWbZsOXV1dfzjwVv51unncdppo7n17xO54YYJXPq7n/L88y9x+bir+P1lP+PZZ6dy+bir2H77wdx+29Vsvc3ulX4LJbfyo1kArJj/eoVrUhrz5i9k3oKF7LDt1ixbtpyjRp3CJRf+ACTaqR3n/+ISzhhz4uqgn+vV6W9wyncv4O4br1xr21Ff+yZnnjKaIR//n3K8jbLr0GsgQNEZ+bP7f6WowHjhm9dWba+A0ztVaNmy5QB06FBHXYcORAT77L0nN998JwBXX30jhx16AAAR0LVrFwC6de3KnDlzK1Npa1W9e/Vkh223BpJGwMCt+jF33gIG9d+SAVtt0eSxE+95iGFDP79W+VszZrFg0WI+uctOJalzLaknilqqWdmCvhJflXRuur6lpN3Kdf22pF27djz5xGTmzHqe++77B9Nff5PFi5ewatUqAGbOmkOfvpsBcMGPLuIrX/kSb77+JLdPuIpTT/t+JatuJTBrzlxefm06O++4bUH7333fQxz0hb3XKp94z4McuN/nkKq2EVo1osilmpWzpX8Z8BngmHT9PeDSMl6/zaivr2fIp/ZnqwFD+NSQT7D9doPX2qchLTfi6MO56qob6T9wCIccehx//vMl/k9dQ5Yvf59vnfNjzjrl/+iy0UbN7v/81FfYoHNnBg/sv9a2u+57iIOG7t36lbQ2pZxB/9MRMQb4ACAiFgEd17WzpNGSnpT05Lhx48pVx6qyZMm7PPSPR/j0p3ele/dutG/fHoAt+m7OnNlJGueEE0Zw4023A/DoY0/RuVMnevXqWbE6W+tZsXIlp53zYw7efx++sPeeBR1z172Np3Zeee11Vq2qZ8dGGhC2tvoil2pWzqC/QlJ70m8/knrTxM8nIsZFxJCIGDJ69Ohy1bHievXqSbduXQHo3Lkz++37WV55ZRoPPvQIX/7ywQCMHHkkE26fDMCMt2ex7z57AbDddlvTuXMn5s1bUJnKW6uJCM698GIGbtWP40d8qaBj6uvrmfzAPxsN+nfd+2Cj5da4Ws7pl3PI5iXArcDHJI0FjgCcgM6z+eabcsX4i2nfvh3t2rXjpptu586J9/LSy69y7TWXccEPz+TZ56ZyxZXXAfCdsy7g8t//glNPPYmIYNSJ36rwO7DW8MzzU7n97vsYPKg/Xz5+DACn/t/xfLRiBRf++vcsXLyEb3znPLYbPJBxvx4LwJPPvsimvXvRr+/ma51v0v3/5LJfXlDW99CWVXfYLk5Zh2xK2g7Yj2RI1X0R8XKBh2ZqyKY1r9aHbFrLtNaQzVP7jygqMP7mzeurtmOtnKN3BgFvRMSlwIvAFyR1L9f1zcysvDn9m4FVkrYG/gQMAK4t4/XNzAoSRf6pZuXM6ddHxEpJXwJ+ExG/lfRMGa9vZlaQah+BU4xyj945BjgOuCMt61DG65uZFaQco3ckfUvSVEkvSrpOUmdJAyQ9Juk1SX+T1DHdt1O6Pi3d3r+l762cQf8EkpuzxkbEG5IGANeU8fpmZlVBUl/gFGBIROwEtAdGAD8Dfh0Rg4FFwKj0kFHAoojYGvh1ul+LlC3oR8RLEXFKRFyXrr8RET8t1/XNzApVpmkY6oANJNUBGwJzgH2Bm9LtfwEOT18flq6Tbt9PLbz1vuQ5fUkv0MTPISJ2LnUdzMzWR7E3WEkaDeTeVTouIlZPLRARsyT9EngbeB+YDDwFLI6IleluM4GGsep9gRnpsSslLQE2Aeavb93K0ZE7vAzXMDNrNcV25KYBfp3zx0jqQdJ6HwAsBm4EhjV2qoZDmti2Xkoe9CPirVJfw8ysNZVh2OVQkvuW5gFIugXYA+guqS5t7W8BzE73nwn0A2am6aBuwMKWXLicN2ftLukJSUslfSRplaR3y3V9M7Mq8jawu6QN09z8fsBLwAMkU9QAHA/clr6ekK6Tbr8/WjidQjnH6f+OpHf6RmAIydDNrct4fTOzgpR6nH5EPCbpJuBpYCXwDEk66E7gekk/TsvGp4eMB66WNI2khT+ipdcu6zNyI2KapPYRsQq4UtIj5by+mVkhynFXbUScB5yXV/w6sNbDpSLiA+DI1rhuOYP+8vRGg2cl/ZxkeFLzT4UwMysz35HbOkam1zsZWEbSKfHlMl7fzKwg9RFFLdWsHOP0t4yIt3NG8XwAnF/q65qZ2drK0dL/e8MLSTeX4XpmZkWp5QejlyOnn3tTwcAyXM/MrCjV/sjDYpQj6Mc6XpuZVaVqnxO/GOUI+rukN2GJZHKhhhuyBEREdC1DHczMjPJMw9C+1NcwM2tNtTxks6w3Z5mZtQXO6ZuZZYhz+mZmGVLL6Z1y3pFrZmYV5pa+mVmeFs5a3CY46JuZ5XFHrplZhtRyTt9B38wsTy2P3nFHrplZhrilb2aWxzl9M7MM8egdM7MMqeWOXOf0zcwyxC19M7M8tTx6x0HfzCyPO3LNzDLEHblmZhlSyy19d+SamWWIW/pmZnnckWtmliH1zumbmWVH7YZ8B30zs7W4I9fMzGqCW/pmZnlquaXvoG9mlsc3Z5mZZUgtt/Sd0zczyxC39M3M8vjmLDOzDHFO3/Pe+XMAAAd2SURBVMwsQ2o5p++gb2aWp5Zb+u7INTPLELf0zczyOL1jZpYhHr1jZpYhtTy1snP6ZmYZ4pa+mVkep3fMzDKkltM7DvpmZnnc0jczy5Babum7I9fMLEPc0jczy+P0ThVY+dGsSlfBqlCHXgMrXQWrQbWc3mkrQV+VrkC1kDQ6IsZVuh5WXfx70bpquaXvnH7bM7rSFbCq5N+LVhRRX9RSzRz0zcwypK2kd8zMysazbFo1cd7WGuPfi1ZUyw9RUS2/OTOzltii505FBcaZC1+s2sEnzumbmWWI0ztVQNImwH3p6mbAKmBeur5bRHxUkYpZxUhaBbyQU3R4RLy5jn37A3dExE6lr1k21HIGxEG/CkTEAuDjAJJ+CCyNiF/m7iNJJOm46h4PZq3l/Yj4eKUrkVW1fHOW0ztVTNLWkl6U9AfgaaCfpMU520dI+lP6elNJt0h6UtLjknavVL2tNCT1l/RPSU+nyx6N7LNj+u//rKTnJQ1Oy7+aU365pPblfwdtRxT5p1CS2kt6RtId6foASY9Jek3S3yR1TMs7pevT0u39W/reHPSr3w7A+Ij4BNDUXBSXAD+PiCHAUcCfylE5K5kN0gD9rKRb07J3gC9ExK7A0ST/5vm+Dvwm/ZYwBJgpaft0/z3T8lXAsaV/C21XRBS1rIdTgZdz1n8G/DoiBgOLgFFp+ShgUURsDfw63a9FnN6pftMj4okC9hsKbJtkgQDoIWmDiHi/dFWzEmosvdMB+J2khsC9TSPH/Qs4R9IWwC0R8Zqk/YBPAk+kvx8bkHyAWAWl/0YHA2OB09MU7r7AV9Jd/gL8EPg9cFj6GuAmkt8DRQs6Hxz0q9+ynNf1rDkPUeec18KdvrXuW8BcYBeSb+kf5O8QEddKeowkmEySdCLJ78ZfIuLscla2LSvTzVkXA2cCG6frmwCLI2Jluj4T6Ju+7gvMAIiIlZKWpPvPX9+LOr3ThqSduIskDZbUDvhizuZ7gTENK2lr0GpLN2BO+nswElgrLy9pIPB6RFwCTAB2JhkZdoSkj6X79JS0Vfmq3fYUm96RNDrtX2tY1pgbSdJw4J2IeCq3uLGqFLBtvbil3/acBdwNvA28BHRKy8cAv5d0Asm/6wPkfAhYTbgMuFnSkST/vssa2edo4KuSVgD/AS6IiIWSvg9MThsLK0h+N94qU73bnGJH76QznjZ1l/SewKGSDiL5xt6VpOXfXVJd2trfApid7j8T6EfSR1NH0gBY2JK6+Y5cM7M8PbpsXVRgXLR0WsF35EraGzgjIoZLuhG4OSKuT0ftPR8Rl0kaA/xPRHxd0gjgSxFxVEvq5vSOmVn1OIukU3caSc5+fFo+HtgkLT8d+G5LL+CWvplZnm5dBhUVGJcsnV61c+84p29mlqeWG8MO+mZmeTwNg5mZ1QQHfas66RwzkQ5NQ9Jdko5vwXm2lLTU88zY+irX3DuV4KBvLSbpTUnvp4F1rqQrJXVp7etExLCI+EuB9Rmac9zbEdElIla1dp2sttVHFLVUMwd9K9YhEdEF2BX4FPD93I1K+PfM2pQyTrhWdv7PaK0iImYBdwE7SXpQ0lhJDwPLgYGSukkaL2mOpFmSftyQdkmnl/2lpPmSXieZN2a19Hwn5qyfJOllSe9JeknSrpKuBrYEbk+/eZzZSJqoj6QJkhamU9SelHPOH0q6QdJV6XmnShqSs/2stN7vSfp3OomZ1Sind8yaIakfcBDwTFo0EhhNMpnUWyQzBq4EtgY+AewPNATyk4DhafkQ4IgmrnMkyWyDx5Hcun4osCAiRpJMTXFImtL5eSOHX0dyO3uf9Bo/yQvehwLXA91J5q35XXrNbYGTgU9FxMbAAcCbzf9UzKqPg74V6+9KHuwyBXgI+Ela/ueImJrOIdITGAacFhHLIuIdkjnBR6T7HgVcHBEzImIhcGET1zuR5LkBT0RiWkQ0O4dM+qG0F3BWRHwQEc+SPHNgZM5uUyJiYtoHcDXJbJaQTGPcCdhBUoeIeDMipjd3TWu7ajm943H6VqzDI+Le3IJ0zvYZOUVbkcwFPydnvv92Ofv0ydu/qSDeD2hJwO0DLIyI9/KuMyRn/T85r5cDndPJr6ZJOo3kG8aOkiYBp0fEbKwmVXvgLoZb+lYquf9rZgAfAr0ionu6dI2IHdPtc0iCeYMtmzjvDGBQAdfMNxvoKWnjnLItafppZP89ccS1EbEXyQdYUMSTi6z6RZFLNXNL30ouIuZImgxcJOkHwFJgALBFRDwE3ACcouQ5octoejKpPwG/kjSF5LnBg4AVaYpnLjBwHXWYIekR4EJJZ5A8dWoU8NXm6p/m9PsCD5M8uOR93GCqaSs/mlW1c+cUy7+4Vi7HAR1JngGwiOSRb5un2/4ITAKeIwnkt6zrJBFxI8nj5a4F3gP+TtJnAElfwPclLU4De75jgP4krf5bgfMi4p4C6t4J+CnJU4r+A3wM+F4Bx5lVHc+yaWaWIW7pm5lliIO+mVmGOOibmWWIg76ZWYY46JuZZYiDvplZhjjom5lliIO+mVmGOOibmWXI/wdy7OWUMOBbfAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Confusion matrix\n",
    "plt.figure(figsize=(6,6))\n",
    "ax = sns.heatmap(test_conf_matrix, fmt='d', annot=True, linewidths=1, square=True)\n",
    "ax.set_xlabel('Predictions', size=12)\n",
    "ax.set_ylabel('True labels', size=12) \n",
    "ax.set_title('Confusion Matrix', size=12); \n",
    "ax.xaxis.set_ticklabels(['True', 'False'])\n",
    "ax.yaxis.set_ticklabels(['True', 'False'])\n",
    "ax.set_ylim(2,0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model with pre-trained word embeddings achieved worse result than the previous models. The generalization accuracy equals 0.862. The confusion matrix depicted above shows that the model misclassified the positive examples (377) more times than negative examples (308)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}