Assignment 2 - NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 1: \n",
    "# Topics in Natural Language Processing (202-2-5381) Fall 2018\n",
    "# Prof. Michael Elhadad\n",
    "\n",
    "### Author: Dina Svetlitsky\n",
    "\n",
    "[Assignment description](https://www.cs.bgu.ac.il/~elhadad/nlp18/hw2.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q1. Questions Classification "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q1.1. Dataset description\n",
    "In this part we will use questions classificaton dataset described in [Li, Dan Roth, Learning Question Classifiers. COLING'02](http://www.aclweb.org/anthology/C02-1150).\n",
    "\n",
    "This paper presents a machine learning approach to question classification. \n",
    "\n",
    "Question Classification(QC) is defined to be a task in which, given a question, maps it to one of k classes, which provide a semantic constraint on\n",
    "the sought-after answer. A question classification module should: (1) provide constraints on the answer types that allow\n",
    "further processing to precisely locate and verify the\n",
    "answer. (2) Should provide information that downstream\n",
    "processes may use in determining answer selection\n",
    "strategies that may be answer type specific,\n",
    "rather than uniform. For example, given the question\n",
    "“Who was the first woman killed in the Vietnam\n",
    "War?” we would like to know that the target of this question is a person, thereby reducing\n",
    "the space of possible answers significantly\n",
    "\n",
    "The authors developed a hierarchical\n",
    "classifier that is guided by a layered semantic hierarchy of answers types, and used it to classify questions into fine-grained classes. The question classifier makes use of a sequence\n",
    "of two simple classifiers, each utilizing the Winnow algorithm within SNoW. The first classifies questions into coarse classes (Coarse Classifier) and the second into fine classes (Fine Classifier).\n",
    "\n",
    "The hierarchical classifier got f1 score of 84.2. However, comparison between the hierarchical and a flat classifier shows that there is no advantage to the hierarchical classifier.\n",
    "The authors also computed f1 score when allowing multiple labels, while checking if one of 5 labels they predicted fits the true labeling. In this evaluation method, the f1 score is much higher and reaches 95. \n",
    "\n",
    "Data is collected from 4 sources: \n",
    "1. 4,500 English questions published by USC (Hovy et al., 2001)\n",
    "2. About 500 manually constructed questions for a few rare classes\n",
    "3. 894 TREC 8 and TREC 9 questions\n",
    "4. 500 questions from TREC 10 which serves as a test set\n",
    "\n",
    "In the TREC competition (Voorhees, 2000), participants are requested to build a system which, given a set of English questions, can automatically extract answers (a short phrase) of no more than 50 bytes from a 5-gigabyte document library.\n",
    "\n",
    "The questions were manually labeled according to question hierarchy:\n",
    "A two-layered taxonomy, which represents a natural semantic classification for typical answers in the TREC task. The hierarchy contains 6 coarse classes (ABBREVIATION, ENTITY, DESCRIPTION, HUMAN, LOCATION and NUMERIC VALUE) and 50 fine classes. Each coarse class contains a non-overlapping set of fine classes. For example the coarse class HUMAN contains a set of 4 fine classes: group, induvidual, title, description. The motivation behind adding a level of coarse classes is that of\n",
    "compatibility with previous work’s definitions, and comprehensibility.\n",
    "\n",
    "There is exactly one label to each question. \n",
    "Annotators were requested to choose the most suitable class according to their own understanding. This methodology might cause slight problems in training, when the labels are ambiguous, since some questions are not treated as positive examples for possible classes as they should be. \n",
    "\n",
    "#### Training Set: \n",
    "5,500 questions from the first three sources were randomly divided into 5 training sets of 1,000, 2,000, 3,000, 4,000 and 5,500 questions.\n",
    "\n",
    "#### Test Set:\n",
    "All 500 TREC 10 questions (source number 4) are used as the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q1.2. Dataset Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the tagged train dataset that contains all 5,500 questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def parse_tagged_quests_dataset(file_name):\n",
    "    dataset_file = open(file_name, \"r\")\n",
    "    quests_tagged = []\n",
    "    for line in dataset_file:\n",
    "        line_arr = line.rstrip().split()\n",
    "        coarse_class = line_arr[0].split(\":\")[0]\n",
    "        fine_class = line_arr[0].split(\":\")[1]\n",
    "        raw_question = \" \".join(line_arr[1:])\n",
    "        \n",
    "        quests_tagged.append((raw_question, coarse_class, fine_class))\n",
    "  \n",
    "    return quests_tagged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('How did serfdom develop in and then leave Russia ?', 'DESC', 'manner'),\n",
       " ('What films featured the character Popeye Doyle ?', 'ENTY', 'cremat')]"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_file = \"questions_dataset/train_5500.label\"\n",
    "train_quests_raw_tagged = parse_tagged_quests_dataset(train_set_file)\n",
    "train_quests_raw_tagged[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of questions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5452"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_quests_raw_tagged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, it is not exactly 5,500 questions, but rather 5,452 questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to perform some analysis we will create a dataframe: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "questions_dataset = []\n",
    "for raw_quest, coarse_tag, fine_tag in train_quests_raw_tagged:\n",
    "    tokens = raw_quest.lower().split(\" \")\n",
    "    questions_dataset.append({'coarse class': coarse_tag, 'fine class': fine_tag, \n",
    "                              #'question': \" \".join(question_tokens), \n",
    "                              'n. of tokens': len(tokens),\n",
    "                              'vocab. size': len(set(tokens))\n",
    "                             })\n",
    "train_df = pd.DataFrame(questions_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will peek at the top lines in the created data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coarse class</th>\n",
       "      <th>fine class</th>\n",
       "      <th>n. of tokens</th>\n",
       "      <th>vocab. size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>DESC</td>\n",
       "      <td>manner</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ENTY</td>\n",
       "      <td>cremat</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DESC</td>\n",
       "      <td>manner</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ENTY</td>\n",
       "      <td>animal</td>\n",
       "      <td>13</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ABBR</td>\n",
       "      <td>exp</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  coarse class fine class  n. of tokens  vocab. size\n",
       "0  DESC         manner     10            10         \n",
       "1  ENTY         cremat     8             8          \n",
       "2  DESC         manner     12            12         \n",
       "3  ENTY         animal     13            11         \n",
       "4  ABBR         exp        8             8          "
      ]
     },
     "execution_count": 216,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the 10 most common words for each label?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "def get_top_coarse_tokens(quests_tagged, top, stop_words = []):\n",
    "    tokens_per_coarse_tag = {}\n",
    "    for raw_quest, coarse_tag, fine_tag in quests_tagged:\n",
    "        quest_tokens = raw_quest.lower().split(\" \")\n",
    "        if coarse_tag not in tokens_per_coarse_tag:\n",
    "            tokens_per_coarse_tag[coarse_tag] = Counter()\n",
    "        for token in quest_tokens:\n",
    "            if token not in stop_words:\n",
    "                tokens_per_coarse_tag[coarse_tag][token] += 1\n",
    "            \n",
    "    return pd.DataFrame({coarse_tag:counter.most_common(top) for coarse_tag, counter in tokens_per_coarse_tag.items()})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ABBR</th>\n",
       "      <th>DESC</th>\n",
       "      <th>ENTY</th>\n",
       "      <th>HUM</th>\n",
       "      <th>LOC</th>\n",
       "      <th>NUM</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(what, 86)</td>\n",
       "      <td>(?, 1151)</td>\n",
       "      <td>(?, 1216)</td>\n",
       "      <td>(?, 1179)</td>\n",
       "      <td>(?, 826)</td>\n",
       "      <td>(?, 892)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(?, 86)</td>\n",
       "      <td>(what, 762)</td>\n",
       "      <td>(what, 1152)</td>\n",
       "      <td>(the, 1017)</td>\n",
       "      <td>(the, 669)</td>\n",
       "      <td>(the, 559)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(for, 54)</td>\n",
       "      <td>(the, 603)</td>\n",
       "      <td>(the, 875)</td>\n",
       "      <td>(who, 597)</td>\n",
       "      <td>(what, 554)</td>\n",
       "      <td>(how, 492)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(the, 48)</td>\n",
       "      <td>(is, 487)</td>\n",
       "      <td>(of, 452)</td>\n",
       "      <td>(what, 547)</td>\n",
       "      <td>(is, 304)</td>\n",
       "      <td>(many, 323)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(does, 48)</td>\n",
       "      <td>(how, 287)</td>\n",
       "      <td>(is, 400)</td>\n",
       "      <td>(of, 386)</td>\n",
       "      <td>(where, 258)</td>\n",
       "      <td>(what, 276)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(stand, 41)</td>\n",
       "      <td>(a, 264)</td>\n",
       "      <td>(a, 310)</td>\n",
       "      <td>(in, 293)</td>\n",
       "      <td>(in, 221)</td>\n",
       "      <td>(of, 246)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(is, 33)</td>\n",
       "      <td>(of, 247)</td>\n",
       "      <td>(in, 302)</td>\n",
       "      <td>(was, 291)</td>\n",
       "      <td>(of, 201)</td>\n",
       "      <td>(in, 245)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(abbreviation, 16)</td>\n",
       "      <td>(do, 213)</td>\n",
       "      <td>('s, 206)</td>\n",
       "      <td>('s, 237)</td>\n",
       "      <td>(country, 123)</td>\n",
       "      <td>(is, 223)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(of, 15)</td>\n",
       "      <td>(to, 130)</td>\n",
       "      <td>(,, 173)</td>\n",
       "      <td>(is, 234)</td>\n",
       "      <td>('s, 102)</td>\n",
       "      <td>(a, 213)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>(in, 10)</td>\n",
       "      <td>(in, 129)</td>\n",
       "      <td>(to, 162)</td>\n",
       "      <td>(a, 159)</td>\n",
       "      <td>(city, 101)</td>\n",
       "      <td>(when, 136)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ABBR         DESC          ENTY          HUM             LOC  \\\n",
       "0  (what, 86)          (?, 1151)    (?, 1216)     (?, 1179)    (?, 826)         \n",
       "1  (?, 86)             (what, 762)  (what, 1152)  (the, 1017)  (the, 669)       \n",
       "2  (for, 54)           (the, 603)   (the, 875)    (who, 597)   (what, 554)      \n",
       "3  (the, 48)           (is, 487)    (of, 452)     (what, 547)  (is, 304)        \n",
       "4  (does, 48)          (how, 287)   (is, 400)     (of, 386)    (where, 258)     \n",
       "5  (stand, 41)         (a, 264)     (a, 310)      (in, 293)    (in, 221)        \n",
       "6  (is, 33)            (of, 247)    (in, 302)     (was, 291)   (of, 201)        \n",
       "7  (abbreviation, 16)  (do, 213)    ('s, 206)     ('s, 237)    (country, 123)   \n",
       "8  (of, 15)            (to, 130)    (,, 173)      (is, 234)    ('s, 102)        \n",
       "9  (in, 10)            (in, 129)    (to, 162)     (a, 159)     (city, 101)      \n",
       "\n",
       "           NUM  \n",
       "0  (?, 892)     \n",
       "1  (the, 559)   \n",
       "2  (how, 492)   \n",
       "3  (many, 323)  \n",
       "4  (what, 276)  \n",
       "5  (of, 246)    \n",
       "6  (in, 245)    \n",
       "7  (is, 223)    \n",
       "8  (a, 213)     \n",
       "9  (when, 136)  "
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_top_coarse_tokens(train_quests_raw_tagged, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that '?' and 'the' are common for all tags, so they are not informative. Let's get rid of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ABBR</th>\n",
       "      <th>DESC</th>\n",
       "      <th>ENTY</th>\n",
       "      <th>HUM</th>\n",
       "      <th>LOC</th>\n",
       "      <th>NUM</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(what, 86)</td>\n",
       "      <td>(what, 762)</td>\n",
       "      <td>(what, 1152)</td>\n",
       "      <td>(who, 597)</td>\n",
       "      <td>(what, 554)</td>\n",
       "      <td>(how, 492)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(for, 54)</td>\n",
       "      <td>(is, 487)</td>\n",
       "      <td>(of, 452)</td>\n",
       "      <td>(what, 547)</td>\n",
       "      <td>(is, 304)</td>\n",
       "      <td>(many, 323)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(does, 48)</td>\n",
       "      <td>(how, 287)</td>\n",
       "      <td>(is, 400)</td>\n",
       "      <td>(of, 386)</td>\n",
       "      <td>(where, 258)</td>\n",
       "      <td>(what, 276)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(stand, 41)</td>\n",
       "      <td>(a, 264)</td>\n",
       "      <td>(a, 310)</td>\n",
       "      <td>(in, 293)</td>\n",
       "      <td>(in, 221)</td>\n",
       "      <td>(of, 246)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(is, 33)</td>\n",
       "      <td>(of, 247)</td>\n",
       "      <td>(in, 302)</td>\n",
       "      <td>(was, 291)</td>\n",
       "      <td>(of, 201)</td>\n",
       "      <td>(in, 245)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(abbreviation, 16)</td>\n",
       "      <td>(do, 213)</td>\n",
       "      <td>('s, 206)</td>\n",
       "      <td>('s, 237)</td>\n",
       "      <td>(country, 123)</td>\n",
       "      <td>(is, 223)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(of, 15)</td>\n",
       "      <td>(to, 130)</td>\n",
       "      <td>(,, 173)</td>\n",
       "      <td>(is, 234)</td>\n",
       "      <td>('s, 102)</td>\n",
       "      <td>(a, 213)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(in, 10)</td>\n",
       "      <td>(in, 129)</td>\n",
       "      <td>(to, 162)</td>\n",
       "      <td>(a, 159)</td>\n",
       "      <td>(city, 101)</td>\n",
       "      <td>(when, 136)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(,, 9)</td>\n",
       "      <td>(does, 129)</td>\n",
       "      <td>(was, 138)</td>\n",
       "      <td>(to, 153)</td>\n",
       "      <td>(,, 87)</td>\n",
       "      <td>(are, 136)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>(mean, 9)</td>\n",
       "      <td>(``, 112)</td>\n",
       "      <td>(name, 118)</td>\n",
       "      <td>(,, 151)</td>\n",
       "      <td>(a, 78)</td>\n",
       "      <td>(did, 121)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ABBR         DESC          ENTY          HUM             LOC  \\\n",
       "0  (what, 86)          (what, 762)  (what, 1152)  (who, 597)   (what, 554)      \n",
       "1  (for, 54)           (is, 487)    (of, 452)     (what, 547)  (is, 304)        \n",
       "2  (does, 48)          (how, 287)   (is, 400)     (of, 386)    (where, 258)     \n",
       "3  (stand, 41)         (a, 264)     (a, 310)      (in, 293)    (in, 221)        \n",
       "4  (is, 33)            (of, 247)    (in, 302)     (was, 291)   (of, 201)        \n",
       "5  (abbreviation, 16)  (do, 213)    ('s, 206)     ('s, 237)    (country, 123)   \n",
       "6  (of, 15)            (to, 130)    (,, 173)      (is, 234)    ('s, 102)        \n",
       "7  (in, 10)            (in, 129)    (to, 162)     (a, 159)     (city, 101)      \n",
       "8  (,, 9)              (does, 129)  (was, 138)    (to, 153)    (,, 87)          \n",
       "9  (mean, 9)           (``, 112)    (name, 118)   (,, 151)     (a, 78)          \n",
       "\n",
       "           NUM  \n",
       "0  (how, 492)   \n",
       "1  (many, 323)  \n",
       "2  (what, 276)  \n",
       "3  (of, 246)    \n",
       "4  (in, 245)    \n",
       "5  (is, 223)    \n",
       "6  (a, 213)     \n",
       "7  (when, 136)  \n",
       "8  (are, 136)   \n",
       "9  (did, 121)   "
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stop_words = ['?', 'the']\n",
    "get_top_coarse_tokens(train_quests_raw_tagged, 10, stop_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('?', 5350),\n",
       " ('the', 3771),\n",
       " ('what', 3377),\n",
       " ('is', 1681),\n",
       " ('of', 1547),\n",
       " ('in', 1200),\n",
       " ('a', 1029),\n",
       " ('how', 789),\n",
       " (\"'s\", 720),\n",
       " ('was', 653)]"
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabulary = Counter(token for raw_quest, coarse_tag, fine_tag in train_quests_raw_tagged \n",
    "                     for token in raw_quest.lower().split(\" \"))\n",
    "vocabulary.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8678"
      ]
     },
     "execution_count": 221,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of words appearing 1-5 times:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({1: 5200, 2: 1304, 3: 598, 4: 369, 5: 261})\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Container object of 5 artists>"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADxlJREFUeJzt3V+oXWeZx/Hvz7RTQ7XY0jMhJHHSiyCkASs9ZAKVYcai\nPU7F9GIoEbS56DQXjVCZAUnmZvAi0CuRwrQQVJqiYwioNFjrEGNFBNt4otWY1NBgU5pD2kRFYm86\nND5zcd4Z9xwTzjnJyV5N3u8HFvtdz17vOs8iF7+9/uydVBWSpD69a+gGJEnDMQQkqWOGgCR1zBCQ\npI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHbtu6Abmc+utt9batWuHbkOSriqHDx/+bVVNzLfdOz4E\n1q5dy/T09NBtSNJVJcmrC9nOy0GS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkC\nktSxd/w3hi/H2h3PDN3Ckjn56L1DtyDpGuSZgCR1zBCQpI4ZApLUsQWFQJKTSY4keTHJdKvdkuRA\nkpfb680j2+9MciLJ8ST3jNTvbPs5keSxJFn6Q5IkLdRizgT+oaruqKrJtr4DOFhV64CDbZ0k64Et\nwO3AFPB4kmVtzhPAQ8C6tkxd/iFIki7V5VwO2gzsaeM9wH0j9b1V9VZVvQKcADYmWQncVFXPV1UB\nT43MkSQNYKEhUMD3kxxOsq3VVlTV6TZ+HVjRxquA10bmnmq1VW08ty5JGshCvyfw4aqaSfLXwIEk\nvx59s6oqSS1VUy1otgG8//3vX6rdSpLmWNCZQFXNtNczwLeBjcAb7RIP7fVM23wGWDMyfXWrzbTx\n3PqF/t7uqpqsqsmJiXn/i0xJ0iWaNwSS3Jjkvf87Bj4G/ArYD2xtm20Fnm7j/cCWJDckuY3ZG8CH\n2qWjc0k2taeCHhiZI0kawEIuB60Avt2e5rwO+M+q+l6SnwL7kjwIvArcD1BVR5PsA44BbwPbq+p8\n29fDwJPAcuDZtkiSBjJvCFTVb4APXqD+O+Dui8zZBey6QH0a2LD4NiVJV4LfGJakjhkCktQxQ0CS\nOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKlj\nhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYI\nSFLHDAFJ6pghIEkdMwQkqWMLDoEky5L8PMl32votSQ4kebm93jyy7c4kJ5IcT3LPSP3OJEfae48l\nydIejiRpMRZzJvAI8NLI+g7gYFWtAw62dZKsB7YAtwNTwONJlrU5TwAPAevaMnVZ3UuSLsuCQiDJ\nauBe4Msj5c3AnjbeA9w3Ut9bVW9V1SvACWBjkpXATVX1fFUV8NTIHEnSABZ6JvAl4PPAn0ZqK6rq\ndBu/Dqxo41XAayPbnWq1VW08ty5JGsi8IZDkE8CZqjp8sW3aJ/taqqaSbEsynWT67NmzS7VbSdIc\nCzkTuAv4ZJKTwF7gI0m+BrzRLvHQXs+07WeANSPzV7faTBvPrf+FqtpdVZNVNTkxMbGIw5EkLca8\nIVBVO6tqdVWtZfaG7w+q6tPAfmBr22wr8HQb7we2JLkhyW3M3gA+1C4dnUuyqT0V9MDIHEnSAK67\njLmPAvuSPAi8CtwPUFVHk+wDjgFvA9ur6nyb8zDwJLAceLYtkqSBLCoEquqHwA/b+HfA3RfZbhew\n6wL1aWDDYpuUJF0ZfmNYkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghI\nUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1\nzBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOzRsCSd6d5FCSXyQ5muQL\nrX5LkgNJXm6vN4/M2ZnkRJLjSe4Zqd+Z5Eh777EkuTKHJUlaiIWcCbwFfKSqPgjcAUwl2QTsAA5W\n1TrgYFsnyXpgC3A7MAU8nmRZ29cTwEPAurZMLeGxSJIWad4QqFlvttXr21LAZmBPq+8B7mvjzcDe\nqnqrql4BTgAbk6wEbqqq56uqgKdG5kiSBrCgewJJliV5ETgDHKiqF4AVVXW6bfI6sKKNVwGvjUw/\n1Wqr2nhu/UJ/b1uS6STTZ8+eXfDBSJIWZ0EhUFXnq+oOYDWzn+o3zHm/mD07WBJVtbuqJqtqcmJi\nYql2K0maY1FPB1XVH4DnmL2W/0a7xEN7PdM2mwHWjExb3WozbTy3LkkayEKeDppI8r42Xg58FPg1\nsB/Y2jbbCjzdxvuBLUluSHIbszeAD7VLR+eSbGpPBT0wMkeSNIDrFrDNSmBPe8LnXcC+qvpOkp8A\n+5I8CLwK3A9QVUeT7AOOAW8D26vqfNvXw8CTwHLg2bZIkgYybwhU1S+BD12g/jvg7ovM2QXsukB9\nGtjwlzMkSUPwG8OS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYI\nSFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAk\ndcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY/OGQJI1SZ5LcizJ0SSPtPotSQ4kebm9\n3jwyZ2eSE0mOJ7lnpH5nkiPtvceS5MocliRpIRZyJvA28K9VtR7YBGxPsh7YARysqnXAwbZOe28L\ncDswBTyeZFnb1xPAQ8C6tkwt4bFIkhZp3hCoqtNV9bM2/iPwErAK2AzsaZvtAe5r483A3qp6q6pe\nAU4AG5OsBG6qquerqoCnRuZIkgawqHsCSdYCHwJeAFZU1en21uvAijZeBbw2Mu1Uq61q47l1SdJA\nFhwCSd4DfBP4XFWdG32vfbKvpWoqybYk00mmz549u1S7lSTNsaAQSHI9swHw9ar6Viu/0S7x0F7P\ntPoMsGZk+upWm2njufW/UFW7q2qyqiYnJiYWeiySpEVayNNBAb4CvFRVXxx5az+wtY23Ak+P1Lck\nuSHJbczeAD7ULh2dS7Kp7fOBkTmSpAFct4Bt7gI+AxxJ8mKr/RvwKLAvyYPAq8D9AFV1NMk+4Biz\nTxZtr6rzbd7DwJPAcuDZtkiSBjJvCFTVj4GLPc9/90Xm7AJ2XaA+DWxYTIOSpCvHbwxLUscMAUnq\nmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4Z\nApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1LHrhm5AV8baHc8M3cKSOfno\nvUO3IF2zPBOQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKlj84ZAkq8mOZPkVyO1W5IcSPJy\ne7155L2dSU4kOZ7knpH6nUmOtPceS5KlPxxJ0mIs5EzgSWBqTm0HcLCq1gEH2zpJ1gNbgNvbnMeT\nLGtzngAeAta1Ze4+JUljNm8IVNWPgN/PKW8G9rTxHuC+kfreqnqrql4BTgAbk6wEbqqq56uqgKdG\n5kiSBnKp9wRWVNXpNn4dWNHGq4DXRrY71Wqr2nhuXZI0oMu+Mdw+2dcS9PJ/kmxLMp1k+uzZs0u5\na0nSiEsNgTfaJR7a65lWnwHWjGy3utVm2nhu/YKqandVTVbV5MTExCW2KEmaz6WGwH5gaxtvBZ4e\nqW9JckOS25i9AXyoXTo6l2RTeyrogZE5kqSBzPtT0km+Afw9cGuSU8C/A48C+5I8CLwK3A9QVUeT\n7AOOAW8D26vqfNvVw8w+abQceLYtkqQBzRsCVfWpi7x190W23wXsukB9GtiwqO4kSVeU3xiWpI4Z\nApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6Nu9vB0lXo7U7nhm6\nhSVx8tF7h25B1zjPBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkd88ti0jXmWvmi\nHPhluXHwTECSOmYISFLHDAFJ6pj3BCRdU66VeyLjuh/imYAkdcwQkKSOGQKS1DFDQJI6ZghIUscM\nAUnq2NhDIMlUkuNJTiTZMe6/L0n6s7GGQJJlwH8AHwfWA59Ksn6cPUiS/mzcZwIbgRNV9Zuq+m9g\nL7B5zD1Ikppxh8Aq4LWR9VOtJkkaQKpqfH8s+Sdgqqr+ua1/BvjbqvrsnO22Adva6geA42NrcvFu\nBX47dBMD6vn4ez526Pv4r4Zj/5uqmphvo3H/dtAMsGZkfXWr/T9VtRvYPa6mLkeS6aqaHLqPofR8\n/D0fO/R9/NfSsY/7ctBPgXVJbkvyV8AWYP+Ye5AkNWM9E6iqt5N8FvgvYBnw1ao6Os4eJEl/Nvaf\nkq6q7wLfHfffvYKuistWV1DPx9/zsUPfx3/NHPtYbwxLkt5Z/NkISeqYIXCJknw1yZkkvxq6l3FL\nsibJc0mOJTma5JGhexqnJO9OcijJL9rxf2HonsYtybIkP0/ynaF7GbckJ5McSfJikumh+7lcXg66\nREn+DngTeKqqNgzdzzglWQmsrKqfJXkvcBi4r6qODdzaWCQJcGNVvZnkeuDHwCNV9fzArY1Nkn8B\nJoGbquoTQ/czTklOApNV9U7/nsCCeCZwiarqR8Dvh+5jCFV1uqp+1sZ/BF6io29+16w32+r1benm\n01SS1cC9wJeH7kWXzxDQZUmyFvgQ8MKwnYxXuxzyInAGOFBVPR3/l4DPA38aupGBFPD9JIfbrxtc\n1QwBXbIk7wG+CXyuqs4N3c84VdX5qrqD2W+9b0zSxSXBJJ8AzlTV4aF7GdCH27/9x4Ht7dLwVcsQ\n0CVp18K/CXy9qr41dD9Dqao/AM8BU0P3MiZ3AZ9s18X3Ah9J8rVhWxqvqpppr2eAbzP768hXLUNA\ni9ZujH4FeKmqvjh0P+OWZCLJ+9p4OfBR4NfDdjUeVbWzqlZX1Vpmf/blB1X16YHbGpskN7aHIUhy\nI/Ax4Kp+QtAQuERJvgH8BPhAklNJHhy6pzG6C/gMs58CX2zLPw7d1BitBJ5L8ktmfw/rQFV196hk\np1YAP07yC+AQ8ExVfW/gni6Lj4hKUsc8E5CkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOG\ngCR17H8A50HwaKMAiN4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20eaddae390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "count_rare_words = Counter(count for count in vocabulary.values() if count < 6)\n",
    "print(count_rare_words)\n",
    "plt.bar(range(1,6), [count_rare_words[count] for count in range(1,6)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.890988707075363"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(count_rare_words.values())/len(vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "90% of the words in the vocabulary occur only 1-5 times."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The describe method can help us gather some statistics on the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coarse class</th>\n",
       "      <th>fine class</th>\n",
       "      <th>n. of tokens</th>\n",
       "      <th>vocab. size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>5452</td>\n",
       "      <td>5452</td>\n",
       "      <td>5452.000000</td>\n",
       "      <td>5452.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>6</td>\n",
       "      <td>47</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>ENTY</td>\n",
       "      <td>ind</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1250</td>\n",
       "      <td>962</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.204512</td>\n",
       "      <td>9.855282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.888865</td>\n",
       "      <td>3.484136</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>7.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>12.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>37.000000</td>\n",
       "      <td>30.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       coarse class fine class  n. of tokens  vocab. size\n",
       "count   5452         5452       5452.000000   5452.000000\n",
       "unique  6            47        NaN           NaN         \n",
       "top     ENTY         ind       NaN           NaN         \n",
       "freq    1250         962       NaN           NaN         \n",
       "mean    NaN          NaN        10.204512     9.855282   \n",
       "std     NaN          NaN        3.888865      3.484136   \n",
       "min     NaN          NaN        3.000000      3.000000   \n",
       "25%     NaN          NaN        7.000000      7.000000   \n",
       "50%     NaN          NaN        10.000000     9.000000   \n",
       "75%     NaN          NaN        12.000000     12.000000  \n",
       "max     NaN          NaN        37.000000     30.000000  "
      ]
     },
     "execution_count": 224,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.describe(include='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">n. of tokens</th>\n",
       "      <th colspan=\"3\" halign=\"left\">vocab. size</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>coarse class</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ABBR</th>\n",
       "      <td>689</td>\n",
       "      <td>8.011628</td>\n",
       "      <td>3.273761</td>\n",
       "      <td>665</td>\n",
       "      <td>7.732558</td>\n",
       "      <td>2.916180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DESC</th>\n",
       "      <td>10235</td>\n",
       "      <td>8.808090</td>\n",
       "      <td>3.840960</td>\n",
       "      <td>9892</td>\n",
       "      <td>8.512909</td>\n",
       "      <td>3.423120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENTY</th>\n",
       "      <td>13463</td>\n",
       "      <td>10.770400</td>\n",
       "      <td>3.706302</td>\n",
       "      <td>13017</td>\n",
       "      <td>10.413600</td>\n",
       "      <td>3.318335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HUM</th>\n",
       "      <td>13643</td>\n",
       "      <td>11.155356</td>\n",
       "      <td>4.423239</td>\n",
       "      <td>13065</td>\n",
       "      <td>10.682747</td>\n",
       "      <td>3.928046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>8327</td>\n",
       "      <td>9.972455</td>\n",
       "      <td>3.290923</td>\n",
       "      <td>8051</td>\n",
       "      <td>9.641916</td>\n",
       "      <td>2.968244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NUM</th>\n",
       "      <td>9278</td>\n",
       "      <td>10.354911</td>\n",
       "      <td>3.350834</td>\n",
       "      <td>9041</td>\n",
       "      <td>10.090402</td>\n",
       "      <td>3.006822</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             n. of tokens                      vocab. size             \\\n",
       "                      sum       mean       std         sum       mean   \n",
       "coarse class                                                            \n",
       "ABBR          689          8.011628   3.273761  665         7.732558    \n",
       "DESC          10235        8.808090   3.840960  9892        8.512909    \n",
       "ENTY          13463        10.770400  3.706302  13017       10.413600   \n",
       "HUM           13643        11.155356  4.423239  13065       10.682747   \n",
       "LOC           8327         9.972455   3.290923  8051        9.641916    \n",
       "NUM           9278         10.354911  3.350834  9041        10.090402   \n",
       "\n",
       "                        \n",
       "                   std  \n",
       "coarse class            \n",
       "ABBR          2.916180  \n",
       "DESC          3.423120  \n",
       "ENTY          3.318335  \n",
       "HUM           3.928046  \n",
       "LOC           2.968244  \n",
       "NUM           3.006822  "
      ]
     },
     "execution_count": 225,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "grouped = train_df.groupby('coarse class')\n",
    "grouped.agg([np.sum, np.mean, np.std])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">n. of tokens</th>\n",
       "      <th colspan=\"3\" halign=\"left\">vocab. size</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fine class</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>abb</th>\n",
       "      <td>171</td>\n",
       "      <td>10.687500</td>\n",
       "      <td>2.651258</td>\n",
       "      <td>159</td>\n",
       "      <td>9.937500</td>\n",
       "      <td>2.264766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>animal</th>\n",
       "      <td>1178</td>\n",
       "      <td>10.517857</td>\n",
       "      <td>3.210294</td>\n",
       "      <td>1130</td>\n",
       "      <td>10.089286</td>\n",
       "      <td>2.820624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>body</th>\n",
       "      <td>159</td>\n",
       "      <td>9.937500</td>\n",
       "      <td>3.434506</td>\n",
       "      <td>148</td>\n",
       "      <td>9.250000</td>\n",
       "      <td>2.380476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>city</th>\n",
       "      <td>1381</td>\n",
       "      <td>10.705426</td>\n",
       "      <td>2.818474</td>\n",
       "      <td>1337</td>\n",
       "      <td>10.364341</td>\n",
       "      <td>2.499806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>code</th>\n",
       "      <td>102</td>\n",
       "      <td>11.333333</td>\n",
       "      <td>3.082207</td>\n",
       "      <td>98</td>\n",
       "      <td>10.888889</td>\n",
       "      <td>2.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>color</th>\n",
       "      <td>395</td>\n",
       "      <td>9.875000</td>\n",
       "      <td>3.039800</td>\n",
       "      <td>388</td>\n",
       "      <td>9.700000</td>\n",
       "      <td>2.812062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>3928</td>\n",
       "      <td>10.820937</td>\n",
       "      <td>3.243932</td>\n",
       "      <td>3848</td>\n",
       "      <td>10.600551</td>\n",
       "      <td>2.871598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <td>1580</td>\n",
       "      <td>10.193548</td>\n",
       "      <td>2.880965</td>\n",
       "      <td>1533</td>\n",
       "      <td>9.890323</td>\n",
       "      <td>2.572501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cremat</th>\n",
       "      <td>2497</td>\n",
       "      <td>12.062802</td>\n",
       "      <td>4.002538</td>\n",
       "      <td>2426</td>\n",
       "      <td>11.719807</td>\n",
       "      <td>3.694491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>currency</th>\n",
       "      <td>29</td>\n",
       "      <td>7.250000</td>\n",
       "      <td>1.258306</td>\n",
       "      <td>29</td>\n",
       "      <td>7.250000</td>\n",
       "      <td>1.258306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <td>1992</td>\n",
       "      <td>9.137615</td>\n",
       "      <td>3.008338</td>\n",
       "      <td>1955</td>\n",
       "      <td>8.967890</td>\n",
       "      <td>2.751435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>def</th>\n",
       "      <td>2700</td>\n",
       "      <td>6.413302</td>\n",
       "      <td>2.345251</td>\n",
       "      <td>2664</td>\n",
       "      <td>6.327791</td>\n",
       "      <td>2.169466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>desc</th>\n",
       "      <td>3194</td>\n",
       "      <td>9.950156</td>\n",
       "      <td>3.803453</td>\n",
       "      <td>3035</td>\n",
       "      <td>9.454829</td>\n",
       "      <td>3.311719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dismed</th>\n",
       "      <td>938</td>\n",
       "      <td>9.106796</td>\n",
       "      <td>4.099050</td>\n",
       "      <td>914</td>\n",
       "      <td>8.873786</td>\n",
       "      <td>3.431914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dist</th>\n",
       "      <td>291</td>\n",
       "      <td>8.558824</td>\n",
       "      <td>2.862412</td>\n",
       "      <td>281</td>\n",
       "      <td>8.264706</td>\n",
       "      <td>2.573864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>event</th>\n",
       "      <td>656</td>\n",
       "      <td>11.714286</td>\n",
       "      <td>3.441534</td>\n",
       "      <td>625</td>\n",
       "      <td>11.160714</td>\n",
       "      <td>3.097077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>exp</th>\n",
       "      <td>518</td>\n",
       "      <td>7.400000</td>\n",
       "      <td>3.103995</td>\n",
       "      <td>506</td>\n",
       "      <td>7.228571</td>\n",
       "      <td>2.824178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>food</th>\n",
       "      <td>1059</td>\n",
       "      <td>10.281553</td>\n",
       "      <td>3.335541</td>\n",
       "      <td>1032</td>\n",
       "      <td>10.019417</td>\n",
       "      <td>3.022726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gr</th>\n",
       "      <td>2089</td>\n",
       "      <td>11.052910</td>\n",
       "      <td>3.554643</td>\n",
       "      <td>2025</td>\n",
       "      <td>10.714286</td>\n",
       "      <td>3.273500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ind</th>\n",
       "      <td>11069</td>\n",
       "      <td>11.506237</td>\n",
       "      <td>4.495166</td>\n",
       "      <td>10560</td>\n",
       "      <td>10.977131</td>\n",
       "      <td>3.959932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>instru</th>\n",
       "      <td>97</td>\n",
       "      <td>9.700000</td>\n",
       "      <td>3.128720</td>\n",
       "      <td>97</td>\n",
       "      <td>9.700000</td>\n",
       "      <td>3.128720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lang</th>\n",
       "      <td>148</td>\n",
       "      <td>9.250000</td>\n",
       "      <td>2.380476</td>\n",
       "      <td>142</td>\n",
       "      <td>8.875000</td>\n",
       "      <td>2.217356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>letter</th>\n",
       "      <td>88</td>\n",
       "      <td>9.777778</td>\n",
       "      <td>1.986063</td>\n",
       "      <td>86</td>\n",
       "      <td>9.555556</td>\n",
       "      <td>1.810463</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>manner</th>\n",
       "      <td>2653</td>\n",
       "      <td>9.612319</td>\n",
       "      <td>3.671769</td>\n",
       "      <td>2582</td>\n",
       "      <td>9.355072</td>\n",
       "      <td>3.351636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>money</th>\n",
       "      <td>836</td>\n",
       "      <td>11.774648</td>\n",
       "      <td>3.590452</td>\n",
       "      <td>803</td>\n",
       "      <td>11.309859</td>\n",
       "      <td>2.998054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mount</th>\n",
       "      <td>210</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>2.449490</td>\n",
       "      <td>200</td>\n",
       "      <td>9.523810</td>\n",
       "      <td>2.204973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ord</th>\n",
       "      <td>72</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>2.366432</td>\n",
       "      <td>70</td>\n",
       "      <td>11.666667</td>\n",
       "      <td>2.732520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>other</th>\n",
       "      <td>7353</td>\n",
       "      <td>10.031378</td>\n",
       "      <td>3.666098</td>\n",
       "      <td>7101</td>\n",
       "      <td>9.687585</td>\n",
       "      <td>3.313628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perc</th>\n",
       "      <td>339</td>\n",
       "      <td>12.555556</td>\n",
       "      <td>3.489912</td>\n",
       "      <td>326</td>\n",
       "      <td>12.074074</td>\n",
       "      <td>3.024591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>period</th>\n",
       "      <td>812</td>\n",
       "      <td>10.826667</td>\n",
       "      <td>3.508189</td>\n",
       "      <td>788</td>\n",
       "      <td>10.506667</td>\n",
       "      <td>3.223115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plant</th>\n",
       "      <td>126</td>\n",
       "      <td>9.692308</td>\n",
       "      <td>2.954788</td>\n",
       "      <td>120</td>\n",
       "      <td>9.230769</td>\n",
       "      <td>2.241794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product</th>\n",
       "      <td>465</td>\n",
       "      <td>11.071429</td>\n",
       "      <td>3.203602</td>\n",
       "      <td>451</td>\n",
       "      <td>10.738095</td>\n",
       "      <td>2.930575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>reason</th>\n",
       "      <td>1936</td>\n",
       "      <td>10.136126</td>\n",
       "      <td>4.491196</td>\n",
       "      <td>1859</td>\n",
       "      <td>9.732984</td>\n",
       "      <td>3.879353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>religion</th>\n",
       "      <td>33</td>\n",
       "      <td>8.250000</td>\n",
       "      <td>0.957427</td>\n",
       "      <td>33</td>\n",
       "      <td>8.250000</td>\n",
       "      <td>0.957427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speed</th>\n",
       "      <td>72</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>2.061553</td>\n",
       "      <td>69</td>\n",
       "      <td>7.666667</td>\n",
       "      <td>1.936492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sport</th>\n",
       "      <td>720</td>\n",
       "      <td>11.612903</td>\n",
       "      <td>4.013726</td>\n",
       "      <td>695</td>\n",
       "      <td>11.209677</td>\n",
       "      <td>3.265001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state</th>\n",
       "      <td>641</td>\n",
       "      <td>9.712121</td>\n",
       "      <td>2.265223</td>\n",
       "      <td>622</td>\n",
       "      <td>9.424242</td>\n",
       "      <td>2.105382</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>substance</th>\n",
       "      <td>375</td>\n",
       "      <td>9.146341</td>\n",
       "      <td>3.142936</td>\n",
       "      <td>364</td>\n",
       "      <td>8.878049</td>\n",
       "      <td>2.767626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol</th>\n",
       "      <td>110</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>2.097618</td>\n",
       "      <td>105</td>\n",
       "      <td>9.545455</td>\n",
       "      <td>1.752920</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>techmeth</th>\n",
       "      <td>443</td>\n",
       "      <td>11.657895</td>\n",
       "      <td>3.033830</td>\n",
       "      <td>431</td>\n",
       "      <td>11.342105</td>\n",
       "      <td>2.622937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>temp</th>\n",
       "      <td>81</td>\n",
       "      <td>10.125000</td>\n",
       "      <td>2.474874</td>\n",
       "      <td>81</td>\n",
       "      <td>10.125000</td>\n",
       "      <td>2.474874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>termeq</th>\n",
       "      <td>1026</td>\n",
       "      <td>11.032258</td>\n",
       "      <td>3.745872</td>\n",
       "      <td>983</td>\n",
       "      <td>10.569892</td>\n",
       "      <td>3.430673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <td>237</td>\n",
       "      <td>9.480000</td>\n",
       "      <td>2.785678</td>\n",
       "      <td>232</td>\n",
       "      <td>9.280000</td>\n",
       "      <td>2.508652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>veh</th>\n",
       "      <td>300</td>\n",
       "      <td>11.111111</td>\n",
       "      <td>3.202563</td>\n",
       "      <td>286</td>\n",
       "      <td>10.592593</td>\n",
       "      <td>2.763210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>volsize</th>\n",
       "      <td>110</td>\n",
       "      <td>8.461538</td>\n",
       "      <td>3.502746</td>\n",
       "      <td>107</td>\n",
       "      <td>8.230769</td>\n",
       "      <td>3.443686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>119</td>\n",
       "      <td>10.818182</td>\n",
       "      <td>3.458849</td>\n",
       "      <td>112</td>\n",
       "      <td>10.181818</td>\n",
       "      <td>2.786330</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>word</th>\n",
       "      <td>307</td>\n",
       "      <td>11.807692</td>\n",
       "      <td>4.223925</td>\n",
       "      <td>293</td>\n",
       "      <td>11.269231</td>\n",
       "      <td>3.758273</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           n. of tokens                      vocab. size                     \n",
       "                    sum       mean       std         sum       mean       std\n",
       "fine class                                                                   \n",
       "abb         171          10.687500  2.651258  159         9.937500   2.264766\n",
       "animal      1178         10.517857  3.210294  1130        10.089286  2.820624\n",
       "body        159          9.937500   3.434506  148         9.250000   2.380476\n",
       "city        1381         10.705426  2.818474  1337        10.364341  2.499806\n",
       "code        102          11.333333  3.082207  98          10.888889  2.666667\n",
       "color       395          9.875000   3.039800  388         9.700000   2.812062\n",
       "count       3928         10.820937  3.243932  3848        10.600551  2.871598\n",
       "country     1580         10.193548  2.880965  1533        9.890323   2.572501\n",
       "cremat      2497         12.062802  4.002538  2426        11.719807  3.694491\n",
       "currency    29           7.250000   1.258306  29          7.250000   1.258306\n",
       "date        1992         9.137615   3.008338  1955        8.967890   2.751435\n",
       "def         2700         6.413302   2.345251  2664        6.327791   2.169466\n",
       "desc        3194         9.950156   3.803453  3035        9.454829   3.311719\n",
       "dismed      938          9.106796   4.099050  914         8.873786   3.431914\n",
       "dist        291          8.558824   2.862412  281         8.264706   2.573864\n",
       "event       656          11.714286  3.441534  625         11.160714  3.097077\n",
       "exp         518          7.400000   3.103995  506         7.228571   2.824178\n",
       "food        1059         10.281553  3.335541  1032        10.019417  3.022726\n",
       "gr          2089         11.052910  3.554643  2025        10.714286  3.273500\n",
       "ind         11069        11.506237  4.495166  10560       10.977131  3.959932\n",
       "instru      97           9.700000   3.128720  97          9.700000   3.128720\n",
       "lang        148          9.250000   2.380476  142         8.875000   2.217356\n",
       "letter      88           9.777778   1.986063  86          9.555556   1.810463\n",
       "manner      2653         9.612319   3.671769  2582        9.355072   3.351636\n",
       "money       836          11.774648  3.590452  803         11.309859  2.998054\n",
       "mount       210          10.000000  2.449490  200         9.523810   2.204973\n",
       "ord         72           12.000000  2.366432  70          11.666667  2.732520\n",
       "other       7353         10.031378  3.666098  7101        9.687585   3.313628\n",
       "perc        339          12.555556  3.489912  326         12.074074  3.024591\n",
       "period      812          10.826667  3.508189  788         10.506667  3.223115\n",
       "plant       126          9.692308   2.954788  120         9.230769   2.241794\n",
       "product     465          11.071429  3.203602  451         10.738095  2.930575\n",
       "reason      1936         10.136126  4.491196  1859        9.732984   3.879353\n",
       "religion    33           8.250000   0.957427  33          8.250000   0.957427\n",
       "speed       72           8.000000   2.061553  69          7.666667   1.936492\n",
       "sport       720          11.612903  4.013726  695         11.209677  3.265001\n",
       "state       641          9.712121   2.265223  622         9.424242   2.105382\n",
       "substance   375          9.146341   3.142936  364         8.878049   2.767626\n",
       "symbol      110          10.000000  2.097618  105         9.545455   1.752920\n",
       "techmeth    443          11.657895  3.033830  431         11.342105  2.622937\n",
       "temp        81           10.125000  2.474874  81          10.125000  2.474874\n",
       "termeq      1026         11.032258  3.745872  983         10.569892  3.430673\n",
       "title       237          9.480000   2.785678  232         9.280000   2.508652\n",
       "veh         300          11.111111  3.202563  286         10.592593  2.763210\n",
       "volsize     110          8.461538   3.502746  107         8.230769   3.443686\n",
       "weight      119          10.818182  3.458849  112         10.181818  2.786330\n",
       "word        307          11.807692  4.223925  293         11.269231  3.758273"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped = train_df.groupby('fine class')\n",
    "grouped.agg([np.sum, np.mean, np.std])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fine class</th>\n",
       "      <th>percent</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>coarse class</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ABBR</th>\n",
       "      <td>86</td>\n",
       "      <td>1.577403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DESC</th>\n",
       "      <td>1162</td>\n",
       "      <td>21.313280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENTY</th>\n",
       "      <td>1250</td>\n",
       "      <td>22.927366</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HUM</th>\n",
       "      <td>1223</td>\n",
       "      <td>22.432135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>835</td>\n",
       "      <td>15.315481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NUM</th>\n",
       "      <td>896</td>\n",
       "      <td>16.434336</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              fine class    percent\n",
       "coarse class                       \n",
       "ABBR          86          1.577403 \n",
       "DESC          1162        21.313280\n",
       "ENTY          1250        22.927366\n",
       "HUM           1223        22.432135\n",
       "LOC           835         15.315481\n",
       "NUM           896         16.434336"
      ]
     },
     "execution_count": 227,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped_by_coarse = train_df.groupby('coarse class')\n",
    "grouped_by_coarse = grouped_by_coarse.count().drop(['n. of tokens', 'vocab. size'], 1)\n",
    "\n",
    "grouped_by_coarse['percent'] = [100 * x / grouped_by_coarse['fine class'].sum() \n",
    "                                for x in grouped_by_coarse['fine class']]\n",
    "grouped_by_coarse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this table we can see that a large percent of questions are tagged as ENTY and HUM. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coarse class</th>\n",
       "      <th>percent</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fine class</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>abb</th>\n",
       "      <td>16</td>\n",
       "      <td>0.293470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>animal</th>\n",
       "      <td>112</td>\n",
       "      <td>2.054292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>body</th>\n",
       "      <td>16</td>\n",
       "      <td>0.293470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>city</th>\n",
       "      <td>129</td>\n",
       "      <td>2.366104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>code</th>\n",
       "      <td>9</td>\n",
       "      <td>0.165077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>color</th>\n",
       "      <td>40</td>\n",
       "      <td>0.733676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>363</td>\n",
       "      <td>6.658107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <td>155</td>\n",
       "      <td>2.842993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cremat</th>\n",
       "      <td>207</td>\n",
       "      <td>3.796772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>currency</th>\n",
       "      <td>4</td>\n",
       "      <td>0.073368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <td>218</td>\n",
       "      <td>3.998533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>def</th>\n",
       "      <td>421</td>\n",
       "      <td>7.721937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>desc</th>\n",
       "      <td>321</td>\n",
       "      <td>5.887748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dismed</th>\n",
       "      <td>103</td>\n",
       "      <td>1.889215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dist</th>\n",
       "      <td>34</td>\n",
       "      <td>0.623624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>event</th>\n",
       "      <td>56</td>\n",
       "      <td>1.027146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>exp</th>\n",
       "      <td>70</td>\n",
       "      <td>1.283933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>food</th>\n",
       "      <td>103</td>\n",
       "      <td>1.889215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gr</th>\n",
       "      <td>189</td>\n",
       "      <td>3.466618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ind</th>\n",
       "      <td>962</td>\n",
       "      <td>17.644901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>instru</th>\n",
       "      <td>10</td>\n",
       "      <td>0.183419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lang</th>\n",
       "      <td>16</td>\n",
       "      <td>0.293470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>letter</th>\n",
       "      <td>9</td>\n",
       "      <td>0.165077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>manner</th>\n",
       "      <td>276</td>\n",
       "      <td>5.062362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>money</th>\n",
       "      <td>71</td>\n",
       "      <td>1.302274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mount</th>\n",
       "      <td>21</td>\n",
       "      <td>0.385180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ord</th>\n",
       "      <td>6</td>\n",
       "      <td>0.110051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>other</th>\n",
       "      <td>733</td>\n",
       "      <td>13.444607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perc</th>\n",
       "      <td>27</td>\n",
       "      <td>0.495231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>period</th>\n",
       "      <td>75</td>\n",
       "      <td>1.375642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plant</th>\n",
       "      <td>13</td>\n",
       "      <td>0.238445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product</th>\n",
       "      <td>42</td>\n",
       "      <td>0.770360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>reason</th>\n",
       "      <td>191</td>\n",
       "      <td>3.503302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>religion</th>\n",
       "      <td>4</td>\n",
       "      <td>0.073368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speed</th>\n",
       "      <td>9</td>\n",
       "      <td>0.165077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sport</th>\n",
       "      <td>62</td>\n",
       "      <td>1.137197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state</th>\n",
       "      <td>66</td>\n",
       "      <td>1.210565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>substance</th>\n",
       "      <td>41</td>\n",
       "      <td>0.752018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol</th>\n",
       "      <td>11</td>\n",
       "      <td>0.201761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>techmeth</th>\n",
       "      <td>38</td>\n",
       "      <td>0.696992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>temp</th>\n",
       "      <td>8</td>\n",
       "      <td>0.146735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>termeq</th>\n",
       "      <td>93</td>\n",
       "      <td>1.705796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <td>25</td>\n",
       "      <td>0.458547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>veh</th>\n",
       "      <td>27</td>\n",
       "      <td>0.495231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>volsize</th>\n",
       "      <td>13</td>\n",
       "      <td>0.238445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>11</td>\n",
       "      <td>0.201761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>word</th>\n",
       "      <td>26</td>\n",
       "      <td>0.476889</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            coarse class    percent\n",
       "fine class                         \n",
       "abb         16            0.293470 \n",
       "animal      112           2.054292 \n",
       "body        16            0.293470 \n",
       "city        129           2.366104 \n",
       "code        9             0.165077 \n",
       "color       40            0.733676 \n",
       "count       363           6.658107 \n",
       "country     155           2.842993 \n",
       "cremat      207           3.796772 \n",
       "currency    4             0.073368 \n",
       "date        218           3.998533 \n",
       "def         421           7.721937 \n",
       "desc        321           5.887748 \n",
       "dismed      103           1.889215 \n",
       "dist        34            0.623624 \n",
       "event       56            1.027146 \n",
       "exp         70            1.283933 \n",
       "food        103           1.889215 \n",
       "gr          189           3.466618 \n",
       "ind         962           17.644901\n",
       "instru      10            0.183419 \n",
       "lang        16            0.293470 \n",
       "letter      9             0.165077 \n",
       "manner      276           5.062362 \n",
       "money       71            1.302274 \n",
       "mount       21            0.385180 \n",
       "ord         6             0.110051 \n",
       "other       733           13.444607\n",
       "perc        27            0.495231 \n",
       "period      75            1.375642 \n",
       "plant       13            0.238445 \n",
       "product     42            0.770360 \n",
       "reason      191           3.503302 \n",
       "religion    4             0.073368 \n",
       "speed       9             0.165077 \n",
       "sport       62            1.137197 \n",
       "state       66            1.210565 \n",
       "substance   41            0.752018 \n",
       "symbol      11            0.201761 \n",
       "techmeth    38            0.696992 \n",
       "temp        8             0.146735 \n",
       "termeq      93            1.705796 \n",
       "title       25            0.458547 \n",
       "veh         27            0.495231 \n",
       "volsize     13            0.238445 \n",
       "weight      11            0.201761 \n",
       "word        26            0.476889 "
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped = train_df.groupby('fine class')\n",
    "grouped = grouped.count().drop(['n. of tokens', 'vocab. size'], 1)\n",
    "grouped['percent'] = [100 * x / grouped['coarse class'].sum() for x in grouped['coarse class']]\n",
    "grouped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like ind and other are common tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"4\" halign=\"left\">fine class</th>\n",
       "      <th colspan=\"8\" halign=\"left\">n. of tokens</th>\n",
       "      <th colspan=\"8\" halign=\"left\">vocab. size</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>unique</th>\n",
       "      <th>top</th>\n",
       "      <th>freq</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>coarse class</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ABBR</th>\n",
       "      <td>86</td>\n",
       "      <td>2</td>\n",
       "      <td>exp</td>\n",
       "      <td>70</td>\n",
       "      <td>86.0</td>\n",
       "      <td>8.011628</td>\n",
       "      <td>3.273761</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>10.75</td>\n",
       "      <td>18.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>7.732558</td>\n",
       "      <td>2.916180</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DESC</th>\n",
       "      <td>1162</td>\n",
       "      <td>4</td>\n",
       "      <td>def</td>\n",
       "      <td>421</td>\n",
       "      <td>1162.0</td>\n",
       "      <td>8.808090</td>\n",
       "      <td>3.840960</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>11.00</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1162.0</td>\n",
       "      <td>8.512909</td>\n",
       "      <td>3.423120</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENTY</th>\n",
       "      <td>1250</td>\n",
       "      <td>22</td>\n",
       "      <td>other</td>\n",
       "      <td>217</td>\n",
       "      <td>1250.0</td>\n",
       "      <td>10.770400</td>\n",
       "      <td>3.706302</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>13.00</td>\n",
       "      <td>31.0</td>\n",
       "      <td>1250.0</td>\n",
       "      <td>10.413600</td>\n",
       "      <td>3.318335</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>30.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HUM</th>\n",
       "      <td>1223</td>\n",
       "      <td>4</td>\n",
       "      <td>ind</td>\n",
       "      <td>962</td>\n",
       "      <td>1223.0</td>\n",
       "      <td>11.155356</td>\n",
       "      <td>4.423239</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>13.00</td>\n",
       "      <td>34.0</td>\n",
       "      <td>1223.0</td>\n",
       "      <td>10.682747</td>\n",
       "      <td>3.928046</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>30.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>835</td>\n",
       "      <td>5</td>\n",
       "      <td>other</td>\n",
       "      <td>464</td>\n",
       "      <td>835.0</td>\n",
       "      <td>9.972455</td>\n",
       "      <td>3.290923</td>\n",
       "      <td>4.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>12.00</td>\n",
       "      <td>29.0</td>\n",
       "      <td>835.0</td>\n",
       "      <td>9.641916</td>\n",
       "      <td>2.968244</td>\n",
       "      <td>4.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NUM</th>\n",
       "      <td>896</td>\n",
       "      <td>13</td>\n",
       "      <td>count</td>\n",
       "      <td>363</td>\n",
       "      <td>896.0</td>\n",
       "      <td>10.354911</td>\n",
       "      <td>3.350834</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>12.00</td>\n",
       "      <td>29.0</td>\n",
       "      <td>896.0</td>\n",
       "      <td>10.090402</td>\n",
       "      <td>3.006822</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>25.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             fine class                    n. of tokens                       \\\n",
       "                  count unique    top freq        count       mean       std   \n",
       "coarse class                                                                   \n",
       "ABBR          86         2      exp    70   86.0         8.011628   3.273761   \n",
       "DESC          1162       4      def    421  1162.0       8.808090   3.840960   \n",
       "ENTY          1250       22     other  217  1250.0       10.770400  3.706302   \n",
       "HUM           1223       4      ind    962  1223.0       11.155356  4.423239   \n",
       "LOC           835        5      other  464  835.0        9.972455   3.290923   \n",
       "NUM           896        13     count  363  896.0        10.354911  3.350834   \n",
       "\n",
       "                                          vocab. size                       \\\n",
       "              min  25%   50%    75%   max       count       mean       std   \n",
       "coarse class                                                                 \n",
       "ABBR          4.0  6.0  7.0   10.75  18.0  86.0        7.732558   2.916180   \n",
       "DESC          3.0  6.0  8.0   11.00  37.0  1162.0      8.512909   3.423120   \n",
       "ENTY          5.0  8.0  10.0  13.00  31.0  1250.0      10.413600  3.318335   \n",
       "HUM           3.0  8.0  11.0  13.00  34.0  1223.0      10.682747  3.928046   \n",
       "LOC           4.0  8.0  10.0  12.00  29.0  835.0       9.641916   2.968244   \n",
       "NUM           5.0  8.0  10.0  12.00  29.0  896.0       10.090402  3.006822   \n",
       "\n",
       "                                          \n",
       "              min  25%   50%   75%   max  \n",
       "coarse class                              \n",
       "ABBR          4.0  6.0  7.0   10.0  17.0  \n",
       "DESC          3.0  6.0  8.0   10.0  28.0  \n",
       "ENTY          5.0  8.0  10.0  12.0  30.0  \n",
       "HUM           3.0  8.0  10.0  13.0  30.0  \n",
       "LOC           4.0  8.0  9.0   11.0  25.0  \n",
       "NUM           5.0  8.0  10.0  12.0  25.0  "
      ]
     },
     "execution_count": 229,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.groupby('coarse class').describe(include='all').dropna(axis=1, how='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<matplotlib.axes._subplots.AxesSubplot object at 0x0000020EAD9EBD68>,\n",
       "        <matplotlib.axes._subplots.AxesSubplot object at 0x0000020EAF799898>]], dtype=object)"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEICAYAAAC55kg0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHGBJREFUeJzt3X+QHOV95/H3x4KAgrExCO9JSLD4IlwFyMjHFqbKcTKJ\ncVAMFcCVInDEwEEhU+b8o7K5IHyuwNnRGScWTmwf+IQhgor5oTPGcAZsC8pT2BULIjjF4peCgKWQ\nLJCNALE4p2PF9/7oZ6PWaHZ3NNsz07P9eVVNbffTP+bbu8+z3+fp7plWRGBmZtX0tl4HYGZmveMk\nYGZWYU4CZmYV5iRgZlZhTgJmZhXmJGBmVmFOAiUk6a8k/UrSiwXsqyZpcxFxmfWCpKsk/UMH9vu4\npFrR++03TgIlI+lIYBg4NiL+XZPl/qduVoCIOC4i6r2Oo9ecBMrnSODliNjW60DMbOZzEmiTpBFJ\nfy7p55Jek3S7pANb3Padkm6W9EtJz0v6vKS3SToFWAPMkzQqaVXDdgcB9+WWj0qaJ+kASX8r6Rfp\n9beSDpjgvT8t6QlJ89P86ZLWS3pV0j9Kel8rxyhpjqTvp+22S/qJJNenGU7S5ZK+01D2d5K+lqbn\nSbo71YlNki7JrTdL0uckPSPpdUmPSFqQ28cLknak8g81vPWBqf69LulRSSe0GO+E9TTV71PS9Ku5\nNvWGpJA0mJZN2EZmhIjwq40XMAI8DMwDDgWeBC5tcdubgbuAg4FB4F+Ai9OyGrB5km33Wg58AVgL\nvBs4HPhH4IuN6wN/CTwKHJ7m3w9sAz4AzAIuSMd1wFTHCHwJ+Cawf3p9CFCv/y5+dfYFHAX8Gjg4\nzc8CtgInp/kHgWuBA4HFwC+B30/L/guwAXgvIOAE4LC07E+Bw4D9yE6HvggcmJZdBbwJ/HGqa38O\nPAfs30K8E9bTVL9PabLNf0/Hsf9UbWQmvHoeQL++UkX409z8XwPfbGG7WcD/IzvnP172CaCepttJ\nAs8AH83NnwqM5NbfAlwD/BR4Z26960jJIle2EfjdqY6RLPHcBfxWr/8WfnX3lerR+Wn6I8AzaXoB\nsGs8QaSyLwGr0vRG4IwW3+MV4IQ0fRWwNrfsbWSJ50Mt7GfCetosCQB/ksrHO0qTtpGZ8PLwfXry\nd+/8Gnh7C9vMIethPJ8rex44YhpxzGuyv3m5+UOApcCXIuK1XPlRwHAa5r4q6VWyhpzfdqJj/Btg\nE/AjSc9KWjaN+K2/3AKcm6b/Y5qHrN5sj4jXc+vm6/YCsg7LXtJpxyfTacdXgXeStZVxL4xPRMRb\nwGb2rKcTabmeSno/8A3grIj4ZSpupY30NSeB7vsV2dD2qFzZkWS99VY0+9rXXzTZ3y9y868ApwN/\nL+mDufIXgOURcUju9ZsRceuUQUS8HhHDEfEe4I+AP5P04RaPwfrb/wJq6brSWexOAr8ADpV0cG7d\nfN1+Afj3jTtL5///AjgbeFdEHAK8RnbKaNyC3PpvA+azZx1vqtV6KundwPeAyyLi/+QWtd1G+oWT\nQJdFxC5gNbBc0sGSjgL+DGj1PuiXgMMkvTNXdivweUmHS5pDdu5/j/1FdivcecB3JZ2Uiq8HLpX0\nAWUOknRaQyNuKl0s+y1JImuwu4C3WjwG62Opl1wH/h54LiKeTOUvkF2P+pKkA9MF1IvZXRe/BXxR\n0sJU394n6TCya2NjZNcP9pP0l8A7Gt72REkfk7Qf8FlgJ9l1sEm1Uk/TPr8D/ENErG7YRdttpF84\nCXSApCPTXQZHTrDKp4A3gGfJzq/eAtzYyr4j4imyf/rPpuHpPOCvgHXAz8kuvD2ayhq3XQNcBPxv\nSf8hItYBl5ANgV8hGzZf2OJhLgTuB0aBnwHXRsSPW9zW+t8twCnsHgWMO5fsZodfAHcCV0bE/WnZ\nNWQdoB8BO4AbgNnAD4EfkN0g8Tzwf8md/knuIjtf/wrwceBjEfEmgKT7JH1ugjhbqafzyS4YfzZ3\nh9CopCOn2Ub6wvhVcjMzqyCPBMzMKsxJwMyswpwEzMwqzEnAzKzC9ut1AFOZM2dODA4O9joM3njj\nDQ466KBeh7GXssYF5YntkUce+VVEHN7rOFpVljo/mbL8badjph9Dq/W+9ElgcHCQdevW9ToM6vU6\ntVqt12HspaxxQXlik/T81GuVR1nq/GTK8redjpl+DK3We58OMjOrMCcBM7MKcxIwM6swJwEzswpz\nEjAzqzAnATOzCnMSMDOrMCcBM7MKcxIwM6uw0n9iuAwGl93D8KIxLlx2T8vbjFx9WgcjMuuswSnq\nerP24DrfnyqXBKaq3GZmVeLTQWZmFeYkYGZWYU4CZmYV5iRgZlZhTgJmZhXmJGBmVmFOAmZNSLpR\n0jZJj+XKbpe0Pr1GJK1P5YOS/jW37Ju5bU6UtEHSJklfk6ReHI/ZRCr3OQGzFq0CvgHcPF4QEX8y\nPi1pBfBabv1nImJxk/1cB1wCPATcCywB7utAvGZt8UjArImIeBDY3mxZ6s2fDdw62T4kzQXeERFr\nIyLIEsqZRcdqNh1TjgQk3QicDmyLiONT2e3Ae9MqhwCvRsRiSYPAk8DGtGxtRFyatjmRrHc1m6xH\n9JnUMMz6zYeAlyLi6VzZ0en00GvA5yPiJ8ARwObcOptT2V4kLQWWAgwMDFCv1zsRd8uGF41Nunxg\n9t7r9DrmfTU6Otp3MTcq4hhaOR20Cg+LzfLOZc9RwFbgyIh4OXV2vifpuH3ZYUSsBFYCDA0NRa1W\nKyrWtkz1PVnDi8ZYsWHPfx8j59U6GFHx6vU6vf49T1cRxzDl6SAPi812k7Qf8DHg9vGyiNgZES+n\n6UeAZ4BjgC3A/Nzm81OZWWlM98Jw4cNi6OzQeKph7kSaDX8n061hZpmHtGWObRpOAZ6KiH+rz5IO\nB7ZHxC5J7wEWAs9GxHZJOySdTDYCPh/4ek+iNpvAdJNA4cNi6OzQeF++Djqv2fB3Mt0aGpd5SFvm\n2KYi6VagBsyRtBm4MiJuAM5h75Hv7wBfkPQm8BZwaUSMj54/ye5rYffRg1Og/uZcm0zbSSA3LD5x\nvCwidgI70/Qjkjwstr4UEedOUH5hk7I7gDsmWH8dcHyhwZkVaDq3iDYdFkualabzw+KtwA5JJ6fr\nCOcDd03jvc3MrABTJoE0LP4Z8F5JmyVdnBZNNCz+ebom8B32HhZ/C9hEduHMdwaZmfXYlKeDPCw2\nM5u5/IlhM7MKcxIwM6swJwEzswpzEjAzqzAnATOzCnMSMDOrMCcBM7MKcxIwM6swJwEzswpzEjAz\nqzAnATOzCnMSMDOrMCcBM7MKcxIwM6swJwEzswpzEjAzqzAnAbMmJN0oaZukx3JlV0naIml9en00\nt+wKSZskbZR0aq78REkb0rKvpcermpWGk4BZc6uAJU3KvxoRi9PrXgBJx5I9bvW4tM2148/aBq4D\nLiF73vbCCfZp1jNOAmZNRMSDwPYpV8ycAdwWETsj4jmy52ifJGku8I6IWBsRAdwMnNmZiM3aM+Uz\nhiXdCJwObIuI41PZVWS9m1+m1T6X6xVdAVwM7AI+HRE/TOUnkvWuZgP3Ap9JDcOsn3xK0vnAOmA4\nIl4BjgDW5tbZnMreTNON5XuRtBRYCjAwMEC9Xi8s4OFFY4Xta9zA7L33W2TM3TA6Otp3MTcq4him\nTAJk/7i/QdaLyftqRHwlX9AwLJ4H3C/pmIjYxe5h8UNkSWAJcN+0ojfrruuALwKRfq4ALipixxGx\nElgJMDQ0FLVarYjdAnDhsnsK29e44UVjrNiw57+PkfNqhb9PJ9XrdYr8PfdCEccw5ekgD4vNMhHx\nUkTsioi3gOuBk9KiLcCC3KrzU9mWNN1YblYarYwEJtKRYTGUc2jcbPg7mW4NM8s8pC1zbO2QNDci\ntqbZs4DxO4fuBm6RdA3ZCHgh8HBE7JK0Q9LJZCPg84Gvdztus8m0mwQ6NiyGcg6Nmw1/J9OtoXGZ\nh7Rljm0qkm4FasAcSZuBK4GapMVk9X4E+ARARDwuaTXwBDAGXJZOgQJ8kt3Xwu7Dp0CtZNpKAhHx\n0vi0pOuB76dZD4ttRoiIc5sU3zDJ+suB5U3K1wHHFxiaWaHaukU0neMf1zgsPkfSAZKOZveweCuw\nQ9LJ6cMy5wN3TSNuMzMrQCu3iHpYbGY2Q02ZBDwsNjObufyJYTOzCnMSMDOrMCcBM7MKcxIwM6sw\nJwEzswpzEjAzqzAnATOzCnMSMDOrMCcBM7MKcxIwM6swJwEzswpzEjAzqzAnATOzCnMSMDOrMCcB\nM7MKcxIwM6swJwGzJiTdKGmbpMdyZX8j6SlJP5d0p6RDUvmgpH+VtD69vpnb5kRJGyRtkvS19HhV\ns9JwEjBrbhWwpKFsDXB8RLwP+BfgityyZyJicXpdmiu/DriE7HnbC5vs06ynpkwC7hFZFUXEg8D2\nhrIfRcRYml0LzJ9sH5LmAu+IiLUREcDNwJmdiNesXVM+Y5isR/QNsgo8bg1wRUSMSfoyWY/o8rTs\nmYhY3GQ/4z2ih4B7yXpEfti89auLgNtz80dLWg+8Bnw+In4CHAFszq2zOZXtRdJSYCnAwMAA9Xq9\nsECHF41NvdI+Gpi9936LjLkbRkdH+y7mRkUcQysPmn9Q0mBD2Y9ys2uBP55sH/keUZof7xE5CVjf\nkfRfgTHg26loK3BkRLws6UTge5KO25d9RsRKYCXA0NBQ1Gq1wuK9cNk9he1r3PCiMVZs2PPfx8h5\ntcLfp5Pq9TpF/p57oYhjaGUkMJVCe0RQzl5Rs57PZLrVwyhzb6bMsbVL0oXA6cCH0ykeImInsDNN\nPyLpGeAYYAt7njKan8rMSmNaSaATPSIoZ6+oWc9nMt3qFZW5N1Pm2NohaQnwF8DvRsSvc+WHA9sj\nYpek95BdAH42IrZL2iHpZLLToOcDX+9F7GYTaTsJuEdkM5mkW4EaMEfSZuBKsmtfBwBr0n0Na9Od\nQL8DfEHSm8BbwKURMX5R+ZNk19Vmk53+9ClQK5W2koB7RDbTRcS5TYpvmGDdO4A7Jli2Dji+wNDM\nCjVlEnCPyMxs5mrl7iD3iMzMZih/YtjMrMKcBMzMKsxJwMyswpwEzMwqzEnAzKzCnATMzCrMScDM\nrMKcBMzMKsxJwMyswpwEzMwqzEnAzKzCnATMzCrMScDMrMKcBMzMKsxJwMyswpwEzMwqzEnAzKzC\nnATMmpB0o6Rtkh7LlR0qaY2kp9PPd+WWXSFpk6SNkk7NlZ8oaUNa9jWl57GalYWTgFlzq4AlDWXL\ngAciYiHwQJpH0rHAOcBxaZtrJc1K21wHXAIsTK/GfZr11JRJwD0iq6KIeBDY3lB8BnBTmr4JODNX\nfltE7IyI54BNwEmS5gLviIi1ERHAzbltzEphygfNk/WIvkFWgceN94iulrQszV/e0COaB9wv6ZiI\n2MXuHtFDwL1kPaL7ijoQsy4YiIitafpFYCBNHwGsza23OZW9maYby/ciaSmwFGBgYIB6vV5Y0MOL\nxgrb17iB2Xvvt8iYu2F0dLTvYm5UxDFMmQQi4kFJgw3FZwC1NH0TUAcuJ9cjAp6TNN4jGiH1iAAk\njfeInASsL0VESIoC97cSWAkwNDQUtVqtqF1z4bJ7CtvXuOFFY6zYsOe/j5HzaoW/TyfV63WK/D33\nQhHH0MpIoJmO9YignL2iZj2fyXSrh1Hm3kyZY2vTS5LmRsTWdKpnWyrfAizIrTc/lW1J043lZqXR\nbhL4N0X3iNI+S9cratbzmUy3ekVl7s2UObY23Q1cAFydft6VK79F0jVkp0EXAg9HxC5JOySdTHYa\n9Hzg690P22xi7SYB94hsRpN0K9kpzzmSNgNXkv3zXy3pYuB54GyAiHhc0mrgCWAMuCxdBwP4JNl1\ntdlkpz99CtRKpd0k4B6RzWgRce4Eiz48wfrLgeVNytcBxxcYmlmhpkwC7hGZmc1crdwd5B6RmdkM\n5U8Mm5lVmJOAmVmFOQmYmVWYk4CZWYU5CZiZVZiTgJlZhTkJmJlVmJOAmVmFOQmYmVWYk4CZWYU5\nCZiZVZiTgJlZhTkJmJlVmJOAmVmFOQmYmVWYk4CZWYVN+0HzvTLY5gPjzcxsN48EzPaBpPdKWp97\n7ZD0WUlXSdqSK/9obpsrJG2StFHSqb2M36xR20nAjcGqKCI2RsTiiFgMnAj8GrgzLf7q+LKIuBdA\n0rHAOcBxwBLgWkmzehG7WTNtnw6KiI3AYoBUqbeQNYb/RNYYvpJfv6ExzAPul3RM7kH0Zv3mw8Az\nEfG8pInWOQO4LSJ2As9J2gScBPysSzGaTaqoawJuDFZF5wC35uY/Jel8YB0wHBGvAEcAa3PrbE5l\ne5C0FFgKMDAwQL1eLyzI4UVjhe1r3MDsvfdbZMzdMDo62ncxNyriGIpKAoU1BmitQXSiYk+mWaWf\nTLcqV5krcpljmy5JvwH8EXBFKroO+CIQ6ecK4KJW9xcRK4GVAENDQ1Gr1QqL9cIO3EQxvGiMFRsa\n/n1seGOf9zNy9WkFRbTv6vU6Rf6ee6GIY5h2Eii6MUBrDaITFXsyTSv9JEbOq3UumJwyV+Qyx1aA\nPwQejYiXAMZ/Aki6Hvh+mt0CLMhtNz+VmZVCEXcH7dUYImJXRLwFXE92ygfcGGxmOZfc6FfS3Nyy\ns4DH0vTdwDmSDpB0NLAQeLhrUZpNoYjTQXs1hojYmmYbG8Mtkq4huzDsxmB9SdJBwEeAT+SK/1rS\nYrIR8Mj4soh4XNJq4AlgDLjMN0NYmUwrCbgxWBVFxBvAYQ1lH59k/eXA8k7HZdaOaSUBNwYzs/7m\nTwybmVWYk4CZWYU5CZiZVZiTgJlZhTkJmJlVmJOAmVmFOQmYmVWYk4CZWYU5CZiZVZiTgJlZhTkJ\nmJlVmJOAmVmFOQmYmVWYk4CZWYU5CZiZVZiTgJlZhTkJmJlVmJOAmVmFFfGgeWticNk9+7zNyNWn\ndSASK5qkEeB1YBcwFhFDkg4FbgcGyZ6tfXZEvJLWvwK4OK3/6Yj4YQ/CNmtqWiMBSSOSNkhaL2ld\nKjtU0hpJT6ef78qtf4WkTZI2Sjp1usGb9dDvRcTiiBhK88uAByJiIfBAmkfSscA5wHHAEuBaSbN6\nEbBZM0WcDnJjMIMzgJvS9E3Ambny2yJiZ0Q8B2wCTupBfGZNdeJ00BlALU3fBNSBy8k1BuA5SeON\n4WcdiMGskwK4X9Iu4H9GxEpgICK2puUvAgNp+ghgbW7bzalsD5KWAksBBgYGqNfrhQU7vGissH2N\nG5hdzH6LPM59NTo62tP3L0IRxzDdJFB4Y4DWGkQnKvZkiqr0k2nnj1nmilzm2KbptyNii6R3A2sk\nPZVfGBEhKfZlh6ntrAQYGhqKWq1WWLAXtnF9airDi8ZYsaGAPuSGN9rarIjrZ/V6nSJ/z71QxDFM\n969YeGNI203ZIDpRsSdTWKWfxMh5tX3epswVucyxTUdEbEk/t0m6k2xE+5KkuRGxVdJcYFtafQuw\nILf5/FRmVgrTuiaQbwzAHo0BwI3BZhpJB0k6eHwa+APgMeBu4IK02gXAXWn6buAcSQdIOhpYCDzc\n3ajNJtZ2EnBjsIoaAH4q6Z/J6u89EfED4GrgI5KeBk5J80TE48Bq4AngB8BlEbGrJ5GbNTGd8xsD\nwJ2SxvdzS0T8QNI/AaslXQw8D5wNWWOQNN4YxnBjsD4UEc8CJzQpfxn48ATbLAeWdzg0s7a0nQTc\nGMzM+p+/NsLMrMKcBMzMKsxJwMyswvwFcmZ9pJ0vJjSbjEcCZmYV5iRgZlZhTgJmZhXmJGBmVmFO\nAmZmFeYkYGZWYU4CZmYV5iRgZlZhTgJmZhXmJGBmVmFOAmZmFeYkYGZWYU4CZmYV5iRgZlZh03nQ\n/AJJP5b0hKTHJX0mlV8laYuk9en10dw2V0jaJGmjpFOLOACzbnK9t5lmOs8TGAOGI+JRSQcDj0ha\nk5Z9NSK+kl9Z0rHAOcBxwDzgfknH+GHz1mdc721GaXskEBFbI+LRNP068CRwxCSbnAHcFhE7I+I5\nYBNwUrvvb9YLrvc20xTyZDFJg8D7gYeADwKfknQ+sI6s1/QKWUNZm9tsMxM0HklLgaUAAwMD1Ov1\nvdYZXjRWROgtG5jd+fdsdpxTGR0dbWu7bihzbEUost63Uueh+/V+It1oD5Mpol7NhPpZxDFMOwlI\nejtwB/DZiNgh6Trgi0CknyuAi/ZlnxGxElgJMDQ0FLVaba91LuzyY/aGF42xYkNnn8Y5cl5tn7ep\n1+s0+/2UQZljm66i630rdR66X+8n0o32MJl22kqjmVA/iziGad0dJGl/sobw7Yj4LkBEvBQRuyLi\nLeB6dg99twALcpvPT2VmfcX13maStlO5JAE3AE9GxDW58rkRsTXNngU8lqbvBm6RdA3ZBbKFwMPt\nvv9M1M5DxFctOagDkdhEXO9tppnOeO6DwMeBDZLWp7LPAedKWkw2LB4BPgEQEY9LWg08QXaHxWW+\nQ8L6kOu9zShtJ4GI+CmgJovunWSb5cDydt/TrNdc722m8SeGzcwqzEnAzKzCnATMzCqsdzf6mpm1\nqZ076UauPq0DkfQ/jwTMzCrMScDMrMKcBMzMKsxJwMyswpwEzMwqzEnAzKzCnATMzCrMScDMrML8\nYbE+t2HLa209aMQfnDEz8EjAzKzSnATMzCrMScDMrMKcBMzMKsxJwMyswnx3kJlVQuPXTw8vGpvy\nzroq3EXX9SQgaQnwd8As4FsRcXW3YzB/H3s3uc73ryq0k66eDpI0C/gfwB8CxwLnSjq2mzGYdZPr\nvJVdt0cCJwGbIuJZAEm3AWcAT3Q5DmvDvvaKWhluN9NvPakpuM5XTDujh3atWnLQtPfR7SRwBPBC\nbn4z8IHGlSQtBZam2VFJG7sQ26Q+DXOAX/U6jkZljQvaj01fLjyUowrfY+v6ts5Ppsz1rlUz4Rh+\n78uTHkNL9b6UF4YjYiWwstdx5ElaFxFDvY6jUVnjgnLHVjZlrPOTmQl/Wx9Dptu3iG4BFuTm56cy\ns5nKdd5KrdtJ4J+AhZKOlvQbwDnA3V2OwaybXOet1Lp6OigixiT9Z+CHZLfL3RgRj3czhmko61C9\nrHFBuWPrij6v85OZCX9bHwOgiCgiEDMz60P+2ggzswpzEjAzqzAngQaSbpS0TdJjubJDJa2R9HT6\n+a4exbZA0o8lPSHpcUmfKUt8kg6U9LCkf06x/beyxGbTU+Y20Yoyt5tWdbJ9OQnsbRWwpKFsGfBA\nRCwEHkjzvTAGDEfEscDJwGXpKwjKEN9O4Pcj4gRgMbBE0sklic2mZxXlbROtKHO7aVXn2ldE+NXw\nAgaBx3LzG4G5aXousLHXMaZY7gI+Urb4gN8EHiX7ZGypYvOr7b9pX7SJFo+llO1mH+IvtH15JNCa\ngYjYmqZfBAZ6GQyApEHg/cBDlCQ+SbMkrQe2AWsiojSxWeH68u9axnbTqk61LyeBfRRZyu3pfbWS\n3g7cAXw2Inbkl/UyvojYFRGLyT4Ve5Kk48sSm3VOv/xdy9puWtWp9uUk0JqXJM0FSD+39SoQSfuT\nVeRvR8R3yxYfQES8CvyY7DxyqWKzwvTV37Uf2k2rim5fTgKtuRu4IE1fQHZOseskCbgBeDIirskt\n6nl8kg6XdEiank12zvWpMsRmHdE3f9cyt5tWdbJ9+RPDDSTdCtTIvmb2JeBK4HvAauBI4Hng7IjY\n3oPYfhv4CbABeCsVf47s/GZP45P0PuAmsq9GeBuwOiK+IOmwXsdm01PmNtGKMrebVnWyfTkJmJlV\nmE8HmZlVmJOAmVmFOQmYmVWYk4CZWYU5CZiZVZiTgJlZhTkJmJlV2P8H10SX2MutXvUAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20eac220b70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_df.hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_label_dist_dict = {coarse_class: Counter(train_df[train_df['coarse class'] == coarse_class]['fine class']) for \n",
    "              coarse_class in train_df['coarse class'].unique()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distribution of flat labels per coarse label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Coarse</th>\n",
       "      <th>Fine</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">DESC</th>\n",
       "      <th>manner</th>\n",
       "      <td>276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>def</th>\n",
       "      <td>421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>reason</th>\n",
       "      <td>191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>desc</th>\n",
       "      <td>274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"22\" valign=\"top\">ENTY</th>\n",
       "      <th>cremat</th>\n",
       "      <td>207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>animal</th>\n",
       "      <td>112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>event</th>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>other</th>\n",
       "      <td>217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>letter</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>religion</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>food</th>\n",
       "      <td>103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>color</th>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>termeq</th>\n",
       "      <td>93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>body</th>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dismed</th>\n",
       "      <td>103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>product</th>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>substance</th>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sport</th>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plant</th>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>techmeth</th>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>instru</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>word</th>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lang</th>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol</th>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>veh</th>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>currency</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">ABBR</th>\n",
       "      <th>exp</th>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abb</th>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">HUM</th>\n",
       "      <th>ind</th>\n",
       "      <td>962</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gr</th>\n",
       "      <td>189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>desc</th>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"13\" valign=\"top\">NUM</th>\n",
       "      <th>date</th>\n",
       "      <td>218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>money</th>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>period</th>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>volsize</th>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>other</th>\n",
       "      <td>52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speed</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perc</th>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>code</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dist</th>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>temp</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ord</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">LOC</th>\n",
       "      <th>state</th>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>other</th>\n",
       "      <td>464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>city</th>\n",
       "      <td>129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mount</th>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    0\n",
       "Coarse Fine          \n",
       "DESC   manner     276\n",
       "       def        421\n",
       "       reason     191\n",
       "       desc       274\n",
       "ENTY   cremat     207\n",
       "       animal     112\n",
       "       event      56 \n",
       "       other      217\n",
       "       letter     9  \n",
       "       religion   4  \n",
       "       food       103\n",
       "       color      40 \n",
       "       termeq     93 \n",
       "       body       16 \n",
       "       dismed     103\n",
       "       product    42 \n",
       "       substance  41 \n",
       "       sport      62 \n",
       "       plant      13 \n",
       "       techmeth   38 \n",
       "       instru     10 \n",
       "       word       26 \n",
       "       lang       16 \n",
       "       symbol     11 \n",
       "       veh        27 \n",
       "       currency   4  \n",
       "ABBR   exp        70 \n",
       "       abb        16 \n",
       "HUM    ind        962\n",
       "       gr         189\n",
       "       title      25 \n",
       "       desc       47 \n",
       "NUM    date       218\n",
       "       count      363\n",
       "       money      71 \n",
       "       period     75 \n",
       "       volsize    13 \n",
       "       other      52 \n",
       "       speed      9  \n",
       "       perc       27 \n",
       "       code       9  \n",
       "       dist       34 \n",
       "       temp       8  \n",
       "       ord        6  \n",
       "       weight     11 \n",
       "LOC    state      66 \n",
       "       other      464\n",
       "       country    155\n",
       "       city       129\n",
       "       mount      21 "
      ]
     },
     "execution_count": 232,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class_tuples = [(coarse, fine) for coarse in train_label_dist_dict for fine in train_label_dist_dict[coarse].keys()]\n",
    "counts = [train_label_dist_dict[coarse][fine] for coarse, fine in class_tuples]\n",
    "index = pd.MultiIndex.from_tuples(class_tuples, names=['Coarse', 'Fine'])\n",
    "train_label_dist_df = pd.DataFrame(counts, index=index)\n",
    "train_label_dist_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q1.3 Baseline classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a baseline classifier for coarse labels by using simple rules. For example, if a query starts with Who or Whom: label it 'Human'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500"
      ]
     },
     "execution_count": 233,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set_file = \"questions_dataset/TREC_10.label\"\n",
    "test_quests_raw_tagged = parse_tagged_quests_dataset(test_set_file)\n",
    "len(test_quests_raw_tagged)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('How far is it from Denver to Aspen ?', 'NUM', 'dist'),\n",
       " ('What county is Modesto , California in ?', 'LOC', 'city')]"
      ]
     },
     "execution_count": 234,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_quests_raw_tagged[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need only coarse tags in this part. Therefore we will create test set with only coarse tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('How far is it from Denver to Aspen ?', 'NUM'),\n",
       " ('What county is Modesto , California in ?', 'LOC')]"
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_quests_raw_coarse_tagged = [(quest,coarse_tag) for quest, coarse_tag, fine_tag in test_quests_raw_tagged]\n",
    "test_quests_raw_coarse_tagged[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, we need an untagged test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['How far is it from Denver to Aspen ?',\n",
       " 'What county is Modesto , California in ?']"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.tag import untag\n",
    "test_quests_raw_untagged = untag(test_quests_raw_coarse_tagged)\n",
    "test_quests_raw_untagged[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the information of most common words for each tag, in order define rules for the baseline classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ABBR</th>\n",
       "      <th>DESC</th>\n",
       "      <th>ENTY</th>\n",
       "      <th>HUM</th>\n",
       "      <th>LOC</th>\n",
       "      <th>NUM</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(what, 86)</td>\n",
       "      <td>(what, 762)</td>\n",
       "      <td>(what, 1152)</td>\n",
       "      <td>(who, 597)</td>\n",
       "      <td>(what, 554)</td>\n",
       "      <td>(how, 492)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(for, 54)</td>\n",
       "      <td>(how, 287)</td>\n",
       "      <td>(of, 452)</td>\n",
       "      <td>(what, 547)</td>\n",
       "      <td>(where, 258)</td>\n",
       "      <td>(many, 323)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(does, 48)</td>\n",
       "      <td>(of, 247)</td>\n",
       "      <td>(in, 302)</td>\n",
       "      <td>(of, 386)</td>\n",
       "      <td>(in, 221)</td>\n",
       "      <td>(what, 276)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(stand, 41)</td>\n",
       "      <td>(do, 213)</td>\n",
       "      <td>('s, 206)</td>\n",
       "      <td>(in, 293)</td>\n",
       "      <td>(of, 201)</td>\n",
       "      <td>(of, 246)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(abbreviation, 16)</td>\n",
       "      <td>(to, 130)</td>\n",
       "      <td>(to, 162)</td>\n",
       "      <td>(was, 291)</td>\n",
       "      <td>(country, 123)</td>\n",
       "      <td>(in, 245)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(of, 15)</td>\n",
       "      <td>(in, 129)</td>\n",
       "      <td>(was, 138)</td>\n",
       "      <td>('s, 237)</td>\n",
       "      <td>('s, 102)</td>\n",
       "      <td>(when, 136)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(in, 10)</td>\n",
       "      <td>(does, 129)</td>\n",
       "      <td>(name, 118)</td>\n",
       "      <td>(to, 153)</td>\n",
       "      <td>(city, 101)</td>\n",
       "      <td>(are, 136)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(mean, 9)</td>\n",
       "      <td>(``, 112)</td>\n",
       "      <td>(are, 115)</td>\n",
       "      <td>(name, 144)</td>\n",
       "      <td>(can, 77)</td>\n",
       "      <td>(did, 121)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(``, 7)</td>\n",
       "      <td>('', 112)</td>\n",
       "      <td>(for, 109)</td>\n",
       "      <td>('', 142)</td>\n",
       "      <td>(i, 75)</td>\n",
       "      <td>(was, 115)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>('', 6)</td>\n",
       "      <td>(are, 106)</td>\n",
       "      <td>(and, 101)</td>\n",
       "      <td>(``, 140)</td>\n",
       "      <td>(to, 74)</td>\n",
       "      <td>(does, 95)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>(national, 5)</td>\n",
       "      <td>(why, 105)</td>\n",
       "      <td>('', 96)</td>\n",
       "      <td>(and, 128)</td>\n",
       "      <td>(was, 69)</td>\n",
       "      <td>(to, 91)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>(bureau, 5)</td>\n",
       "      <td>(you, 99)</td>\n",
       "      <td>(``, 95)</td>\n",
       "      <td>(for, 101)</td>\n",
       "      <td>(state, 65)</td>\n",
       "      <td>('s, 90)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>(investigation, 5)</td>\n",
       "      <td>(and, 95)</td>\n",
       "      <td>(did, 93)</td>\n",
       "      <td>(first, 82)</td>\n",
       "      <td>(on, 64)</td>\n",
       "      <td>(there, 79)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>(on, 4)</td>\n",
       "      <td>(i, 85)</td>\n",
       "      <td>(on, 71)</td>\n",
       "      <td>(on, 68)</td>\n",
       "      <td>(are, 63)</td>\n",
       "      <td>(for, 72)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>('s, 4)</td>\n",
       "      <td>('s, 81)</td>\n",
       "      <td>(fear, 66)</td>\n",
       "      <td>(president, 61)</td>\n",
       "      <td>(world, 62)</td>\n",
       "      <td>(on, 68)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  ABBR         DESC          ENTY              HUM  \\\n",
       "0   (what, 86)          (what, 762)  (what, 1152)  (who, 597)        \n",
       "1   (for, 54)           (how, 287)   (of, 452)     (what, 547)       \n",
       "2   (does, 48)          (of, 247)    (in, 302)     (of, 386)         \n",
       "3   (stand, 41)         (do, 213)    ('s, 206)     (in, 293)         \n",
       "4   (abbreviation, 16)  (to, 130)    (to, 162)     (was, 291)        \n",
       "5   (of, 15)            (in, 129)    (was, 138)    ('s, 237)         \n",
       "6   (in, 10)            (does, 129)  (name, 118)   (to, 153)         \n",
       "7   (mean, 9)           (``, 112)    (are, 115)    (name, 144)       \n",
       "8   (``, 7)             ('', 112)    (for, 109)    ('', 142)         \n",
       "9   ('', 6)             (are, 106)   (and, 101)    (``, 140)         \n",
       "10  (national, 5)       (why, 105)   ('', 96)      (and, 128)        \n",
       "11  (bureau, 5)         (you, 99)    (``, 95)      (for, 101)        \n",
       "12  (investigation, 5)  (and, 95)    (did, 93)     (first, 82)       \n",
       "13  (on, 4)             (i, 85)      (on, 71)      (on, 68)          \n",
       "14  ('s, 4)             ('s, 81)     (fear, 66)    (president, 61)   \n",
       "\n",
       "               LOC          NUM  \n",
       "0   (what, 554)     (how, 492)   \n",
       "1   (where, 258)    (many, 323)  \n",
       "2   (in, 221)       (what, 276)  \n",
       "3   (of, 201)       (of, 246)    \n",
       "4   (country, 123)  (in, 245)    \n",
       "5   ('s, 102)       (when, 136)  \n",
       "6   (city, 101)     (are, 136)   \n",
       "7   (can, 77)       (did, 121)   \n",
       "8   (i, 75)         (was, 115)   \n",
       "9   (to, 74)        (does, 95)   \n",
       "10  (was, 69)       (to, 91)     \n",
       "11  (state, 65)     ('s, 90)     \n",
       "12  (on, 64)        (there, 79)  \n",
       "13  (are, 63)       (for, 72)    \n",
       "14  (world, 62)     (on, 68)     "
      ]
     },
     "execution_count": 237,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import string\n",
    "stop_words = ['?', 'the', 'a', 'is'] + [punct for punct in string.punctuation]\n",
    "get_top_coarse_tokens(train_quests_raw_tagged, 15, stop_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bigrams can provide additional information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_bigram_tokens(raw_quest):\n",
    "    quest_tokens = raw_quest.lower().split(\" \") \n",
    "    return [\" \".join([quest_tokens[i], quest_tokens[i+1]]) for i in range(len(quest_tokens)-1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_top_coarse_tokens_bigrams(quests_tagged, top, stop_words = []):\n",
    "    tokens_per_coarse_tag = {}\n",
    "    for raw_quest, coarse_tag, fine_tag in quests_tagged:\n",
    "        quest_tokens = get_bigram_tokens(raw_quest)\n",
    "        if coarse_tag not in tokens_per_coarse_tag:\n",
    "            tokens_per_coarse_tag[coarse_tag] = Counter()\n",
    "        for token in quest_tokens:\n",
    "            if token not in stop_words:\n",
    "                tokens_per_coarse_tag[coarse_tag][token] += 1\n",
    "            \n",
    "    return pd.DataFrame({coarse_tag:counter.most_common(top) for coarse_tag, counter in tokens_per_coarse_tag.items()})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ABBR</th>\n",
       "      <th>DESC</th>\n",
       "      <th>ENTY</th>\n",
       "      <th>HUM</th>\n",
       "      <th>LOC</th>\n",
       "      <th>NUM</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(what does, 48)</td>\n",
       "      <td>(what is, 420)</td>\n",
       "      <td>(what is, 253)</td>\n",
       "      <td>(of the, 153)</td>\n",
       "      <td>(is the, 139)</td>\n",
       "      <td>(how many, 323)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(stand for, 41)</td>\n",
       "      <td>(is the, 191)</td>\n",
       "      <td>(is the, 167)</td>\n",
       "      <td>(who was, 144)</td>\n",
       "      <td>(what is, 95)</td>\n",
       "      <td>(is the, 136)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(for ?, 32)</td>\n",
       "      <td>(how do, 125)</td>\n",
       "      <td>(of the, 103)</td>\n",
       "      <td>(was the, 124)</td>\n",
       "      <td>(what country, 86)</td>\n",
       "      <td>(what is, 111)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(what is, 30)</td>\n",
       "      <td>(of the, 84)</td>\n",
       "      <td>(is a, 84)</td>\n",
       "      <td>('' ?, 107)</td>\n",
       "      <td>(where is, 77)</td>\n",
       "      <td>(are there, 71)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(is the, 16)</td>\n",
       "      <td>(what are, 77)</td>\n",
       "      <td>(what was, 78)</td>\n",
       "      <td>(who is, 107)</td>\n",
       "      <td>(in the, 74)</td>\n",
       "      <td>(was the, 66)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(the abbreviation, 14)</td>\n",
       "      <td>(is a, 73)</td>\n",
       "      <td>(what 's, 75)</td>\n",
       "      <td>(is the, 106)</td>\n",
       "      <td>(where can, 67)</td>\n",
       "      <td>(in the, 61)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(does the, 12)</td>\n",
       "      <td>(what does, 68)</td>\n",
       "      <td>(in the, 75)</td>\n",
       "      <td>(in the, 97)</td>\n",
       "      <td>(can i, 65)</td>\n",
       "      <td>(how much, 57)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(mean ?, 8)</td>\n",
       "      <td>(do you, 63)</td>\n",
       "      <td>(what are, 75)</td>\n",
       "      <td>(what is, 62)</td>\n",
       "      <td>(of the, 61)</td>\n",
       "      <td>(when was, 53)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(abbreviation for, 7)</td>\n",
       "      <td>('' ?, 59)</td>\n",
       "      <td>('s the, 64)</td>\n",
       "      <td>(the first, 61)</td>\n",
       "      <td>(the world, 57)</td>\n",
       "      <td>(how long, 50)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>(abbreviation of, 5)</td>\n",
       "      <td>(the origin, 53)</td>\n",
       "      <td>(was the, 63)</td>\n",
       "      <td>(name of, 55)</td>\n",
       "      <td>(what city, 49)</td>\n",
       "      <td>(when did, 49)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>(of the, 5)</td>\n",
       "      <td>(origin of, 53)</td>\n",
       "      <td>(fear of, 62)</td>\n",
       "      <td>(what was, 49)</td>\n",
       "      <td>(i find, 44)</td>\n",
       "      <td>(of the, 45)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>(the national, 5)</td>\n",
       "      <td>(mean ?, 51)</td>\n",
       "      <td>(name of, 62)</td>\n",
       "      <td>(the name, 45)</td>\n",
       "      <td>(in ?, 38)</td>\n",
       "      <td>(what year, 43)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>(national bureau, 5)</td>\n",
       "      <td>(how can, 48)</td>\n",
       "      <td>(the name, 60)</td>\n",
       "      <td>(: ``, 33)</td>\n",
       "      <td>(are the, 37)</td>\n",
       "      <td>(in a, 41)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>(bureau of, 5)</td>\n",
       "      <td>(do i, 42)</td>\n",
       "      <td>(a fear, 59)</td>\n",
       "      <td>(name the, 30)</td>\n",
       "      <td>(the largest, 35)</td>\n",
       "      <td>(there in, 37)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>(of investigation, 5)</td>\n",
       "      <td>(how does, 39)</td>\n",
       "      <td>('' ?, 55)</td>\n",
       "      <td>(who invented, 29)</td>\n",
       "      <td>(where did, 34)</td>\n",
       "      <td>(many people, 35)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      ABBR              DESC            ENTY  \\\n",
       "0   (what does, 48)         (what is, 420)    (what is, 253)   \n",
       "1   (stand for, 41)         (is the, 191)     (is the, 167)    \n",
       "2   (for ?, 32)             (how do, 125)     (of the, 103)    \n",
       "3   (what is, 30)           (of the, 84)      (is a, 84)       \n",
       "4   (is the, 16)            (what are, 77)    (what was, 78)   \n",
       "5   (the abbreviation, 14)  (is a, 73)        (what 's, 75)    \n",
       "6   (does the, 12)          (what does, 68)   (in the, 75)     \n",
       "7   (mean ?, 8)             (do you, 63)      (what are, 75)   \n",
       "8   (abbreviation for, 7)   ('' ?, 59)        ('s the, 64)     \n",
       "9   (abbreviation of, 5)    (the origin, 53)  (was the, 63)    \n",
       "10  (of the, 5)             (origin of, 53)   (fear of, 62)    \n",
       "11  (the national, 5)       (mean ?, 51)      (name of, 62)    \n",
       "12  (national bureau, 5)    (how can, 48)     (the name, 60)   \n",
       "13  (bureau of, 5)          (do i, 42)        (a fear, 59)     \n",
       "14  (of investigation, 5)   (how does, 39)    ('' ?, 55)       \n",
       "\n",
       "                   HUM                 LOC                NUM  \n",
       "0   (of the, 153)       (is the, 139)       (how many, 323)    \n",
       "1   (who was, 144)      (what is, 95)       (is the, 136)      \n",
       "2   (was the, 124)      (what country, 86)  (what is, 111)     \n",
       "3   ('' ?, 107)         (where is, 77)      (are there, 71)    \n",
       "4   (who is, 107)       (in the, 74)        (was the, 66)      \n",
       "5   (is the, 106)       (where can, 67)     (in the, 61)       \n",
       "6   (in the, 97)        (can i, 65)         (how much, 57)     \n",
       "7   (what is, 62)       (of the, 61)        (when was, 53)     \n",
       "8   (the first, 61)     (the world, 57)     (how long, 50)     \n",
       "9   (name of, 55)       (what city, 49)     (when did, 49)     \n",
       "10  (what was, 49)      (i find, 44)        (of the, 45)       \n",
       "11  (the name, 45)      (in ?, 38)          (what year, 43)    \n",
       "12  (: ``, 33)          (are the, 37)       (in a, 41)         \n",
       "13  (name the, 30)      (the largest, 35)   (there in, 37)     \n",
       "14  (who invented, 29)  (where did, 34)     (many people, 35)  "
      ]
     },
     "execution_count": 240,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_top_coarse_tokens_bigrams(train_quests_raw_tagged, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be seen that for example the bigrams 'how many', 'how much' and 'how long' are common in 'NUM'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The proportion of each coarse tag can be also useful"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fine class</th>\n",
       "      <th>percent</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>coarse class</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ABBR</th>\n",
       "      <td>86</td>\n",
       "      <td>1.577403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DESC</th>\n",
       "      <td>1162</td>\n",
       "      <td>21.313280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENTY</th>\n",
       "      <td>1250</td>\n",
       "      <td>22.927366</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HUM</th>\n",
       "      <td>1223</td>\n",
       "      <td>22.432135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>835</td>\n",
       "      <td>15.315481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NUM</th>\n",
       "      <td>896</td>\n",
       "      <td>16.434336</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              fine class    percent\n",
       "coarse class                       \n",
       "ABBR          86          1.577403 \n",
       "DESC          1162        21.313280\n",
       "ENTY          1250        22.927366\n",
       "HUM           1223        22.432135\n",
       "LOC           835         15.315481\n",
       "NUM           896         16.434336"
      ]
     },
     "execution_count": 241,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped_by_coarse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class BaselineCalssifier():\n",
    "    TAGS = ['ABBR', 'NUM', 'HUM', 'LOC', 'DESC', 'ENTY']\n",
    "    \n",
    "    TAG_WORDS_UNIGRAM = {\n",
    "                 'HUM': ['who', 'whom'],\n",
    "                 'LOC': ['where', 'country', 'city', 'world'],\n",
    "                 'NUM': ['when', 'date'],\n",
    "                 'DESC': ['how'],\n",
    "                 'ABBR': ['abbreviation'],\n",
    "                 'ENTY': ['name']\n",
    "                }\n",
    "    \n",
    "    TAG_WORDS_BIGRAM = {\n",
    "                 'HUM': [],\n",
    "                 'LOC': [],\n",
    "                 'NUM': ['how many', 'how much', 'how long', 'how far', 'what year'],\n",
    "                 'DESC': ['what is', 'what are', 'what does', 'how do'],\n",
    "                 'ABBR': ['stand for'],\n",
    "                 'ENTY': []\n",
    "                }\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.default_tag = 'ENTY' #this is a common tag, but does not have specific words associations\n",
    "        \n",
    "    def _quest_contains_any_word(self, quest, words):\n",
    "        return any(word in quest for word in words)\n",
    "    \n",
    "    def _get_bigram_tokens(self, quest_tokens):\n",
    "        return [\" \".join([quest_tokens[i], quest_tokens[i+1]]) for i in range(len(quest_tokens)-1)]\n",
    "        \n",
    "    def tag_quest(self, quest):\n",
    "        quest_tokens = quest.lower().split(\" \")\n",
    "        quest_tokens_set = set(quest_tokens)\n",
    "        for tag in self.TAGS:\n",
    "            if self._quest_contains_any_word(quest_tokens_set, self.TAG_WORDS_UNIGRAM[tag]) or self._quest_contains_any_word(self._get_bigram_tokens(quest_tokens), self.TAG_WORDS_BIGRAM[tag]):\n",
    "                return (quest, tag)\n",
    "        return (quest, self.default_tag)\n",
    "    \n",
    "    def batch_tag(self, quests):\n",
    "        tagged_quests = []\n",
    "        for quest in quests:\n",
    "            tagged_quests.append(self.tag_quest(quest))\n",
    "        return tagged_quests\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('How long did Rip Van Winkle sleep ?', 'NUM')"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline_classifier = BaselineCalssifier()\n",
    "test_quest = test_quests_raw_untagged[46]\n",
    "baseline_classifier.tag_quest(test_quest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('How far is it from Denver to Aspen ?', 'NUM'),\n",
       " ('What county is Modesto , California in ?', 'ENTY')]"
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline_classifier.batch_tag(test_quests_raw_untagged[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "We will start by evaluating the accuracy of the baseline classifier. \n",
    "The accuracy is the number of samples predicted correctly, divided by the total number of samples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def accuracy(classifier, tagged_dataset):\n",
    "    \"\"\"Recieves a tagged dataset, and calculates the accuracy of classifier on the same dataset\"\"\"\n",
    "    pred = classifier.batch_tag(untag(tagged_dataset))\n",
    "    correct = [l == r for (q,l), (q,r) in zip(tagged_dataset, pred)]\n",
    "\n",
    "    if correct:\n",
    "        return sum(correct) / len(correct) * 100\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "66.8"
      ]
     },
     "execution_count": 246,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(baseline_classifier, test_quests_raw_coarse_tagged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does it compare to a default classifier, that taggs all question with the same tag?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Default_Classifier():\n",
    "    def __init__(self, default_tag):\n",
    "        self.default_tag = default_tag\n",
    "    \n",
    "    def tag_quest(self, quest):\n",
    "        return (quest, self.default_tag)\n",
    "    \n",
    "    def batch_tag(self, quests):\n",
    "        tagged_quests = []\n",
    "        for quest in quests:\n",
    "            tagged_quests.append(self.tag_quest(quest))\n",
    "        return tagged_quests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tag: ABBR, Accuracy 1.7999999999999998\n",
      "Tag: NUM, Accuracy 22.6\n",
      "Tag: HUM, Accuracy 13.0\n",
      "Tag: LOC, Accuracy 16.2\n",
      "Tag: DESC, Accuracy 27.6\n",
      "Tag: ENTY, Accuracy 18.8\n"
     ]
    }
   ],
   "source": [
    "for tag in baseline_classifier.TAGS:\n",
    "    def_classifier = Default_Classifier(tag)\n",
    "    print(\"Tag: \" + tag + \", Accuracy \" + str(accuracy(def_classifier, test_quests_raw_coarse_tagged)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The highest accuracy of 27.6% can be achieved by tagging all questions as 'DESC'. It seems like the baseline heuristic classifier does a much better job than the default classifier, that is encouraging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for more specific evaluation metrics: **precision**, **recall** and **f1-score** per coarse label.\n",
    "_sklearn_ has a very convenient method for this calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "       ABBR      1.000     0.778     0.875         9\n",
      "        NUM      0.985     0.575     0.726       113\n",
      "        HUM      1.000     0.723     0.839        65\n",
      "        LOC      0.932     0.506     0.656        81\n",
      "       DESC      0.532     0.957     0.684       138\n",
      "       ENTY      0.477     0.447     0.462        94\n",
      "\n",
      "avg / total      0.758     0.668     0.671       500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "pred = baseline_classifier.batch_tag(test_quests_raw_untagged)\n",
    "y_pred = [y for x, y in pred]\n",
    "y_test = [y for x, y in test_quests_raw_coarse_tagged]\n",
    "\n",
    "print(metrics.classification_report(y_test, y_pred, digits=3, labels=baseline_classifier.TAGS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Precision - what proportion of questions that were predicted as x are actually x\n",
    "For 'HUM' it is 100% - meaning that when a question was tagged as 'HUM' it is indead 'HUM'.\n",
    "For 'ENTY' it is quite low (47%). It means that more that 50% of the times that 'ENTY' tag was given to a question, it was not correct. This makes sense as this is the default tag.\n",
    "\n",
    "#### Recall - what proportion of questions that has a label x, were predicted as x\n",
    "Even though the precision of 'DESC' is not high (53%) - its recall is pretty high (95%). It means that most of the questions with 'DESC' tag, were tagged correctly.\n",
    "\n",
    "There is a tradeoff between precision and recall.\n",
    "\n",
    "#### F1-score - harmonic mean of percision and recall  \n",
    "f1 score is high for ABBR, NUM and HUM, and low for LOC, DESC and ENTY."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Actual</th>\n",
       "      <th>ABBR</th>\n",
       "      <th>NUM</th>\n",
       "      <th>HUM</th>\n",
       "      <th>LOC</th>\n",
       "      <th>DESC</th>\n",
       "      <th>ENTY</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ABBR</th>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NUM</th>\n",
       "      <td>0</td>\n",
       "      <td>65</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HUM</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>47</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DESC</th>\n",
       "      <td>2</td>\n",
       "      <td>42</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "      <td>132</td>\n",
       "      <td>51</td>\n",
       "      <td>248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENTY</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>24</td>\n",
       "      <td>6</td>\n",
       "      <td>42</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>9</td>\n",
       "      <td>113</td>\n",
       "      <td>65</td>\n",
       "      <td>81</td>\n",
       "      <td>138</td>\n",
       "      <td>94</td>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Actual     ABBR  NUM  HUM  LOC  DESC  ENTY  All\n",
       "Predicted                                      \n",
       "ABBR       7     0    0    0    0     0     7  \n",
       "NUM        0     65   0    0    0     1     66 \n",
       "HUM        0     0    47   0    0     0     47 \n",
       "LOC        0     3    0    41   0     0     44 \n",
       "DESC       2     42   5    16   132   51    248\n",
       "ENTY       0     3    13   24   6     42    88 \n",
       "All        9     113  65   81   138   94    500"
      ]
     },
     "execution_count": 250,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_true_s = pd.Categorical(y_test, categories=baseline_classifier.TAGS)\n",
    "y_pred_s = pd.Categorical(y_pred, categories=baseline_classifier.TAGS)\n",
    "pd.crosstab(y_pred_s, y_true_s, rownames=['Predicted'], colnames=['Actual'], margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The confusion matrix shows what kind of mistakes this classifier tends to make.\n",
    "We can see that NUM questions are often tagged by DESC label. ENTY and NUM are often tagged as DESC."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q1.4 Features-based classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part we will implement a feature based classifier, using the types of features described in the paper Section 3.2: words, POS tags, NER tags, chunks and related words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4.1 Feature extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Encoding of features:**\n",
    "\n",
    "**Word features**: Word features can be encoded in different ways: noise words (stop words) filtered or not, with or without lemmatization, with or without case normalization (all lower-case). When analalyzing the data, we have seen that the text is sparse. Therefore we want the dictionary to be as small as possible, and we want the bag of words to be most informative. Hence, we will use tokenization as lower case, lemmatization and finally tf-idf (as there are tokens that occur in all questions, making them not informative). We will also take only 1000 words with heighest tfidf values.\n",
    "\n",
    "**POS features:** POS features can be encoded in different ways: as a bag of POS-tags, or associated with the word in a bag-of-tagged words such as 'Apple/PROPN'. Again, as the data is sparse, we would like to have less dependence between features. Therefore we decided to decode POS tags as a bag of POS-tags.\n",
    "\n",
    "**Chunks:** Noun chunks are group of words which correspond to a single nominal phrase. For each chunk, we wil extract the central noun in the chunk. Chunks can be encoded as a bag of chunk-roots.\n",
    "\n",
    "**NER:** Named Entity Recognition tags can be encoded as a bag of NERs.\n",
    "\n",
    "**Related words:** Related words can be \"learned\" from the training dataset by detecting words which have a high chi-square value with each class. We will extract bag of words as described in Word features, than perform a chi2 test to select 100 words with height chi2 score. This will simulate words that are very related to specific classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4.2 scikit-learn based classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Spacy](https://spacy.io/usage/spacy-101#annotations-ner) is a library that can be used for pre-processing of the questions - including POS tagging, Named Entity Recognition and Noun Chunks detection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Apple, U.K., $1 billion)"
      ]
     },
     "execution_count": 252,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp('Apple is looking at buying U.K. startup for $1 billion')\n",
    "doc.ents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORG\n",
      "GPE\n",
      "MONEY\n"
     ]
    }
   ],
   "source": [
    "for ent in doc.ents:\n",
    "    print(ent.label_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple apple PROPN NNP nsubj Xxxxx True False\n",
      "    \n",
      "is be VERB VBZ aux xx True True\n",
      "    \n",
      "looking look VERB VBG ROOT xxxx True False\n",
      "    \n",
      "at at ADP IN prep xx True True\n",
      "    \n",
      "buying buy VERB VBG pcomp xxxx True False\n",
      "    \n",
      "U.K. u.k. PROPN NNP compound X.X. False False\n",
      "    \n",
      "startup startup NOUN NN dobj xxxx True False\n",
      "    \n",
      "for for ADP IN prep xxx True True\n",
      "    \n",
      "$ $ SYM $ quantmod $ False False\n",
      "    \n",
      "1 1 NUM CD compound d False False\n",
      "    \n",
      "billion billion NUM CD pobj xxxx True False\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "for token in doc:\n",
    "    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n",
    "          token.shape_, token.is_alpha, token.is_stop)\n",
    "    print(\"    \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'model'"
      ]
     },
     "execution_count": 255,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp('Apple announced a new model yesterday.')\n",
    "chunks = list(doc.noun_chunks)\n",
    "chunks[1].root.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will process the train and test data using spacy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_quests_processed_tagged = [(nlp(quest), coarse, fine) for quest, coarse, fine in train_quests_raw_tagged]\n",
    "test_quests_processed_tagged = [(nlp(quest), coarse, fine) for quest, coarse, fine  in test_quests_raw_tagged]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of processed train questions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(How did serfdom develop in and then leave Russia ?, 'DESC', 'manner')\n",
      "How lemma: how POS: ADV\n",
      "did lemma: do POS: VERB\n",
      "serfdom lemma: serfdom POS: NOUN\n",
      "develop lemma: develop POS: VERB\n",
      "in lemma: in POS: ADP\n",
      "and lemma: and POS: CCONJ\n",
      "then lemma: then POS: ADV\n",
      "leave lemma: leave POS: VERB\n",
      "Russia lemma: russia POS: PROPN\n",
      "? lemma: ? POS: PUNCT\n",
      "NER:  Russia GPE\n",
      "Chunk root:  serfdom\n",
      "Chunk root:  Russia\n"
     ]
    }
   ],
   "source": [
    "train_quest = train_quests_processed_tagged[0]\n",
    "print(train_quest)\n",
    "for token in train_quest[0]:\n",
    "    print(token.text, \"lemma:\", token.lemma_, \"POS:\", token.pos_)\n",
    "for ent in train_quest[0].ents:\n",
    "    print(\"NER: \", ent.text, ent.label_)\n",
    "for chunk in train_quest[0].noun_chunks:\n",
    "    print(\"Chunk root: \", chunk.root.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of processed test questions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(How far is it from Denver to Aspen ?, 'NUM', 'dist')\n",
      "How lemma: how POS: ADV\n",
      "far lemma: far POS: ADV\n",
      "is lemma: be POS: VERB\n",
      "it lemma: -PRON- POS: PRON\n",
      "from lemma: from POS: ADP\n",
      "Denver lemma: denver POS: PROPN\n",
      "to lemma: to POS: ADP\n",
      "Aspen lemma: aspen POS: PROPN\n",
      "? lemma: ? POS: PUNCT\n",
      "NER:  Denver GPE\n",
      "Chunk root:  it\n",
      "Chunk root:  Denver\n",
      "Chunk root:  Aspen\n"
     ]
    }
   ],
   "source": [
    "test_quest = test_quests_processed_tagged[0]\n",
    "print(test_quest)\n",
    "for token in test_quest[0]:\n",
    "    print(token.text, \"lemma:\", token.lemma_, \"POS:\", token.pos_)\n",
    "for ent in test_quest[0].ents:\n",
    "    print(\"NER: \", ent.text, ent.label_)\n",
    "for chunk in test_quest[0].noun_chunks:\n",
    "    print(\"Chunk root: \", chunk.root.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7483"
      ]
     },
     "execution_count": 259,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_counter = Counter(token.lemma_ for quest in train_quests_processed_tagged for token in quest[0])\n",
    "len(vocab_counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The vocabulary with lemmas rather than tokens is now smaller than with just lowercase (8678)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Level: coarse\n",
      "________________________________________________________________________________\n",
      "SGD\n",
      "Accuracy 0.874\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "       ABBR       0.78      1.00      0.88         7\n",
      "       DESC       0.92      0.85      0.89       149\n",
      "       ENTY       0.81      0.78      0.79        98\n",
      "        HUM       0.94      0.86      0.90        71\n",
      "        LOC       0.81      0.92      0.86        72\n",
      "        NUM       0.88      0.97      0.93       103\n",
      "\n",
      "avg / total       0.88      0.87      0.87       500\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Actual</th>\n",
       "      <th>ABBR</th>\n",
       "      <th>DESC</th>\n",
       "      <th>ENTY</th>\n",
       "      <th>HUM</th>\n",
       "      <th>LOC</th>\n",
       "      <th>NUM</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ABBR</th>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DESC</th>\n",
       "      <td>2</td>\n",
       "      <td>127</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENTY</th>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>76</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HUM</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>61</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>66</td>\n",
       "      <td>3</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NUM</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>9</td>\n",
       "      <td>138</td>\n",
       "      <td>94</td>\n",
       "      <td>65</td>\n",
       "      <td>81</td>\n",
       "      <td>113</td>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Actual     ABBR  DESC  ENTY  HUM  LOC  NUM  All\n",
       "Predicted                                      \n",
       "ABBR       7     0     0     0    0    0    7  \n",
       "DESC       2     127   8     2    4    6    149\n",
       "ENTY       0     9     76    2    8    3    98 \n",
       "HUM        0     0     7     61   2    1    71 \n",
       "LOC        0     1     2     0    66   3    72 \n",
       "NUM        0     1     1     0    1    100  103\n",
       "All        9     138   94    65   81   113  500"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Level: flat\n",
      "________________________________________________________________________________\n",
      "SGD\n",
      "Accuracy 0.816\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Dina\\Programs\\Anaconda\\lib\\site-packages\\sklearn\\metrics\\classification.py:1113: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Dina\\Programs\\Anaconda\\lib\\site-packages\\sklearn\\metrics\\classification.py:1115: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples.\n",
      "  'recall', 'true', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        abb       1.00      1.00      1.00         1\n",
      "     animal       0.62      0.77      0.69        13\n",
      "       body       0.00      0.00      0.00         0\n",
      "       city       0.67      0.86      0.75        14\n",
      "      color       1.00      1.00      1.00        10\n",
      "      count       1.00      0.64      0.78        14\n",
      "    country       1.00      1.00      1.00         3\n",
      "     cremat       0.00      0.00      0.00         2\n",
      "   currency       1.00      0.86      0.92         7\n",
      "       date       1.00      0.96      0.98        49\n",
      "        def       0.97      0.89      0.93       134\n",
      "       desc       0.60      0.55      0.57        11\n",
      "     dismed       0.50      0.33      0.40         3\n",
      "       dist       0.44      1.00      0.61         7\n",
      "      event       0.50      0.20      0.29         5\n",
      "        exp       0.75      0.86      0.80         7\n",
      "       food       0.75      0.60      0.67         5\n",
      "         gr       0.50      0.43      0.46         7\n",
      "        ind       0.95      0.90      0.92        58\n",
      "     instru       1.00      1.00      1.00         1\n",
      "       lang       1.00      1.00      1.00         2\n",
      "     letter       0.00      0.00      0.00         1\n",
      "     manner       1.00      0.40      0.57         5\n",
      "      money       0.33      1.00      0.50         1\n",
      "      mount       0.67      0.67      0.67         3\n",
      "      other       0.68      0.71      0.69        70\n",
      "       perc       0.67      1.00      0.80         2\n",
      "     period       0.75      0.55      0.63        11\n",
      "      plant       1.00      1.00      1.00         5\n",
      "    product       0.00      0.00      0.00         1\n",
      "     reason       1.00      0.86      0.92         7\n",
      "      speed       0.83      1.00      0.91         5\n",
      "      sport       1.00      0.33      0.50         3\n",
      "      state       0.57      0.80      0.67         5\n",
      "  substance       0.47      0.78      0.58         9\n",
      "   techmeth       1.00      1.00      1.00         1\n",
      "       temp       0.80      1.00      0.89         4\n",
      "     termeq       1.00      0.88      0.93         8\n",
      "      title       0.00      0.00      0.00         0\n",
      "        veh       0.50      1.00      0.67         2\n",
      "     weight       1.00      1.00      1.00         4\n",
      "\n",
      "avg / total       0.85      0.82      0.82       500\n",
      "\n",
      "Classification Level: hierarchical\n",
      "________________________________________________________________________________\n",
      "Ridge Classifier\n",
      "Accuracy 0.826\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Dina\\Programs\\Anaconda\\lib\\site-packages\\sklearn\\metrics\\classification.py:1113: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Dina\\Programs\\Anaconda\\lib\\site-packages\\sklearn\\metrics\\classification.py:1115: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples.\n",
      "  'recall', 'true', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        abb       0.00      0.00      0.00         0\n",
      "     animal       0.56      0.75      0.64        12\n",
      "       body       0.50      0.50      0.50         2\n",
      "       city       0.67      0.86      0.75        14\n",
      "      color       1.00      1.00      1.00        10\n",
      "      count       1.00      0.82      0.90        11\n",
      "    country       1.00      1.00      1.00         3\n",
      "     cremat       0.00      0.00      0.00         2\n",
      "   currency       1.00      1.00      1.00         6\n",
      "       date       0.98      0.94      0.96        49\n",
      "        def       0.95      0.89      0.92       132\n",
      "       desc       0.60      0.60      0.60        10\n",
      "     dismed       0.50      0.50      0.50         2\n",
      "       dist       0.44      1.00      0.61         7\n",
      "      event       0.00      0.00      0.00         2\n",
      "        exp       0.75      0.86      0.80         7\n",
      "       food       0.75      1.00      0.86         3\n",
      "         gr       0.83      0.71      0.77         7\n",
      "        ind       0.96      0.87      0.91        61\n",
      "     instru       1.00      1.00      1.00         1\n",
      "       lang       1.00      1.00      1.00         2\n",
      "     manner       1.00      0.40      0.57         5\n",
      "      money       0.33      1.00      0.50         1\n",
      "      mount       0.67      1.00      0.80         2\n",
      "      other       0.76      0.67      0.71        83\n",
      "       perc       1.00      0.75      0.86         4\n",
      "     period       0.88      0.58      0.70        12\n",
      "      plant       1.00      1.00      1.00         5\n",
      "    product       0.25      0.50      0.33         2\n",
      "     reason       0.83      1.00      0.91         5\n",
      "      speed       1.00      1.00      1.00         6\n",
      "      sport       1.00      1.00      1.00         1\n",
      "      state       0.43      0.75      0.55         4\n",
      "  substance       0.33      1.00      0.50         5\n",
      "   techmeth       1.00      1.00      1.00         1\n",
      "       temp       0.80      1.00      0.89         4\n",
      "     termeq       1.00      0.70      0.82        10\n",
      "      title       0.00      0.00      0.00         0\n",
      "        veh       0.75      1.00      0.86         3\n",
      "     weight       1.00      1.00      1.00         4\n",
      "\n",
      "avg / total       0.86      0.83      0.83       500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.feature_selection import SelectKBest, chi2\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import PassiveAggressiveClassifier\n",
    "from sklearn.linear_model import RidgeClassifier\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.linear_model import Perceptron\n",
    "from sklearn.linear_model import PassiveAggressiveClassifier\n",
    "from sklearn.naive_bayes import BernoulliNB, MultinomialNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.neighbors import NearestCentroid\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.utils.extmath import density\n",
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "from sklearn import metrics\n",
    "from IPython.display import display\n",
    "\n",
    "    \n",
    "class ItemSelector(BaseEstimator, TransformerMixin):\n",
    "    \"\"\"For data grouped by feature, select subset of data at a provided key.\n",
    "\n",
    "    The data is expected to be stored in a 2D data structure, where the first\n",
    "    index is over features and the second is over samples.\n",
    "    Parameters\n",
    "    ----------\n",
    "    key : hashable, required\n",
    "        The key corresponding to the desired value in a mappable.\n",
    "    \"\"\"\n",
    "    def __init__(self, key):\n",
    "        self.key = key\n",
    "\n",
    "    def fit(self, x, y=None):\n",
    "        return self\n",
    "\n",
    "    def transform(self, data_dict):\n",
    "        return data_dict[self.key]\n",
    "    \n",
    "class FeatureExtractor(BaseEstimator, TransformerMixin):\n",
    "    \"\"\"Extract the tags from questions.\n",
    "\n",
    "    Takes a sequence of strings and produces a dict of sequences.  Keys are\n",
    "    `bow`, `pos`, 'chunk', 'ent'.\n",
    "    \"\"\"\n",
    "    def fit(self, x, y=None):\n",
    "        return self\n",
    "    \n",
    "    def _extract_quest_ents(self, quest):\n",
    "        return \" \".join([ent.label_ for ent in doc.ents])\n",
    "    \n",
    "    def _extract_quest_chunk_roots(self, quest):\n",
    "        chunks = list(quest.noun_chunks)\n",
    "        return \" \".join([chunk.root.text for chunk in chunks])\n",
    "    \n",
    "    def _extract_quest_lemmas(self, quest):\n",
    "        return \" \".join([token.lemma_ for token in quest])\n",
    "    \n",
    "    def _extract_quest_pos(self, quest):\n",
    "        return \" \".join([token.pos_ for token in quest])\n",
    "\n",
    "    def transform(self, quests_processed):\n",
    "        features = np.recarray(shape=(len(quests_processed),),\n",
    "                               dtype=[('bow', object), ('pos', object), ('chunk', object), ('ent', object)])\n",
    "        for i, quest in enumerate(quests_processed):\n",
    "            features['bow'][i] = self._extract_quest_lemmas(quest)\n",
    "           \n",
    "            features['pos'][i] = self._extract_quest_pos(quest)\n",
    "            \n",
    "            features['chunk'][i] = self._extract_quest_chunk_roots(quest)\n",
    "            \n",
    "            features['ent'][i] = self._extract_quest_ents(quest)\n",
    "\n",
    "        return features\n",
    "\n",
    "class CLFResult():\n",
    "    \"\"\"represents a classifier result\"\"\"\n",
    "    def __init__(self, clf, clf_level, y_pred, y_test):\n",
    "        self.clf = clf\n",
    "        self.clf_level = clf_level\n",
    "        self.y_pred = y_pred\n",
    "        self.y_test = y_test\n",
    "        self.acc = metrics.accuracy_score(y_test, y_pred)\n",
    "    \n",
    "    def get_clf(self):\n",
    "        return self.clf\n",
    "    \n",
    "    def get_clf_level(self):\n",
    "        return self.clf_level\n",
    "    \n",
    "    def get_y_pred(self):\n",
    "        return self.y_pred\n",
    "    \n",
    "    def get_y_test(self):\n",
    "        return self.y_test\n",
    "    \n",
    "    def get_acc(self):\n",
    "        return self.acc\n",
    "\n",
    "def get_tfidf_pipeline(name, max_feat):\n",
    "    \"\"\"select a feature by name, then apply tfidf vectorizer\"\"\"\n",
    "    return  (name, Pipeline([\n",
    "                ('selector', ItemSelector(key=name)),\n",
    "                ('tfidf', TfidfVectorizer(max_features=max_feat)),\n",
    "            ]))\n",
    "\n",
    "def get_bow_chi2_pipeline(name, select_chi2):\n",
    "    \"\"\"related words: extract bag of words, apply count vectorizer, then select k best words using chi2 test\"\"\"\n",
    "    return (name, Pipeline([\n",
    "                ('selector', ItemSelector(key='bow')),\n",
    "                ('count', CountVectorizer()),\n",
    "                ('chi2', SelectKBest(chi2, select_chi2)),\n",
    "            ]))\n",
    "    \n",
    "def get_pipeline(clf):\n",
    "    \"\"\"a pipeline that extracts features, unifies them and classifies using clf\"\"\"\n",
    "    features_names = ['bow', 'pos', 'chunk', 'ent']\n",
    "    features_pipelines = [get_tfidf_pipeline(name, 1000) for name in features_names]\n",
    "    features_pipelines.append(get_bow_chi2_pipeline('bow-chi2',100))\n",
    "    \n",
    "    return Pipeline([\n",
    "    # Extract features\n",
    "    ('features', FeatureExtractor()),\n",
    "\n",
    "    # Use FeatureUnion to combine the features\n",
    "    ('union', FeatureUnion(\n",
    "        transformer_list=features_pipelines,\n",
    "    )),\n",
    "\n",
    "    # Use a classifier on the combined features\n",
    "    clf,\n",
    "])\n",
    "\n",
    "def get_y(label, quests_tagged):\n",
    "    \"\"\"get a list of the corresponding labels for quests_tagged\"\"\"\n",
    "    if label == 'coarse':\n",
    "        return [quest[1] for quest in quests_tagged]\n",
    "    elif label == 'flat':\n",
    "        return [quest[2] for quest in quests_tagged]\n",
    "\n",
    "def untag(quests):\n",
    "    return [quest[0] for quest in quests]\n",
    "\n",
    "def classify(clf_level, clf, train, test):\n",
    "    \"\"\"fit according to train and predict test, use clf_level labeling\"\"\"\n",
    "    x_train = untag(train)\n",
    "    y_train = get_y(clf_level, train)\n",
    "\n",
    "    x_test = untag(test)\n",
    "    y_test = get_y(clf_level, test)\n",
    "    \n",
    "    pipeline = get_pipeline(clf)\n",
    "    pipeline.fit(x_train, y_train)\n",
    "    y = pipeline.predict(x_test)\n",
    "    \n",
    "    return y, y_test\n",
    "\n",
    "def partition_by_coarse(dataset, train_label_dist_dict):\n",
    "    \"\"\"partition dataset according to the keys in train_label_dist_dict\"\"\"\n",
    "    partitions = {key:[] for key in train_label_dist_dict.keys()}\n",
    "    for quest in dataset:\n",
    "        coarse_label = quest[1]\n",
    "        partitions[coarse_label].append(quest)\n",
    "    return partitions\n",
    "\n",
    "def print_metrics(clf, clf_level, y_pred, y_test):\n",
    "    print(\"Classification Level: \" + clf_level)\n",
    "    print('_' * 80)\n",
    "    print(clf[0])    \n",
    "    print(\"Accuracy \" + str(metrics.accuracy_score(y_test, y_pred)))\n",
    "    print(metrics.classification_report(y_pred, y_test))\n",
    "    if clf_level == 'coarse':\n",
    "        y_test_s = pd.Categorical(y_test)\n",
    "        y_pred_s = pd.Categorical(y_pred)\n",
    "        display(pd.crosstab(y_pred_s, y_test_s, rownames=['Predicted'], colnames=['Actual'], margins=True))\n",
    "\n",
    "def choose_classifier(clf_level, clfs, train, test):\n",
    "    \"\"\"choose classifier from clfs, with the heighst accuracy\"\"\"\n",
    "    clfs_results = []\n",
    "    for clf in clfs:\n",
    "        y_pred, y_test = classify(clf_level, clf, train, test)\n",
    "        \n",
    "        clf_result = CLFResult(clf, clf_level, y_pred, y_test)\n",
    "        clfs_results.append(clf_result)\n",
    "        \n",
    "    max_clf_result = max(clfs_results, key=lambda clf_res:clf_res.get_acc())\n",
    "    return max_clf_result\n",
    "\n",
    "def hierarchical_clf(coarse_clf, clfs, clf_level): \n",
    "    \"\"\"classify in two steps: first by coarse labels, then each coarse group by flat labels\"\"\"\n",
    "    # first classify by coarse labels\n",
    "    y_pred_coarse, _ = classify('coarse', coarse_clf, train_quests_processed_tagged, test_quests_processed_tagged)\n",
    "    \n",
    "    # annotate test set using prediction of coarse labels \n",
    "    test_with_coarse_pred = zip(test_quests_processed_tagged, y_pred_coarse)\n",
    "    \n",
    "    # partition test and train by the number of coarse labels\n",
    "    test_partitions = partition_by_coarse(test_with_coarse_pred, train_label_dist_dict)\n",
    "    train_partitions = partition_by_coarse(train_quests_processed_tagged, train_label_dist_dict)\n",
    "    \n",
    "    clfs_results = []\n",
    "    \n",
    "    for clf in clfs:\n",
    "        y_pred_flat_total = []\n",
    "        y_test_flat_total = []\n",
    "        for coarse in train_label_dist_dict.keys():\n",
    "            train = train_partitions[coarse]\n",
    "            test = untag(test_partitions[coarse])   \n",
    "            y_pred_flat, y_test_flat = classify('flat', clf, train, test)\n",
    "\n",
    "            y_pred_flat_total += list(y_pred_flat)\n",
    "            y_test_flat_total += y_test_flat\n",
    "        \n",
    "        clf_result = CLFResult(clf, clf_level, y_pred_flat_total, y_test_flat_total)\n",
    "        clfs_results.append(clf_result)\n",
    "                        \n",
    "    max_clf_result = max(clfs_results, key=lambda clf_res:clf_res.get_acc())\n",
    "    \n",
    "    return max_clf_result\n",
    "    \n",
    "#different supervised non-binary classifiers\n",
    "clfs = [(\"SGD\", SGDClassifier(alpha=.0001, n_iter=50, penalty=\"elasticnet\", random_state=2)),\n",
    "    (\"Perceptron\", Perceptron(n_iter=50)),\n",
    "    (\"Ridge Classifier\", RidgeClassifier(tol=1e-2, solver=\"sag\")),\n",
    "    (\"Passive-Aggressive\", PassiveAggressiveClassifier(n_iter=50)),\n",
    "    (\"kNN\", KNeighborsClassifier(n_neighbors=10)),\n",
    "    (\"Random forest\", RandomForestClassifier(n_estimators=100, random_state=1)),\n",
    "    (\"NearestCentroid (aka Rocchio classifier)\", NearestCentroid()),\n",
    "    (\"Multinomial Naive Bayes\", MultinomialNB(alpha=.01)),\n",
    "    (\"Bernouli Naive Base\", BernoulliNB(alpha=.01)),\n",
    "   ]\n",
    "    \n",
    "classification_types = ['coarse', 'flat', 'hierarchical']\n",
    "#classifiers with highest accuracy for coarse and flat\n",
    "max_clfs = {}\n",
    "for clf_level in classification_types:\n",
    "    if clf_level == 'hierarchical':#first classify by coarse, then by flat\n",
    "        max_clf_result = hierarchical_clf(max_clfs['coarse'].get_clf(), clfs, clf_level)  \n",
    "    else:\n",
    "        max_clf_result = choose_classifier(clf_level, clfs, train_quests_processed_tagged, test_quests_processed_tagged)\n",
    "        \n",
    "    max_clfs[clf_level] = max_clf_result\n",
    "    print_metrics(max_clf_result.get_clf(), clf_level, max_clf_result.get_y_pred(), max_clf_result.get_y_test()) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the results of the coarse classifier are pretty good, much better than the baseline classifier. The accuracy is 0.868, and the average precision, recall and f1-score are 0.87. The classifier with the heighest accuracy is SGD.\n",
    "\n",
    "The flat classifier is not accurate as the coarse classifier, which makes sense as there are 50 labels as opposed to 6.\n",
    "\n",
    "The hierarchical classifier does a slightly less better job than the flat, as in the paper. This can be explained by the trade-off between a smaller set of labels, and the accuracy of the coarse classifier.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENTY tagged as DESC\n",
      "--------------------------------------------------------------------------------\n",
      "(What is the birthstone for June ?, 'ENTY', 'substance')\n",
      "(What is the sales tax in Minnesota ?, 'ENTY', 'other')\n",
      "(What is the electrical output in Madrid , Spain ?, 'ENTY', 'other')\n",
      "(What is plastic made of ?, 'ENTY', 'substance')\n",
      "(What is the criterion for being legally blind ?, 'ENTY', 'other')\n",
      "(What birthstone is turquoise ?, 'ENTY', 'substance')\n",
      "(What was the name of the plane Lindbergh flew solo across the Atlantic ?, 'ENTY', 'veh')\n",
      "(What is the active ingredient in baking soda ?, 'ENTY', 'food')\n",
      "================================================================================\n",
      "DESC tagged as ENTY\n",
      "--------------------------------------------------------------------------------\n",
      "(What is done with worn or outdated flags ?, 'DESC', 'desc')\n",
      "(What does a defibrillator do ?, 'DESC', 'desc')\n",
      "(What does your spleen do ?, 'DESC', 'desc')\n",
      "(What do meteorologists do ?, 'DESC', 'desc')\n",
      "(What is the esophagus used for ?, 'DESC', 'reason')\n",
      "(What is acid rain ?, 'DESC', 'def')\n",
      "(What is acetic acid ?, 'DESC', 'def')\n",
      "(What is mad cow disease ?, 'DESC', 'def')\n",
      "(What is die-casting ?, 'DESC', 'def')\n",
      "================================================================================\n",
      "LOC tagged as ENTY\n",
      "--------------------------------------------------------------------------------\n",
      "(What imaginary line is halfway between the North and South Poles ?, 'LOC', 'other')\n",
      "(What is the highest dam in the U.S. ?, 'LOC', 'other')\n",
      "(What peninsula is Spain part of ?, 'LOC', 'other')\n",
      "(What city 's newspaper is called `` The Enquirer '' ?, 'LOC', 'city')\n",
      "(What province is Montreal in ?, 'LOC', 'state')\n",
      "(What French province is cognac produced in ?, 'LOC', 'state')\n",
      "(What is the location of Lake Champlain ?, 'LOC', 'other')\n",
      "(What city 's newspaper is called `` The Star '' ?, 'LOC', 'city')\n",
      "================================================================================\n",
      "NUM tagged as LOC\n",
      "--------------------------------------------------------------------------------\n",
      "(What is the length of the coastline of the state of Alaska ?, 'NUM', 'dist')\n",
      "(What is the distance in miles from the earth to the sun ?, 'NUM', 'dist')\n",
      "(What is the depth of the Nile river ?, 'NUM', 'dist')\n",
      "================================================================================\n",
      "ENTY tagged as HUM\n",
      "--------------------------------------------------------------------------------\n",
      "(What is the proper name for a female walrus ?, 'ENTY', 'animal')\n",
      "(What is a group of turkeys called ?, 'ENTY', 'animal')\n",
      "(The U.S. Department of Treasury first issued paper currency for the U.S. during which war ?, 'ENTY', 'event')\n",
      "(What is a group of frogs called ?, 'ENTY', 'animal')\n",
      "(What was President Lyndon Johnson 's reform program called ?, 'ENTY', 'event')\n",
      "(What is a baby lion called ?, 'ENTY', 'animal')\n",
      "(What was FDR 's dog 's name ?, 'ENTY', 'animal')\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "def get_pair_confusion(label1, label2, y_test, y_pred):\n",
    "    \"\"\"get indexes of samples with label1 that were tagged as label2\"\"\"\n",
    "    return [i for i in range(len(y_pred)) if y_test[i]==label1 and y_pred[i]==label2]\n",
    "\n",
    "clf_results = max_clfs['coarse']\n",
    "y_pred = clf_results.get_y_pred()\n",
    "y_test = clf_results.get_y_test()\n",
    "\n",
    "label_pairs = [('ENTY', 'DESC'), ('DESC', 'ENTY'), ('LOC', 'ENTY'), ('NUM', 'LOC'), ('ENTY', 'HUM')]\n",
    "for label1, label2 in label_pairs:\n",
    "    print(label1 + \" tagged as \" + label2)\n",
    "    print(\"-\" * 80)\n",
    "    conf_inds = get_pair_confusion(label1, label2, y_test, y_pred)\n",
    "    for ind in conf_inds:\n",
    "        print(test_quests_processed_tagged[ind])\n",
    "    print(\"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Analyzing the mistakes made by the classification method, we can observe:\n",
    "\n",
    "**ENTY tagged as DESC:**\n",
    "Questions starting with \"What is\" are usually DESC.\n",
    "\n",
    "**DESC tagged as ENTY:**\n",
    "All these questions start with \"What is\", so it is not clear why they were tagged as entity.\n",
    "\n",
    "**LOC tagged as ENTY:**\n",
    "These questions were tagged as ENTY probably because they don't start with where. The pattern of the question fits to ENTY.\n",
    "\n",
    "**NUM tagged as LOC:**\n",
    "These questions were tagged as LOC probably because they contain a location name such as Alaska, Nile river and earth. The classifier should give more weight to the words: length, distance, depth etc. Maybe those words are missing in the train data.\n",
    "\n",
    "**ENTY tagged as HUM:**\n",
    "A human is a sort of entity, so is is not completely wrong to tag ENTY instead of HUM.\n",
    "To avoid this mistake, we can use a list of animal names, such that if a question subject is an animal, it cannot be tagged as HUM, or give more weight to NER tags, as the questions tagged with HUM usually should have a Person as a subject."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1081\n",
      "animal tagged as ind\n",
      "--------------------------------------------------------------------------------\n",
      "(What is the proper name for a female walrus ?, 'ENTY', 'animal')\n",
      "(What is a baby lion called ?, 'ENTY', 'animal')\n",
      "================================================================================\n",
      "animal tagged as gr\n",
      "--------------------------------------------------------------------------------\n",
      "(What is a group of turkeys called ?, 'ENTY', 'animal')\n",
      "(What is a group of frogs called ?, 'ENTY', 'animal')\n",
      "================================================================================\n",
      "product tagged as event\n",
      "--------------------------------------------------------------------------------\n",
      "(What is the name of the satellite that the Soviet Union sent into space in 1957 ?, 'ENTY', 'product')\n",
      "(What was the name of the first U.S. satellite sent into space ?, 'ENTY', 'product')\n",
      "================================================================================\n",
      "product tagged as other\n",
      "--------------------------------------------------------------------------------\n",
      "(What was the first satellite to go into space ?, 'ENTY', 'product')\n",
      "(What was the most popular toy in 1957 ?, 'ENTY', 'product')\n",
      "================================================================================\n",
      "dist tagged as other\n",
      "--------------------------------------------------------------------------------\n",
      "(What is the diameter of a golf ball ?, 'NUM', 'dist')\n",
      "(What is the distance in miles from the earth to the sun ?, 'NUM', 'dist')\n",
      "(What is the depth of the Nile river ?, 'NUM', 'dist')\n",
      "================================================================================\n",
      "desc tagged as other\n",
      "--------------------------------------------------------------------------------\n",
      "(What is done with worn or outdated flags ?, 'DESC', 'desc')\n",
      "(What does a defibrillator do ?, 'DESC', 'desc')\n",
      "(What does your spleen do ?, 'DESC', 'desc')\n",
      "(What do meteorologists do ?, 'DESC', 'desc')\n",
      "================================================================================\n",
      "exp tagged as def\n",
      "--------------------------------------------------------------------------------\n",
      "(What is TMJ ?, 'ABBR', 'exp')\n",
      "(What does the technical term ISDN mean ?, 'ABBR', 'exp')\n",
      "================================================================================\n",
      "substance tagged as other\n",
      "--------------------------------------------------------------------------------\n",
      "(What mineral helps prevent osteoporosis ?, 'ENTY', 'substance')\n",
      "(What gasses are in the troposphere ?, 'ENTY', 'substance')\n",
      "(What kind of gas is in a fluorescent bulb ?, 'ENTY', 'substance')\n",
      "(What is the birthstone of October ?, 'ENTY', 'substance')\n",
      "================================================================================\n",
      "city tagged as cremat\n",
      "--------------------------------------------------------------------------------\n",
      "(What city 's newspaper is called `` The Enquirer '' ?, 'LOC', 'city')\n",
      "(What city 's newspaper is called `` The Star '' ?, 'LOC', 'city')\n",
      "================================================================================\n",
      "city tagged as other\n",
      "--------------------------------------------------------------------------------\n",
      "(What county is Modesto , California in ?, 'LOC', 'city')\n",
      "(Where is Milan ?, 'LOC', 'city')\n",
      "(What county is Phoenix , AZ in ?, 'LOC', 'city')\n",
      "================================================================================\n",
      "money tagged as count\n",
      "--------------------------------------------------------------------------------\n",
      "(How much was a ticket for the Titanic ?, 'NUM', 'money')\n",
      "(What is the conversion rate between dollars and pounds ?, 'NUM', 'money')\n",
      "================================================================================\n",
      "other tagged as def\n",
      "--------------------------------------------------------------------------------\n",
      "(What hemisphere is the Philippines in ?, 'LOC', 'other')\n",
      "(What is the longest major league baseball-winning streak ?, 'ENTY', 'other')\n",
      "(What is the sales tax in Minnesota ?, 'ENTY', 'other')\n",
      "(What is the melting point of copper ?, 'NUM', 'other')\n",
      "(What is the criterion for being legally blind ?, 'ENTY', 'other')\n",
      "(What is the brightest star ?, 'LOC', 'other')\n",
      "(What is the melting point of gold ?, 'NUM', 'other')\n",
      "(What are Canada 's two territories ?, 'LOC', 'other')\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "clf_results = max_clfs['flat']\n",
    "y_pred = clf_results.get_y_pred()\n",
    "y_test = clf_results.get_y_test()\n",
    "\n",
    "flat_labels = list(set([fine for coarse, dist in train_label_dist_dict.items() for fine, count in dist.items()]))\n",
    "\n",
    "label_pairs = [(label1, label2) for i, label1 in enumerate(flat_labels) for label2 in flat_labels[i+1:] if i<len(flat_labels)-1]\n",
    "print(len(label_pairs))\n",
    "\n",
    "for label1, label2 in label_pairs:\n",
    "    conf_inds = get_pair_confusion(label1, label2, y_test, y_pred)\n",
    "    if len(conf_inds)>1:\n",
    "        print(label1 + \" tagged as \" + label2)\n",
    "        print(\"-\" * 80)\n",
    "        for ind in conf_inds:\n",
    "            print(test_quests_processed_tagged[ind])\n",
    "        print(\"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**other tagged as count**: All these questions ask about population. Population can be regarded as count.\n",
    "\n",
    "**other tagged as city**: \"What New York City structure is also known as the Twin Towers\" - The classifier decided the the focus of the question is New York and not the scructure.\n",
    "\n",
    "**other tagged as period**: the first 2 questions ask about life expectancy, and the third about how often something happens. period can be the right tag for these questions\n",
    "\n",
    "**other tagged as substance**: I have no idea why they were tagged as substance\n",
    "\n",
    "**animal tagged as group** : It is pretty ambigous since these questions ask about groups.\n",
    "\n",
    "overall we can observe that these mistakes were made for ambiguous questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Note that the dataset for fine classification is quite small (5,500 questions in the training dataset for 50 labels). To determine whether the model overfits on this data, we should use cross validation. If the classifier does much better for experimenation with different features, different parameters and different classifiers. Only as a last step, we should run the classifier on the test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q2. Sequence Labelling for Named Entity Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The task of Named Entity Recognition (NER) involves the recognition of names of persons, locations, organizations, dates in free text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "The dataset we will use for this question is derived from the CoNLL 2002 shared task - which is about NER in Spanish and Dutch.\n",
    "\n",
    "The data contains entities of four types: persons (PER), organizations (ORG), locations (LOC) and miscellaneous entities that do not belong to the previous three groups (MISC). The tagging scheme is a variant of the IOB scheme originally put forward by Ramshaw and Marcus (1995). Named entities are assumed to be non-recursive and non-overlapping.\n",
    "\n",
    "An example of a the sentence \"Wolff, currently a journalist in Argentina, played with Del Bosque in the final years of the seventies in Real Madrid.\" tagged by NER:\n",
    "\n",
    "Wol B-PER\n",
    "\n",
    ", O\n",
    "\n",
    "currently O\n",
    "\n",
    "a O\n",
    "\n",
    "journalist O\n",
    "\n",
    "in O\n",
    "\n",
    "Argentina B-LOC\n",
    "\n",
    ", O\n",
    "\n",
    "played O\n",
    "\n",
    "with O\n",
    "\n",
    "Del B-PER\n",
    "\n",
    "Bosque I-PER\n",
    "\n",
    "Words tagged with O are outside of named entities. \n",
    "The B-XXX tag is used for the first word in a named entity of type XXX and IXXX is used for all other words in named entities\n",
    "of type XXX.\n",
    "\n",
    "NER involves 2 sub-tasks: identifying the boundaries of such expressions (the open and close brackets) and labelling the expressions (with tags such as PER, LOC).\n",
    "\n",
    "The Spanish data is a collection of news wire articles made available by the Spanish EFE News Agency. The articles are from May 2000.\n",
    "\n",
    "The Dutch data consist of four editions of the Belgian newspaper \"De Morgen\" of 2000 (June 2, July 1, August 1 and September 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import conll2002\n",
    "\n",
    "train_sp = list(conll2002.iob_sents('esp.train')) # In Spanish\n",
    "dev_sp = list(conll2002.iob_sents('esp.testa')) # In Spanish\n",
    "test_sp = list(conll2002.iob_sents('esp.testb')) # In Spanish\n",
    "\n",
    "train_du = list(conll2002.iob_sents('ned.train')) # In Dutch\n",
    "dev_du = list(conll2002.iob_sents('ned.testa')) # In Dutch\n",
    "test_du = list(conll2002.iob_sents('ned.testb')) # In Dutch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC', 'O']"
      ]
     },
     "execution_count": 264,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LABELS = [label1+label2 for label2 in [\"PER\", \"ORG\", \"LOC\", \"MISC\"] for label1 in [\"B-\", \"I-\"]]\n",
    "LABELS.append(\"O\")\n",
    "LABELS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Melbourne ( Australia ) , 25 may ( EFE ) .'"
      ]
     },
     "execution_count": 265,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\" \".join([word[0] for word in train_sp[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Melbourne', 'NP', 'B-LOC'),\n",
       " ('(', 'Fpa', 'O'),\n",
       " ('Australia', 'NP', 'B-LOC'),\n",
       " (')', 'Fpt', 'O'),\n",
       " (',', 'Fc', 'O'),\n",
       " ('25', 'Z', 'O'),\n",
       " ('may', 'NC', 'O'),\n",
       " ('(', 'Fpa', 'O'),\n",
       " ('EFE', 'NC', 'B-ORG'),\n",
       " (')', 'Fpt', 'O'),\n",
       " ('.', 'Fp', 'O')]"
      ]
     },
     "execution_count": 266,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_sp[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of sentences in tha Spanish dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train 8323\n",
      "dev 1915\n",
      "test 1517\n"
     ]
    }
   ],
   "source": [
    "print(\"train\", len(train_sp))\n",
    "print(\"dev\", len(dev_sp))\n",
    "print(\"test\", len(test_sp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of sentences in the Dutch dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train 15806\n",
      "dev 2895\n",
      "test 5195\n"
     ]
    }
   ],
   "source": [
    "print(\"train\", len(train_du))\n",
    "print(\"dev\", len(dev_du))\n",
    "print(\"test\", len(test_du))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how the label distribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spanish train\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B-LOC</th>\n",
       "      <th>B-MISC</th>\n",
       "      <th>B-ORG</th>\n",
       "      <th>B-PER</th>\n",
       "      <th>I-LOC</th>\n",
       "      <th>I-MISC</th>\n",
       "      <th>I-ORG</th>\n",
       "      <th>I-PER</th>\n",
       "      <th>O</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4913.000000</td>\n",
       "      <td>2173.000000</td>\n",
       "      <td>7390.000000</td>\n",
       "      <td>4321.000000</td>\n",
       "      <td>1891.000000</td>\n",
       "      <td>3212.00000</td>\n",
       "      <td>4992.000000</td>\n",
       "      <td>3903.000000</td>\n",
       "      <td>231920.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <td>1.855958</td>\n",
       "      <td>0.820883</td>\n",
       "      <td>2.791682</td>\n",
       "      <td>1.632322</td>\n",
       "      <td>0.714353</td>\n",
       "      <td>1.21338</td>\n",
       "      <td>1.885802</td>\n",
       "      <td>1.474416</td>\n",
       "      <td>87.611205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  B-LOC       B-MISC        B-ORG        B-PER        I-LOC  \\\n",
       "count       4913.000000  2173.000000  7390.000000  4321.000000  1891.000000   \n",
       "proportion  1.855958     0.820883     2.791682     1.632322     0.714353      \n",
       "\n",
       "                I-MISC        I-ORG        I-PER              O  \n",
       "count       3212.00000  4992.000000  3903.000000  231920.000000  \n",
       "proportion  1.21338     1.885802     1.474416     87.611205      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spanish dev\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B-LOC</th>\n",
       "      <th>B-MISC</th>\n",
       "      <th>B-ORG</th>\n",
       "      <th>B-PER</th>\n",
       "      <th>I-LOC</th>\n",
       "      <th>I-MISC</th>\n",
       "      <th>I-ORG</th>\n",
       "      <th>I-PER</th>\n",
       "      <th>O</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>984.000000</td>\n",
       "      <td>445.000000</td>\n",
       "      <td>1700.000000</td>\n",
       "      <td>1222.000000</td>\n",
       "      <td>337.000000</td>\n",
       "      <td>654.000000</td>\n",
       "      <td>1366.000000</td>\n",
       "      <td>859.000000</td>\n",
       "      <td>45356.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <td>1.859305</td>\n",
       "      <td>0.840844</td>\n",
       "      <td>3.212214</td>\n",
       "      <td>2.309015</td>\n",
       "      <td>0.636774</td>\n",
       "      <td>1.235758</td>\n",
       "      <td>2.581108</td>\n",
       "      <td>1.623113</td>\n",
       "      <td>85.701869</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 B-LOC      B-MISC        B-ORG        B-PER       I-LOC  \\\n",
       "count       984.000000  445.000000  1700.000000  1222.000000  337.000000   \n",
       "proportion  1.859305    0.840844    3.212214     2.309015     0.636774     \n",
       "\n",
       "                I-MISC        I-ORG       I-PER             O  \n",
       "count       654.000000  1366.000000  859.000000  45356.000000  \n",
       "proportion  1.235758    2.581108     1.623113    85.701869     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spanish test\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B-LOC</th>\n",
       "      <th>B-MISC</th>\n",
       "      <th>B-ORG</th>\n",
       "      <th>B-PER</th>\n",
       "      <th>I-LOC</th>\n",
       "      <th>I-MISC</th>\n",
       "      <th>I-ORG</th>\n",
       "      <th>I-PER</th>\n",
       "      <th>O</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1084.000000</td>\n",
       "      <td>339.000000</td>\n",
       "      <td>1400.000000</td>\n",
       "      <td>735.000000</td>\n",
       "      <td>325.000000</td>\n",
       "      <td>557.000000</td>\n",
       "      <td>1104.000000</td>\n",
       "      <td>634.00000</td>\n",
       "      <td>45355.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <td>2.103506</td>\n",
       "      <td>0.657831</td>\n",
       "      <td>2.716706</td>\n",
       "      <td>1.426271</td>\n",
       "      <td>0.630664</td>\n",
       "      <td>1.080861</td>\n",
       "      <td>2.142317</td>\n",
       "      <td>1.23028</td>\n",
       "      <td>88.011565</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  B-LOC      B-MISC        B-ORG       B-PER       I-LOC  \\\n",
       "count       1084.000000  339.000000  1400.000000  735.000000  325.000000   \n",
       "proportion  2.103506     0.657831    2.716706     1.426271    0.630664     \n",
       "\n",
       "                I-MISC        I-ORG      I-PER             O  \n",
       "count       557.000000  1104.000000  634.00000  45355.000000  \n",
       "proportion  1.080861    2.142317     1.23028    88.011565     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dutch train\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B-LOC</th>\n",
       "      <th>B-MISC</th>\n",
       "      <th>B-ORG</th>\n",
       "      <th>B-PER</th>\n",
       "      <th>I-LOC</th>\n",
       "      <th>I-MISC</th>\n",
       "      <th>I-ORG</th>\n",
       "      <th>I-PER</th>\n",
       "      <th>O</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>3208.000000</td>\n",
       "      <td>3338.000000</td>\n",
       "      <td>2082.000000</td>\n",
       "      <td>4716.000000</td>\n",
       "      <td>467.000000</td>\n",
       "      <td>1405.000000</td>\n",
       "      <td>1199.000000</td>\n",
       "      <td>2883.000000</td>\n",
       "      <td>183346.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <td>1.583072</td>\n",
       "      <td>1.647224</td>\n",
       "      <td>1.027418</td>\n",
       "      <td>2.327234</td>\n",
       "      <td>0.230453</td>\n",
       "      <td>0.693334</td>\n",
       "      <td>0.591678</td>\n",
       "      <td>1.422692</td>\n",
       "      <td>90.476895</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  B-LOC       B-MISC        B-ORG        B-PER       I-LOC  \\\n",
       "count       3208.000000  3338.000000  2082.000000  4716.000000  467.000000   \n",
       "proportion  1.583072     1.647224     1.027418     2.327234     0.230453     \n",
       "\n",
       "                 I-MISC        I-ORG        I-PER              O  \n",
       "count       1405.000000  1199.000000  2883.000000  183346.000000  \n",
       "proportion  0.693334     0.591678     1.422692     90.476895      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dutch dev\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B-LOC</th>\n",
       "      <th>B-MISC</th>\n",
       "      <th>B-ORG</th>\n",
       "      <th>B-PER</th>\n",
       "      <th>I-LOC</th>\n",
       "      <th>I-MISC</th>\n",
       "      <th>I-ORG</th>\n",
       "      <th>I-PER</th>\n",
       "      <th>O</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>479.000000</td>\n",
       "      <td>748.000000</td>\n",
       "      <td>686.000000</td>\n",
       "      <td>703.000000</td>\n",
       "      <td>64.00000</td>\n",
       "      <td>215.000000</td>\n",
       "      <td>396.00000</td>\n",
       "      <td>423.000000</td>\n",
       "      <td>33973.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <td>1.270995</td>\n",
       "      <td>1.984769</td>\n",
       "      <td>1.820256</td>\n",
       "      <td>1.865365</td>\n",
       "      <td>0.16982</td>\n",
       "      <td>0.570488</td>\n",
       "      <td>1.05076</td>\n",
       "      <td>1.122403</td>\n",
       "      <td>90.145143</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 B-LOC      B-MISC       B-ORG       B-PER     I-LOC  \\\n",
       "count       479.000000  748.000000  686.000000  703.000000  64.00000   \n",
       "proportion  1.270995    1.984769    1.820256    1.865365    0.16982    \n",
       "\n",
       "                I-MISC      I-ORG       I-PER             O  \n",
       "count       215.000000  396.00000  423.000000  33973.000000  \n",
       "proportion  0.570488    1.05076    1.122403    90.145143     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dutch test\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B-LOC</th>\n",
       "      <th>B-MISC</th>\n",
       "      <th>B-ORG</th>\n",
       "      <th>B-PER</th>\n",
       "      <th>I-LOC</th>\n",
       "      <th>I-MISC</th>\n",
       "      <th>I-ORG</th>\n",
       "      <th>I-PER</th>\n",
       "      <th>O</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>774.000000</td>\n",
       "      <td>1187.000000</td>\n",
       "      <td>882.000000</td>\n",
       "      <td>1098.000000</td>\n",
       "      <td>49.000000</td>\n",
       "      <td>410.000000</td>\n",
       "      <td>551.0</td>\n",
       "      <td>807.000000</td>\n",
       "      <td>63117.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <td>1.123775</td>\n",
       "      <td>1.723412</td>\n",
       "      <td>1.280581</td>\n",
       "      <td>1.594192</td>\n",
       "      <td>0.071143</td>\n",
       "      <td>0.595281</td>\n",
       "      <td>0.8</td>\n",
       "      <td>1.171688</td>\n",
       "      <td>91.639927</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 B-LOC       B-MISC       B-ORG        B-PER      I-LOC  \\\n",
       "count       774.000000  1187.000000  882.000000  1098.000000  49.000000   \n",
       "proportion  1.123775    1.723412     1.280581    1.594192     0.071143    \n",
       "\n",
       "                I-MISC  I-ORG       I-PER             O  \n",
       "count       410.000000  551.0  807.000000  63117.000000  \n",
       "proportion  0.595281    0.8    1.171688    91.639927     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "import pandas as pd\n",
    "from IPython.display import display\n",
    "\n",
    "def get_label_dist(dataset):\n",
    "    label_counter = Counter([word[-1] for sent in dataset for word in sent])\n",
    "    label_dist = {label:{\"count\": count, \"proportion\": count/sum(label_counter.values())*100} \n",
    "                  for label, count in label_counter.items()}\n",
    "    return pd.DataFrame(label_dist)\n",
    "\n",
    "print(\"Spanish train\")\n",
    "display(get_label_dist(train_sp))\n",
    "print(\"Spanish dev\")\n",
    "display(get_label_dist(dev_sp))\n",
    "print(\"Spanish test\")\n",
    "display(get_label_dist(test_sp))\n",
    "\n",
    "print(\"Dutch train\")\n",
    "display(get_label_dist(train_du))\n",
    "print(\"Dutch dev\")\n",
    "display(get_label_dist(dev_du))\n",
    "print(\"Dutch test\")\n",
    "display(get_label_dist(test_du))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the labels are unbalanced. In the Spanish dataset, ~88% of the labels in the train, dev and test dataset are \"O\", which makes sense as most words in a sentence are not entities. This means that if we tag each word in the test set as \"O\" we will get 88 precent accuracy. The same goes for the Dutch dataset ~90% tagged as \"O\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Features that have been found appropriate for NER in previous work:\n",
    "1. The word form (the string as it appears in the sentence)\n",
    "2. The POS of the word (which is provided in the dataset)\n",
    "3. ORT - a feature that captures the orthographic (letter) structure of the word. It can have any of the following values: number, contains-digit, contains-hyphen, capitalized, all-capitals, URL, punctuation, regular.\n",
    "4. prefix1: first letter of the word\n",
    "5. prefix2: first two letters of the word\n",
    "6. prefix3: first three letters of the word\n",
    "7. suffix1: last letter of the word\n",
    "8. suffix2: last two letters of the word\n",
    "9. suffix3: last three letters of the word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "def features(sentence, index):\n",
    "    word = sentence[index][0]\n",
    "    postag = sentence[index][1]\n",
    "    PUNCT = set(string.punctuation)\n",
    "    \n",
    "    return{ \n",
    "        'word': word,\n",
    "        'is_first': index == 0,\n",
    "        'is_last': index == len(sentence) - 1,\n",
    "        'is_punct': word in PUNCT,\n",
    "        'is_capitalized': word[0].isupper(),\n",
    "        'is_all_caps': word.isupper(),\n",
    "        'is_all_lower': word.islower(),\n",
    "        'prefix-1': word[0],\n",
    "        'prefix-2': word[:2],\n",
    "        'prefix-3': word[:3],\n",
    "        'suffix-1': word[-1],\n",
    "        'suffix-2': word[-2:],\n",
    "        'suffix-3': word[-3:],\n",
    "        'has_hyphen': '-' in word,\n",
    "        'is_numeric': word.isdigit(),\n",
    "        'capitals_inside': any(c.isupper() for c in word[1:]),\n",
    "        'pos': postag,\n",
    "        'pos_prefix-2': postag[:2]\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'capitals_inside': False,\n",
       " 'has_hyphen': False,\n",
       " 'is_all_caps': False,\n",
       " 'is_all_lower': False,\n",
       " 'is_capitalized': True,\n",
       " 'is_first': True,\n",
       " 'is_last': False,\n",
       " 'is_numeric': False,\n",
       " 'is_punct': False,\n",
       " 'pos': 'NP',\n",
       " 'pos_prefix-2': 'NP',\n",
       " 'prefix-1': 'M',\n",
       " 'prefix-2': 'Me',\n",
       " 'prefix-3': 'Mel',\n",
       " 'suffix-1': 'e',\n",
       " 'suffix-2': 'ne',\n",
       " 'suffix-3': 'rne',\n",
       " 'word': 'Melbourne'}"
      ]
     },
     "execution_count": 271,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features(train_sp[0], 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import metrics\n",
    "\n",
    "def create_dataset(dataset):\n",
    "    \"\"\"\n",
    "    returns:\n",
    "    x: a list of dictionaries of the dataset, each dictionary contains the features of a word\n",
    "    y: the tags of words in the dataset\n",
    "    \"\"\"\n",
    "    x = [features(sent, i) for sent in dataset for i in range(len(sent))]\n",
    "    y = [word[-1] for sent in dataset for word in sent]\n",
    "    \n",
    "    return x, y\n",
    "\n",
    "x_train_sp, y_train_sp = create_dataset(train_sp)\n",
    "x_dev_sp, y_dev_sp = create_dataset(dev_sp) \n",
    "x_test_sp, y_test_sp = create_dataset(test_sp) \n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('vectorizer', DictVectorizer(sparse=True)),\n",
    "    ('classifier', LogisticRegression())\n",
    "])\n",
    "\n",
    "pipeline.fit(x_train_sp, y_train_sp)\n",
    "y_pred_dev_sp = pipeline.predict(x_dev_sp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.932033331444\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.699     0.635     0.666      1222\n",
      "      I-PER      0.537     0.587     0.561       859\n",
      "      B-ORG      0.739     0.726     0.733      1700\n",
      "      I-ORG      0.575     0.285     0.381      1366\n",
      "      B-LOC      0.548     0.788     0.646       984\n",
      "      I-LOC      0.674     0.368     0.476       337\n",
      "     B-MISC      0.554     0.438     0.489       445\n",
      "     I-MISC      0.506     0.193     0.279       654\n",
      "          O      0.976     0.997     0.986     45356\n",
      "\n",
      "avg / total      0.925     0.932     0.925     52923\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy\", metrics.accuracy_score(y_dev_sp, y_pred_dev_sp))\n",
    "print(metrics.classification_report(y_dev_sp, y_pred_dev_sp, digits=3, labels=LABELS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At first glance it looks like the accuracy and the average precision, recall and f1-score are high. But remember that the data is unbalanced - almost 90% of the words are tagged as \"O\". It can be notices the precision, recall and f1 scores for most labels are very low, the average is high because of the \"O\" tag. \n",
    "\n",
    "To get a better evaluation, we will remove \"O\" tags and then calculate the measures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.6254550970873787\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.699     0.635     0.666      1222\n",
      "      I-PER      0.537     0.587     0.561       859\n",
      "      B-ORG      0.739     0.726     0.733      1700\n",
      "      I-ORG      0.575     0.285     0.381      1366\n",
      "      B-LOC      0.548     0.788     0.646       984\n",
      "      I-LOC      0.674     0.368     0.476       337\n",
      "     B-MISC      0.554     0.438     0.489       445\n",
      "     I-MISC      0.506     0.193     0.279       654\n",
      "\n",
      "avg / total      0.621     0.545     0.563      7567\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def accuracy(y_pred, y, exclude):\n",
    "    correct = [l == r for (l, r) in zip(y_pred, y) if r not in exclude]\n",
    "    return sum(correct) / len(correct)\n",
    "\n",
    "print(\"Accuracy\", accuracy(y_dev_sp, y_pred_dev_sp, [\"O\"]))\n",
    "print(metrics.classification_report(y_dev_sp, y_pred_dev_sp, digits=3, labels=LABELS[:-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see that that the measures are pretty low.\n",
    "\n",
    "We will try to improve the classification by using different parameters of logistic regression classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import metrics\n",
    "import collections\n",
    "\n",
    "class CLFResult():\n",
    "    \"\"\"represents a classifier result\"\"\"\n",
    "    def __init__(self, clf, y_pred, y):\n",
    "        self.clf = clf\n",
    "        self.y_pred = y_pred\n",
    "        self.y = y\n",
    "    \n",
    "    def get_clf(self):\n",
    "        return self.clf\n",
    "    \n",
    "    def get_y_pred(self):\n",
    "        return self.y_pred\n",
    "    \n",
    "    def get_y(self):\n",
    "        return self.y\n",
    "\n",
    "def create_dataset(dataset, features):\n",
    "    \"\"\"\n",
    "    returns:\n",
    "    x: a list of dictionaries of the dataset, each dictionary contains the features of a word\n",
    "    y: the tags of words in the dataset\n",
    "    \"\"\"\n",
    "    x = [features(sent, i) for sent in dataset for i in range(len(sent))]\n",
    "    y = [word[-1] for sent in dataset for word in sent]\n",
    "    \n",
    "    return x, y\n",
    "\n",
    "def accuracy(y_pred, y, exclude):\n",
    "    \"\"\"computes average accuracy score, excluding categories in exclude\"\"\"\n",
    "    correct = [l == r for (l, r) in zip(y_pred, y) if r not in exclude]\n",
    "    return sum(correct) / len(correct)\n",
    "\n",
    "def get_pipeline(c):\n",
    "    return Pipeline([\n",
    "        ('vectorizer', DictVectorizer(sparse=True)),\n",
    "        #the sag solver is for multiclass problems, and is usually faster than other solvers\n",
    "        ('classifier', LogisticRegression(multi_class='multinomial', max_iter=1000,  solver='sag', C=c))\n",
    "    ])\n",
    "\n",
    "def compare_models(x_train, y_train, x_dev, y_dev, C):\n",
    "    predicts = {}\n",
    "    for c in C:\n",
    "        print(\"Classification started with c=\", c)\n",
    "\n",
    "        clf = get_pipeline(c)\n",
    "        clf.fit(x_train, y_train)\n",
    "        y_pred = clf.predict(x_dev)\n",
    "        predicts[c] = CLFResult(clf, y_pred, y_dev)\n",
    "\n",
    "        print(\"Classification ended with c=\", c)\n",
    "    return predicts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification started with c= 0.01\n",
      "Classification ended with c= 0.01\n",
      "Classification started with c= 1\n",
      "Classification ended with c= 1\n",
      "Classification started with c= 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Dina\\Programs\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\sag.py:286: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification ended with c= 10\n",
      "Classification started with c= 100\n",
      "Classification ended with c= 100\n"
     ]
    }
   ],
   "source": [
    "x_train_sp, y_train_sp = create_dataset(train_sp, features)\n",
    "x_dev_sp, y_dev_sp = create_dataset(dev_sp, features) \n",
    "x_test_sp, y_test_sp = create_dataset(test_sp, features) \n",
    "\n",
    "#Inverse of regularization strength. Large values give more freedom to the model, smaller values constrain the model.\n",
    "C = [0.01, 1, 10, 100]\n",
    "\n",
    "predicts = compare_models(x_train_sp, y_train_sp, x_dev_sp, y_dev_sp, C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_report(predicts, C=predicts.keys()):\n",
    "    for c in C:\n",
    "        cls_res = predicts[c]\n",
    "        y_dev_sp = cls_res.get_y_pred()\n",
    "        y_pred_dev_sp = cls_res.get_y()\n",
    "        print(\"C:\", c)\n",
    "        print(\"-\"*80)\n",
    "        print(\"Accuracy\", accuracy(y_dev_sp, y_pred_dev_sp, [\"O\"]))\n",
    "        print(metrics.classification_report(y_dev_sp, y_pred_dev_sp, digits=3, labels=LABELS[:-1]))\n",
    "        print(\"-\"*80)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C: 0.01\n",
      "--------------------------------------------------------------------------------\n",
      "Accuracy 0.42130302629840094\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.516     0.569     0.541      1107\n",
      "      I-PER      0.478     0.421     0.448       976\n",
      "      B-ORG      0.686     0.540     0.604      2163\n",
      "      I-ORG      0.167     0.435     0.241       524\n",
      "      B-LOC      0.638     0.389     0.484      1613\n",
      "      I-LOC      0.095     0.821     0.170        39\n",
      "     B-MISC      0.182     0.704     0.289       115\n",
      "     I-MISC      0.017     0.647     0.033        17\n",
      "\n",
      "avg / total      0.559     0.486     0.502      6554\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "C: 1\n",
      "--------------------------------------------------------------------------------\n",
      "Accuracy 0.543940795559667\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.623     0.703     0.661      1082\n",
      "      I-PER      0.581     0.529     0.554       943\n",
      "      B-ORG      0.721     0.744     0.732      1647\n",
      "      I-ORG      0.288     0.577     0.384       681\n",
      "      B-LOC      0.791     0.551     0.650      1411\n",
      "      I-LOC      0.383     0.648     0.481       199\n",
      "     B-MISC      0.452     0.540     0.492       372\n",
      "     I-MISC      0.199     0.468     0.279       278\n",
      "\n",
      "avg / total      0.608     0.622     0.601      6613\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "C: 10\n",
      "--------------------------------------------------------------------------------\n",
      "Accuracy 0.54473371217127\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.624     0.711     0.665      1073\n",
      "      I-PER      0.570     0.546     0.558       898\n",
      "      B-ORG      0.718     0.750     0.734      1627\n",
      "      I-ORG      0.288     0.579     0.385       681\n",
      "      B-LOC      0.793     0.561     0.657      1391\n",
      "      I-LOC      0.407     0.566     0.473       242\n",
      "     B-MISC      0.452     0.519     0.483       387\n",
      "     I-MISC      0.208     0.395     0.273       344\n",
      "\n",
      "avg / total      0.601     0.621     0.599      6643\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "C: 100\n",
      "--------------------------------------------------------------------------------\n",
      "Accuracy 0.5346901017576319\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.607     0.703     0.652      1055\n",
      "      I-PER      0.548     0.533     0.541       883\n",
      "      B-ORG      0.707     0.742     0.724      1620\n",
      "      I-ORG      0.280     0.545     0.370       701\n",
      "      B-LOC      0.789     0.554     0.651      1401\n",
      "      I-LOC      0.407     0.552     0.468       248\n",
      "     B-MISC      0.449     0.514     0.480       389\n",
      "     I-MISC      0.208     0.379     0.269       359\n",
      "\n",
      "avg / total      0.589     0.608     0.587      6656\n",
      "\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print_report(predicts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The heighest scires are obtained using c=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add features of previous and next word and see if it improves the classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 279,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import string\n",
    "def features2(sentence, index):\n",
    "    word = sentence[index][0]\n",
    "    postag = sentence[index][1]\n",
    "    PUNCT = set(string.punctuation)\n",
    "    \n",
    "    return{ \n",
    "        'word': word,\n",
    "        'is_first': index == 0,\n",
    "        'is_last': index == len(sentence) - 1,\n",
    "        'is_punct': word in PUNCT,\n",
    "        'is_capitalized': word[0].isupper(),\n",
    "        'is_all_caps': word.isupper(),\n",
    "        'is_all_lower': word.islower(),\n",
    "        'prefix-1': word[0],\n",
    "        'prefix-2': word[:2],\n",
    "        'prefix-3': word[:3],\n",
    "        'suffix-1': word[-1],\n",
    "        'suffix-2': word[-2:],\n",
    "        'suffix-3': word[-3:],\n",
    "        'prev_word': '' if index == 0 else sentence[index - 1][0],\n",
    "        'prev_word_is_punct': False if index == 0 else sentence[index - 1][0] in PUNCT,\n",
    "        'prev_word_is_cap': False if index == 0 else sentence[index - 1][0][0].isupper(),\n",
    "        'prev_word_is_all_cap': False if index == 0 else sentence[index - 1][0].isupper(),\n",
    "        'prev_word_is_all_low': False if index == 0 else sentence[index - 1][0].islower(),\n",
    "        'prev_word_pos': False if index == 0 else sentence[index - 1][1],\n",
    "        'prev_word_pos_prefix-2': False if index == 0 else sentence[index - 1][1][:2],\n",
    "        'next_word': '' if index == len(sentence) - 1 else sentence[index + 1][0],\n",
    "        'next_word_is_punct': False if index == len(sentence) - 1 else sentence[index + 1][0] in PUNCT,\n",
    "        'next_word_is_cap': False if index == len(sentence) - 1 else sentence[index + 1][0][0].isupper(),\n",
    "        'next_word_is_all_cap': False if index == len(sentence) - 1 else sentence[index + 1][0].isupper(),\n",
    "        'next_word_is_all_low': False if index == len(sentence) - 1 else sentence[index + 1][0].islower(),\n",
    "        'next_word_pos': False if index == len(sentence) - 1 else sentence[index + 1][1],\n",
    "        'next_word_pos_prefix-2': False if index == len(sentence) - 1 else sentence[index + 1][1][:2],\n",
    "        'has_hyphen': '-' in word,\n",
    "        'is_numeric': word.isdigit(),\n",
    "        'capitals_inside': any(c.isupper() for c in word[1:]),\n",
    "        'pos': postag,\n",
    "        'pos_prefix-2': postag[:2]\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification started with c= 1\n",
      "Classification ended with c= 1\n"
     ]
    }
   ],
   "source": [
    "x_train_sp, y_train_sp = create_dataset(train_sp, features2)\n",
    "x_dev_sp, y_dev_sp = create_dataset(dev_sp, features2) \n",
    "x_test_sp, y_test_sp = create_dataset(test_sp, features2) \n",
    "\n",
    "#Inverse of regularization strength. Large values give more freedom to the model, smaller values constrain the model.\n",
    "C = [1]\n",
    "\n",
    "predicts2 = compare_models(x_train_sp, y_train_sp, x_dev_sp, y_dev_sp, C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results on validation set with basic features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C: 1\n",
      "--------------------------------------------------------------------------------\n",
      "Accuracy 0.543940795559667\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.623     0.703     0.661      1082\n",
      "      I-PER      0.581     0.529     0.554       943\n",
      "      B-ORG      0.721     0.744     0.732      1647\n",
      "      I-ORG      0.288     0.577     0.384       681\n",
      "      B-LOC      0.791     0.551     0.650      1411\n",
      "      I-LOC      0.383     0.648     0.481       199\n",
      "     B-MISC      0.452     0.540     0.492       372\n",
      "     I-MISC      0.199     0.468     0.279       278\n",
      "\n",
      "avg / total      0.608     0.622     0.601      6613\n",
      "\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print_report(predicts, [1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results on validation set with improved features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 283,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C: 1\n",
      "--------------------------------------------------------------------------------\n",
      "Accuracy 0.6881194661028148\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.802     0.873     0.836      1122\n",
      "      I-PER      0.854     0.816     0.835       900\n",
      "      B-ORG      0.765     0.808     0.786      1611\n",
      "      I-ORG      0.558     0.707     0.624      1078\n",
      "      B-LOC      0.789     0.597     0.680      1300\n",
      "      I-LOC      0.608     0.663     0.635       309\n",
      "     B-MISC      0.483     0.592     0.532       363\n",
      "     I-MISC      0.358     0.571     0.440       410\n",
      "\n",
      "avg / total      0.710     0.734     0.716      7093\n",
      "\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print_report(predicts2, [1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f1 scores are higher for all labels, espacially for I-XXX labels. This can be explained by the fact that it helps to know the previous word in order tag a word with I-XXX label."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Results on test set with improved features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.756482912588013\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.844     0.888     0.865       735\n",
      "      I-PER      0.844     0.907     0.875       634\n",
      "      B-ORG      0.785     0.849     0.816      1400\n",
      "      I-ORG      0.715     0.638     0.674      1104\n",
      "      B-LOC      0.752     0.743     0.747      1084\n",
      "      I-LOC      0.703     0.452     0.551       325\n",
      "     B-MISC      0.546     0.457     0.498       339\n",
      "     I-MISC      0.580     0.318     0.411       557\n",
      "\n",
      "avg / total      0.744     0.713     0.722      6178\n",
      "\n"
     ]
    }
   ],
   "source": [
    "clf_result = predicts2[1]\n",
    "clf = clf_result.get_clf()\n",
    "y_pred_sp = clf.predict(x_test_sp)\n",
    "\n",
    "print(\"Accuracy\", accuracy(y_test_sp, y_pred_sp, [\"O\"]))\n",
    "print(metrics.classification_report(y_test_sp, y_pred_sp, digits=3, labels=LABELS[:-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The evaluation of the test set shows that the model is succesful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q2.3 Comparing Models\n",
    "\n",
    "In this part we will compare the model we designed with a PyTorch model taken from https://github.com/DSKSD/DeepNLP-models-Pytorch/blob/master/notebooks/04.Window-Classifier-for-NER.ipynb. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Window Classification and Neural Networks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "import nltk\n",
    "import random\n",
    "import numpy as np\n",
    "from collections import Counter\n",
    "flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "random.seed(1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.3.0b0+591e73e\n",
      "3.2.3\n"
     ]
    }
   ],
   "source": [
    "print(torch.__version__)\n",
    "print(nltk.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "USE_CUDA = torch.cuda.is_available()\n",
    "#gpus = [0]\n",
    "#torch.cuda.set_device(gpus[0])\n",
    "\n",
    "FloatTensor = torch.cuda.FloatTensor if USE_CUDA else torch.FloatTensor\n",
    "LongTensor = torch.cuda.LongTensor if USE_CUDA else torch.LongTensor\n",
    "ByteTensor = torch.cuda.ByteTensor if USE_CUDA else torch.ByteTensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def getBatch(batch_size, train_data):\n",
    "    random.shuffle(train_data)\n",
    "    sindex = 0\n",
    "    eindex = batch_size\n",
    "    while eindex < len(train_data):\n",
    "        batch = train_data[sindex: eindex]\n",
    "        temp = eindex\n",
    "        eindex = eindex + batch_size\n",
    "        sindex = temp\n",
    "        yield batch\n",
    "    \n",
    "    if eindex >= len(train_data):\n",
    "        batch = train_data[sindex:]\n",
    "        yield batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prepare_sequence(seq, word2index):\n",
    "    idxs = list(map(lambda w: word2index[w] if word2index.get(w) is not None else word2index[\"<UNK>\"], seq))\n",
    "    return Variable(LongTensor(idxs))\n",
    "\n",
    "def prepare_word(word, word2index):\n",
    "    return Variable(LongTensor([word2index[word]]) if word2index.get(word) is not None else LongTensor([word2index[\"<UNK>\"]]))\n",
    "\n",
    "def prepare_tag(tag,tag2index):\n",
    "    return Variable(LongTensor([tag2index[tag]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model classifies universal data of the Spanish and Dutch sentences together "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# combine train and dev of Spanish and Dutch to one universal train dataset\n",
    "train_uni = train_sp + dev_sp + train_du + dev_du\n",
    "# combine test of Spanish and Dutch to one universal test dataset\n",
    "test_uni = test_sp + test_du\n",
    "\n",
    "def convert_data(corpus):\n",
    "    data = []\n",
    "    for cor in corpus:\n",
    "        sent, pos, tag = list(zip(*cor))\n",
    "        data.append([sent, tag])\n",
    "    return data\n",
    "        \n",
    "train_uni_data = convert_data(train_uni)\n",
    "test_uni_data = convert_data(test_uni)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Melbourne', 'NP', 'B-LOC'),\n",
       " ('(', 'Fpa', 'O'),\n",
       " ('Australia', 'NP', 'B-LOC'),\n",
       " (')', 'Fpt', 'O'),\n",
       " (',', 'Fc', 'O'),\n",
       " ('25', 'Z', 'O'),\n",
       " ('may', 'NC', 'O'),\n",
       " ('(', 'Fpa', 'O'),\n",
       " ('EFE', 'NC', 'B-ORG'),\n",
       " (')', 'Fpt', 'O'),\n",
       " ('.', 'Fp', 'O')]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_uni[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "28939\n",
      "[('Melbourne', '(', 'Australia', ')', ',', '25', 'may', '(', 'EFE', ')', '.'), ('B-LOC', 'O', 'B-LOC', 'O', 'O', 'O', 'O', 'O', 'B-ORG', 'O', 'O')]\n"
     ]
    }
   ],
   "source": [
    "print(len(train_uni))\n",
    "print(train_uni_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sents_train, tags_train = list(zip(*train_uni_data))\n",
    "sents_test, tags_test = list(zip(*test_uni_data))\n",
    "\n",
    "vocab = list(set(flatten(sents_train+sents_test)))\n",
    "tagset = list(set(flatten(tags_train+tags_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word2index={'<UNK>' : 0, '<DUMMY>' : 1} # dummy token is for start or end of sentence\n",
    "for vo in vocab:\n",
    "    if word2index.get(vo) is None:\n",
    "        word2index[vo] = len(word2index)\n",
    "index2word = {v:k for k, v in word2index.items()}\n",
    "\n",
    "tag2index = {}\n",
    "for tag in tagset:\n",
    "    if tag2index.get(tag) is None:\n",
    "        tag2index[tag] = len(tag2index)\n",
    "index2tag={v:k for k, v in tag2index.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is segmented to windows of size 2, each window representing a word in a sentence and 2 previous and 2 next words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_windows(WINDOW_SIZE, data):\n",
    "    windows = []\n",
    "    for sample in data:\n",
    "        dummy = ['<DUMMY>'] * WINDOW_SIZE\n",
    "        window = list(nltk.ngrams(dummy + list(sample[0]) + dummy, WINDOW_SIZE * 2 + 1))\n",
    "        windows.extend([[list(window[i]), sample[1][i]] for i in range(len(sample[0]))])\n",
    "    return windows\n",
    "    \n",
    "WINDOW_SIZE = 2\n",
    "windows_train = create_windows(WINDOW_SIZE, train_uni_data)\n",
    "windows_test = create_windows(WINDOW_SIZE, test_uni_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Window of the word 'Melbourne'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['<DUMMY>', '<DUMMY>', 'Melbourne', '(', 'Australia'], 'B-LOC']"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "windows_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['<DUMMY>', 'Melbourne', '(', 'Australia', ')'], 'O']"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "windows_train[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['<DUMMY>', '<DUMMY>', 'La', 'Coruña', ','], 'B-LOC']"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "windows_test[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "557969"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(windows_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "120408"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(windows_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class WindowClassifier(nn.Module): \n",
    "    def __init__(self, vocab_size, embedding_size, window_size, hidden_size, output_size):\n",
    "\n",
    "        super(WindowClassifier, self).__init__()\n",
    "        \n",
    "        self.embed = nn.Embedding(vocab_size, embedding_size)\n",
    "        self.h_layer1 = nn.Linear(embedding_size * (window_size * 2 + 1), hidden_size)\n",
    "        self.h_layer2 = nn.Linear(hidden_size, hidden_size)\n",
    "        self.o_layer = nn.Linear(hidden_size, output_size)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "        self.dropout = nn.Dropout(0.3)\n",
    "        \n",
    "    def forward(self, inputs, is_training=False): \n",
    "        embeds = self.embed(inputs) # BxWxD\n",
    "        concated = embeds.view(-1, embeds.size(1)*embeds.size(2)) # Bx(W*D)\n",
    "        h0 = self.relu(self.h_layer1(concated))\n",
    "        if is_training:\n",
    "            h0 = self.dropout(h0)\n",
    "        h1 = self.relu(self.h_layer2(h0))\n",
    "        if is_training:\n",
    "            h1 = self.dropout(h1)\n",
    "        out = self.softmax(self.o_layer(h1))\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "EMBEDDING_SIZE = 50 # x (WINDOW_SIZE*2+1) = 250\n",
    "HIDDEN_SIZE = 300\n",
    "EPOCH = 3\n",
    "LEARNING_RATE = 0.001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = WindowClassifier(len(word2index), EMBEDDING_SIZE, WINDOW_SIZE, HIDDEN_SIZE, len(tag2index))\n",
    "if USE_CUDA:\n",
    "    model = model.cuda()\n",
    "loss_function = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0/3] mean_loss : 2.21\n",
      "[0/3] mean_loss : 0.48\n",
      "[0/3] mean_loss : 0.37\n",
      "[0/3] mean_loss : 0.32\n",
      "[0/3] mean_loss : 0.28\n",
      "[1/3] mean_loss : 0.35\n",
      "[1/3] mean_loss : 0.22\n",
      "[1/3] mean_loss : 0.21\n",
      "[1/3] mean_loss : 0.20\n",
      "[1/3] mean_loss : 0.19\n",
      "[2/3] mean_loss : 0.21\n",
      "[2/3] mean_loss : 0.15\n",
      "[2/3] mean_loss : 0.15\n",
      "[2/3] mean_loss : 0.14\n",
      "[2/3] mean_loss : 0.14\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(EPOCH):\n",
    "    losses = []\n",
    "    for i,batch in enumerate(getBatch(BATCH_SIZE, windows_train)):\n",
    "        x, y = list(zip(*batch))\n",
    "        inputs = torch.cat([prepare_sequence(sent, word2index).view(1, -1) for sent in x])\n",
    "        targets = torch.cat([prepare_tag(tag, tag2index) for tag in y])\n",
    "        model.zero_grad()\n",
    "        preds = model(inputs, is_training=True)\n",
    "        loss = loss_function(preds, targets)\n",
    "        losses.append(loss.data.tolist()[0])\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        if i % 1000 == 0:\n",
    "            print(\"[%d/%d] mean_loss : %0.2f\" %(epoch, EPOCH, np.mean(losses)))\n",
    "            losses = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "window_clf_test_results = []\n",
    "\n",
    "for test in windows_test:\n",
    "    x, y = test[0], test[1]\n",
    "    input_ = prepare_sequence(x, word2index).view(1, -1)\n",
    "\n",
    "    i = model(input_).max(1)[1]\n",
    "    pred = index2tag[i.data.tolist()[0]]\n",
    "    window_clf_test_results.append([pred, y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation of pytorch model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.6971263179656813\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.712     0.654     0.681      1833\n",
      "      I-PER      0.816     0.721     0.766      1441\n",
      "      B-ORG      0.722     0.588     0.648      2282\n",
      "      I-ORG      0.798     0.600     0.685      1655\n",
      "      B-LOC      0.675     0.598     0.634      1858\n",
      "      I-LOC      0.568     0.412     0.478       374\n",
      "     B-MISC      0.581     0.410     0.481      1526\n",
      "     I-MISC      0.452     0.291     0.354       967\n",
      "\n",
      "avg / total      0.690     0.565     0.620     11936\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "def accuracy(y_pred, y, exclude):\n",
    "    \"\"\"computes average accuracy score, excluding categories in exclude\"\"\"\n",
    "    correct = [l == r for (l, r) in zip(y_pred, y) if r not in exclude]\n",
    "    return sum(correct) / len(correct)\n",
    "\n",
    "LABELS = [label1+label2 for label2 in [\"PER\", \"ORG\", \"LOC\", \"MISC\"] for label1 in [\"B-\", \"I-\"]]\n",
    "LABELS.append(\"O\")\n",
    "\n",
    "y_pred_pytorch, y_test = list(zip(*window_clf_test_results))\n",
    "print(\"Accuracy\", accuracy(y_test, y_pred_pytorch, [\"O\"]))\n",
    "print(metrics.classification_report(y_test, y_pred_pytorch, digits=3, labels=LABELS[:-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will test our model on the universal dataset (Spanish and Dutch mixed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification started with c= 1\n",
      "Classification ended with c= 1\n"
     ]
    }
   ],
   "source": [
    "x_train_uni, y_train_uni = create_dataset(train_uni, features2) \n",
    "x_test_uni, y_test_uni = create_dataset(test_uni, features2) \n",
    "\n",
    "#Inverse of regularization strength. Large values give more freedom to the model, smaller values constrain the model.\n",
    "C = [1]\n",
    "\n",
    "predicts_uni = compare_models(x_train_uni, y_train_uni, x_test_uni, y_test_uni, C)\n",
    "\n",
    "clf_result = predicts_uni[1]\n",
    "clf = clf_result.get_clf()\n",
    "y_pred_lr = clf_result.get_y_pred()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.7803622159090909\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "      B-PER      0.804     0.870     0.836      1833\n",
      "      I-PER      0.826     0.922     0.871      1441\n",
      "      B-ORG      0.784     0.796     0.790      2282\n",
      "      I-ORG      0.739     0.636     0.684      1655\n",
      "      B-LOC      0.794     0.770     0.782      1858\n",
      "      I-LOC      0.704     0.457     0.554       374\n",
      "     B-MISC      0.770     0.691     0.728      1526\n",
      "     I-MISC      0.658     0.353     0.459       967\n",
      "\n",
      "avg / total      0.773     0.736     0.749     11936\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy\", accuracy(y_test_uni, y_pred_lr, [\"O\"]))\n",
    "print(metrics.classification_report(y_test_uni, y_pred_lr, digits=3, labels=LABELS[:-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Looks like the average f1 score of our model is higher than the pytorch model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'O'"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_pytorch[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test_uni_words = [word for sent in test_uni for word in sent]\n",
    "pytorch_right_lr_wrong_words = []\n",
    "for i in range(len(x_test_uni_words)):\n",
    "    y = y_test_uni[i]\n",
    "    y_pytorch = y_pred_pytorch[i]\n",
    "    y_lr = y_pred_lr[i]\n",
    "    if y != 'O' and y_lr != 'O' and y_pytorch != 'O':\n",
    "        if y_pytorch == y and y_lr != y:\n",
    "            pytorch_right_lr_wrong_words.append((x_test_uni_words[i], y_pytorch, y_lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of samples with a tag different than 'O' in which pytorch predicts the right tag and our model predicts the wrong tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "451"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pytorch_right_lr_wrong_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(('Arévalo', 'NC', 'B-LOC'), 'B-LOC', 'B-PER')"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pytorch_right_lr_wrong_words[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of tags in which the difference in the tags is because of the bounderies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wrong_bounds = [x for x in pytorch_right_lr_wrong_words if x[1][0] != x[2][0]]\n",
    "len(wrong_bounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(('Arévalo', 'NC', 'B-LOC'), 'B-LOC', 'I-ORG')"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wrong_bounds[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lr_right_pytorch_wrong_words = []\n",
    "for i in range(len(x_test_uni_words)):\n",
    "    y = y_test_uni[i]\n",
    "    y_pytorch = y_pred_pytorch[i]\n",
    "    y_lr = y_pred_lr[i]\n",
    "    if y != 'O' and y_lr != 'O' and y_pytorch != 'O':\n",
    "        if y_pytorch != y and y_lr == y:\n",
    "            lr_right_pytorch_wrong_words.append((x_test_uni_words[i], y_pytorch, y_lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of samples with a tag different than 'O' in which our model predicts the right tag and pytorch predicts the wrong tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "899"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(lr_right_pytorch_wrong_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(('Victorino', 'NC', 'I-MISC'), 'I-ORG', 'I-MISC')"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_right_pytorch_wrong_words[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "185"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wrong_bounds2 = [x for x in lr_right_pytorch_wrong_words if x[1][0] != x[2][0]]\n",
    "len(wrong_bounds2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(('Gabriel', 'VMN', 'B-PER'), 'I-LOC', 'B-PER')"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wrong_bounds2[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Pytorch</th>\n",
       "      <th>B-PER</th>\n",
       "      <th>I-PER</th>\n",
       "      <th>B-ORG</th>\n",
       "      <th>I-ORG</th>\n",
       "      <th>B-LOC</th>\n",
       "      <th>I-LOC</th>\n",
       "      <th>B-MISC</th>\n",
       "      <th>I-MISC</th>\n",
       "      <th>O</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Our Model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>B-PER</th>\n",
       "      <td>1222</td>\n",
       "      <td>30</td>\n",
       "      <td>51</td>\n",
       "      <td>9</td>\n",
       "      <td>76</td>\n",
       "      <td>3</td>\n",
       "      <td>36</td>\n",
       "      <td>14</td>\n",
       "      <td>543</td>\n",
       "      <td>1984</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I-PER</th>\n",
       "      <td>42</td>\n",
       "      <td>1078</td>\n",
       "      <td>9</td>\n",
       "      <td>34</td>\n",
       "      <td>9</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>43</td>\n",
       "      <td>379</td>\n",
       "      <td>1609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B-ORG</th>\n",
       "      <td>82</td>\n",
       "      <td>7</td>\n",
       "      <td>1424</td>\n",
       "      <td>28</td>\n",
       "      <td>169</td>\n",
       "      <td>3</td>\n",
       "      <td>98</td>\n",
       "      <td>16</td>\n",
       "      <td>491</td>\n",
       "      <td>2318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I-ORG</th>\n",
       "      <td>8</td>\n",
       "      <td>36</td>\n",
       "      <td>40</td>\n",
       "      <td>921</td>\n",
       "      <td>46</td>\n",
       "      <td>41</td>\n",
       "      <td>20</td>\n",
       "      <td>86</td>\n",
       "      <td>226</td>\n",
       "      <td>1424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B-LOC</th>\n",
       "      <td>59</td>\n",
       "      <td>6</td>\n",
       "      <td>102</td>\n",
       "      <td>60</td>\n",
       "      <td>1106</td>\n",
       "      <td>28</td>\n",
       "      <td>25</td>\n",
       "      <td>24</td>\n",
       "      <td>390</td>\n",
       "      <td>1800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I-LOC</th>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>26</td>\n",
       "      <td>10</td>\n",
       "      <td>137</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>35</td>\n",
       "      <td>243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B-MISC</th>\n",
       "      <td>34</td>\n",
       "      <td>10</td>\n",
       "      <td>67</td>\n",
       "      <td>29</td>\n",
       "      <td>45</td>\n",
       "      <td>4</td>\n",
       "      <td>623</td>\n",
       "      <td>33</td>\n",
       "      <td>523</td>\n",
       "      <td>1368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I-MISC</th>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>4</td>\n",
       "      <td>40</td>\n",
       "      <td>8</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>251</td>\n",
       "      <td>162</td>\n",
       "      <td>518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>O</th>\n",
       "      <td>230</td>\n",
       "      <td>77</td>\n",
       "      <td>156</td>\n",
       "      <td>97</td>\n",
       "      <td>179</td>\n",
       "      <td>29</td>\n",
       "      <td>254</td>\n",
       "      <td>137</td>\n",
       "      <td>107985</td>\n",
       "      <td>109144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>1683</td>\n",
       "      <td>1273</td>\n",
       "      <td>1857</td>\n",
       "      <td>1244</td>\n",
       "      <td>1648</td>\n",
       "      <td>271</td>\n",
       "      <td>1077</td>\n",
       "      <td>621</td>\n",
       "      <td>110734</td>\n",
       "      <td>120408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Pytorch    B-PER  I-PER  B-ORG  I-ORG  B-LOC  I-LOC  B-MISC  I-MISC       O  \\\n",
       "Our Model                                                                     \n",
       "B-PER       1222     30     51      9     76      3      36      14     543   \n",
       "I-PER         42   1078      9     34      9     14       1      43     379   \n",
       "B-ORG         82      7   1424     28    169      3      98      16     491   \n",
       "I-ORG          8     36     40    921     46     41      20      86     226   \n",
       "B-LOC         59      6    102     60   1106     28      25      24     390   \n",
       "I-LOC          2     12      4     26     10    137       0      17      35   \n",
       "B-MISC        34     10     67     29     45      4     623      33     523   \n",
       "I-MISC         4     17      4     40      8     12      20     251     162   \n",
       "O            230     77    156     97    179     29     254     137  107985   \n",
       "All         1683   1273   1857   1244   1648    271    1077     621  110734   \n",
       "\n",
       "Pytorch       All  \n",
       "Our Model          \n",
       "B-PER        1984  \n",
       "I-PER        1609  \n",
       "B-ORG        2318  \n",
       "I-ORG        1424  \n",
       "B-LOC        1800  \n",
       "I-LOC         243  \n",
       "B-MISC       1368  \n",
       "I-MISC        518  \n",
       "O          109144  \n",
       "All        120408  "
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "y_pytorch_s = pd.Categorical(y_pred_pytorch, categories=LABELS)\n",
    "y_lr_s = pd.Categorical(y_pred_lr, categories=LABELS)\n",
    "pd.crosstab(y_lr_s, y_pytorch_s, rownames=['Our Model'], colnames=['Pytorch'], margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Pytorch</th>\n",
       "      <th>LOC</th>\n",
       "      <th>ORG</th>\n",
       "      <th>PER</th>\n",
       "      <th>MISC</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Our Model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>1281</td>\n",
       "      <td>192</td>\n",
       "      <td>79</td>\n",
       "      <td>66</td>\n",
       "      <td>1618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ORG</th>\n",
       "      <td>259</td>\n",
       "      <td>2413</td>\n",
       "      <td>133</td>\n",
       "      <td>220</td>\n",
       "      <td>3025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PER</th>\n",
       "      <td>102</td>\n",
       "      <td>103</td>\n",
       "      <td>2372</td>\n",
       "      <td>94</td>\n",
       "      <td>2671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MISC</th>\n",
       "      <td>69</td>\n",
       "      <td>140</td>\n",
       "      <td>65</td>\n",
       "      <td>927</td>\n",
       "      <td>1201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>1711</td>\n",
       "      <td>2848</td>\n",
       "      <td>2649</td>\n",
       "      <td>1307</td>\n",
       "      <td>8515</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Pytorch     LOC   ORG   PER  MISC   All\n",
       "Our Model                              \n",
       "LOC        1281   192    79    66  1618\n",
       "ORG         259  2413   133   220  3025\n",
       "PER         102   103  2372    94  2671\n",
       "MISC         69   140    65   927  1201\n",
       "All        1711  2848  2649  1307  8515"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pytorch_s = pd.Categorical([y[2:] for y in y_pred_pytorch], categories=['LOC','ORG','PER','MISC'])\n",
    "y_lr_s = pd.Categorical([y[2:] for y in y_pred_lr], categories=['LOC','ORG','PER','MISC'])\n",
    "pd.crosstab(y_lr_s, y_pytorch_s, rownames=['Our Model'], colnames=['Pytorch'], margins=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Pytorch</th>\n",
       "      <th>B</th>\n",
       "      <th>I</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Our Model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>5219</td>\n",
       "      <td>304</td>\n",
       "      <td>5523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I</th>\n",
       "      <td>227</td>\n",
       "      <td>2765</td>\n",
       "      <td>2992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>5446</td>\n",
       "      <td>3069</td>\n",
       "      <td>8515</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Pytorch       B     I   All\n",
       "Our Model                  \n",
       "B          5219   304  5523\n",
       "I           227  2765  2992\n",
       "All        5446  3069  8515"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pytorch_s = pd.Categorical([y[0] for y in y_pred_pytorch], categories=['B','I'])\n",
    "y_lr_s = pd.Categorical([y[0] for y in y_pred_lr], categories=['B','I'])\n",
    "pd.crosstab(y_lr_s, y_pytorch_s, rownames=['Our Model'], colnames=['Pytorch'], margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the most disagreement between the two models is not about the bounderies but rather the entity tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def agreement(y1, y2, exclude):\n",
    "    \"\"\"computes average accuracy score, excluding categories in exclude\"\"\"\n",
    "    correct = [l == r for (l, r) in zip(y1, y2) if r not in exclude and l not in exclude]\n",
    "    return sum(correct) / len(correct)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Models agreement 79.4128009395185\n"
     ]
    }
   ],
   "source": [
    "print(\"Models agreement\", agreement(y_pred_pytorch, y_pred_lr, [\"O\"])*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two models agree on 80% of the tags, when both models tag a words with a tag different than \"O\". "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}