language-models-part2-solved.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import gutenberg\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import math\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Language modeling 2\n",
    "\n",
    "In the previous exercises we saw how to calculate a language model out of raw text. In this set of exercises you will learn how to deal with zero probabilities and how to evaluate LMs. Let's get started!\n",
    "\n",
    "In order to make everything easier and to avoid repeating code (even though it can be good to internalize doing some things), here you have a part of the previous exercise to calculate language models for Shakespeare's and Carrol's book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prod (ite):\n",
    "    if len(ite)==0:\n",
    "        return 1\n",
    "    else:\n",
    "        return ite[0]*prod(ite[1:])\n",
    "        \n",
    "alice = gutenberg.sents(\"carroll-alice.txt\")\n",
    "hamlet = gutenberg.sents(\"shakespeare-hamlet.txt\")\n",
    "\n",
    "#Create a LM for Alice's Adventures in Wonderland\n",
    "alice_wbounds = [[\"<s>\"]+sentence+[\"</s>\"] for sentence in alice]\n",
    "alice_words = [word for sentence in alice_wbounds for word in sentence]\n",
    "cfreq_alice_bigrams= nltk.ConditionalFreqDist(nltk.bigrams(alice_words))\n",
    "cprob_alice_bigrams = nltk.ConditionalProbDist(cfreq_alice_bigrams, nltk.MLEProbDist)\n",
    "\n",
    "#Create a LM for Shakespeare's Hamlet\n",
    "hamlet_wbounds = [[\"<s>\"]+sentence+[\"</s>\"] for sentence in hamlet]\n",
    "hamlet_words = [word for sentence in hamlet_wbounds for word in sentence]\n",
    "cfreq_hamlet_bigrams= nltk.ConditionalFreqDist(nltk.bigrams(hamlet_words))\n",
    "cprob_hamlet_bigrams = nltk.ConditionalProbDist(cfreq_hamlet_bigrams, nltk.MLEProbDist)\n",
    "\n",
    "def prob_bigram_lm(sent, lm):\n",
    "    sent_wb = \"<s> \" + sent + \" </s>\"\n",
    "    return prod([lm[bigram[0]].prob(bigram[1]) for bigram in nltk.bigrams(sent_wb.split())])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now learned in class how to smooth a language model in a pretty simple way. This was interesting for the cases in which some words don't appear in a corpus.\n",
    "\n",
    "For example, the conditional probability $P(w_{n}=screen | w_{n-1}=The)$ is $0.0$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cprob_alice_bigrams[\"The\"].prob('screen')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "\n",
    "Use the variable `cfreq_alice_bigrams` to calculate $P_{Laplace}(w_{n}=screen | w_{n-1}=The)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.00032"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#YOUR CODE HERE\n",
    "V_alice = len(cfreq_alice_bigrams.conditions())\n",
    "\n",
    "(cfreq_alice_bigrams['The'].get(\"screen\",0) + 1)/(alice_words.count('The')+V_alice)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "As you may have imagined, we can estimate the smoothed probabilities in NLTK. It is actually pretty simple, as we only need to change the probability distribution when calculating a conditional probaility distribution.\n",
    "\n",
    "Please, check the documentation of NLTK, mainly the one about probability, and check how you can estimate the smoothed probability distributions. Once you know how to do it, create a variable with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#YOUR CODE HERE\n",
    "cprob_alice_bigrams_smooth = nltk.ConditionalProbDist(cfreq_alice_bigrams, nltk.LaplaceProbDist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you create the smoothed language model, you can use it as if it was a regular language model. Then, calculate again $P_{Laplace}(w_{n}=screen | w_{n-1}=The)$ using the new language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.005988023952095809"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#YOUR CODE HERE\n",
    "cprob_alice_bigrams_smooth[\"The\"].prob('screen')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you get the same result? Why? Think about it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3\n",
    "\n",
    "Now, following the instructions from the previous class, please calculate three smoothed language models, based on bigrams, trigrams and quadrigrams, from Carrol's book.\n",
    "\n",
    "#### Hint: This doesn't work:\n",
    "\n",
    "`cfreq_corpus1_trigrams= nltk.ConditionalFreqDist(nltk.trigrams(words))`\n",
    "\n",
    "#### Hint: Recall the word boundaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#YOUR CODE HERE\n",
    "alice_wbounds = [[\"<s>\"]+sentence+[\"</s>\"] for sentence in alice]\n",
    "alice_wbounds_2 = [[\"<s>\",\"<s>\"]+sentence+[\"</s>\",\"</s>\"] for sentence in alice]\n",
    "alice_wbounds_3 = [[\"<s>\",\"<s>\",\"<s>\"]+sentence+[\"</s>\",\"</s>\",\"</s>\"] for sentence in alice]\n",
    "alice_words = [word for sentence in alice_wbounds for word in sentence]+[\"<s>\"]\n",
    "alice_words_2 = [word for sentence in alice_wbounds_2 for word in sentence]+[\"<s>\",\"<s>\"]\n",
    "alice_words_3 = [word for sentence in alice_wbounds_3 for word in sentence]+[\"<s>\",\"<s>\",\"<s>\"]\n",
    "\n",
    "cfreq_alice_trigrams = nltk.ConditionalFreqDist([(' '.join(i[:-1]), i[-1]) for i in nltk.trigrams(alice_words_2)])\n",
    "cfreq_alice_quadrigrams = nltk.ConditionalFreqDist([(' '.join(i[:-1]), i[-1]) for i in nltk.ngrams(alice_words_3,4)])\n",
    "\n",
    "#If we don't change the bins, then, if we look for the conditional probabilitiy distribution of a word\n",
    "#that doesn't appear in the training corpus, it's going to raise an exception\n",
    "cprob_alice_bigrams_smooth = nltk.ConditionalProbDist(cfreq_alice_bigrams, nltk.LaplaceProbDist, bins=V_alice)\n",
    "cprob_alice_trigrams_smooth = nltk.ConditionalProbDist(cfreq_alice_trigrams, nltk.LaplaceProbDist, bins=(V_alice**2))\n",
    "cprob_alice_quadrigrams_smooth = nltk.ConditionalProbDist(cfreq_alice_quadrigrams, nltk.LaplaceProbDist, bins=(V_alice**3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4\n",
    "\n",
    "To conclude, calculate the perplexity of the three language models using this string as a test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The cat was brown. the brownie was really good'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_string = 'The cat was brown. the brownie was really good'\n",
    "test_string2 = '''People were going home'''\n",
    "test_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def addbounds (list_sentences, n):\n",
    "    list_sent_wb = [''.join([\"<s> \"]*(n-1)) + sent + ''.join([\" </s>\"]*(n-1)) for sent in list_sentences]\n",
    "    return ' '.join(list_sent_wb) + \" \" + ''.join([\" <s>\"]*(n-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<s> the cat was brown </s> <s> the brownie was really good </s>  <s>'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_splited = [sentence.lower().strip() for sentence in test_string.replace(\"\\n\", \" \").split(\".\")]\n",
    "sentence_splited2 = [sentence.lower().strip() for sentence in test_string2.replace(\"\\n\", \" \").split(\".\")]\n",
    "addbounds(sentence_splited,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.261502301937878e-37"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def prob_ngram_lm(sent, lm, n):\n",
    "    ngrams = [(' '.join(i[:-1]), i[-1]) for i in nltk.ngrams(sent.split(),n)]\n",
    "    return prod([lm[ngram[0]].prob(ngram[1]) for ngram in ngrams])\n",
    "\n",
    "prob_ngram_lm(addbounds(sentence_splited,2), cprob_alice_bigrams_smooth, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perplexity (sentence, lm, n):\n",
    "    N = len(sentence.split()) + (n-1)*2\n",
    "    print (N, sentence)\n",
    "    return (prob_ngram_lm(sentence,lm, n))**(-1/N)\n",
    "\n",
    "def prod (ite):\n",
    "    res = 0.0\n",
    "    for el in ite:\n",
    "        res=res + np.log(el)\n",
    "    return np.exp(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16 <s> the cat was brown </s> <s> the brownie was really good </s>  <s>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "195.14174551177027"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perplexity(addbounds(sentence_splited,2), cprob_alice_bigrams_smooth, 2)\n",
    "#perplexity(addbounds(sentence_splited2,2), cprob_alice_bigrams_smooth, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23 <s> <s> the cat was brown </s> </s> <s> <s> the brownie was really good </s> </s>  <s> <s>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "28955.31127061816"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perplexity(addbounds(sentence_splited,3), cprob_alice_trigrams_smooth, 3)\n",
    "#perplexity(addbounds(sentence_splited2,3), cprob_alice_trigrams_smooth, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30 <s> <s> <s> the cat was brown </s> </s> </s> <s> <s> <s> the brownie was really good </s> </s> </s>  <s> <s> <s>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3705101.9084645724"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perplexity(addbounds(sentence_splited,4), cprob_alice_quadrigrams_smooth, 4)\n",
    "#perplexity(addbounds(sentence_splited2,4), cprob_alice_quadrigrams_smooth, 4)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}