Part 0.0: Deep_learning_for_NLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Deep_learning_for_NLP.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1E2dJmhOSkvs-3OJ0jnUZ7DthGfG8L1wi",
      "authorship_tag": "ABX9TyPCscwZ+hhMphNJ09Dt3LPP"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y5zkptMKXHre"
      },
      "source": [
        "#Chapter 5 How to Clean Text Manually and with NLTK\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7m8ETvE7YlBJ"
      },
      "source": [
        "##Manual Tokenization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q8ABLrLOYw-G"
      },
      "source": [
        "###Load Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rZ3E-7jYjViq"
      },
      "source": [
        "# Load data\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/metamorphosis_clean.txt'\n",
        "file = open(filename, 'rt')\n",
        "text = file.read()\n",
        "file.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TFkvwq-HZ_iE"
      },
      "source": [
        "###Split by Whitespace\n",
        "\n",
        "Running the example splits the document into a long list of words and prints the first 100 for\n",
        "us to review. We can see that punctuation is preserved (e.g. wasn’t and armour-like), which is\n",
        "nice. We can also see that end of sentence punctuation is kept with the last word (e.g. thought.),\n",
        "which is not great.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h9Bpg4CAZxxC",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "c3c8af28-9305-4111-f42b-60ff7126ef84"
      },
      "source": [
        "# Split into words by white space\n",
        "words = text.split()\n",
        "print(words[:100])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['\\ufeffOne', 'morning,', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams,', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin.', 'He', 'lay', 'on', 'his', 'armour-like', 'back,', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly,', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections.', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment.', 'His', 'many', 'legs,', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him,', 'waved', 'about', 'helplessly', 'as', 'he', 'looked.', '\"What\\'s', 'happened', 'to', 'me?\"', 'he', 'thought.', 'It', \"wasn't\", 'a', 'dream.', 'His', 'room,', 'a', 'proper', 'human']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SxeEI7t-eiUB"
      },
      "source": [
        "###Select Words\n",
        "\n",
        "Again, running the example we can see that we get our list of words. This time, we can see\n",
        "that armour-like is now two words armour and like (fine) but contractions like What’s is also\n",
        "two words What and s (not great)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C0sThzXZaWud",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "0a608d94-60fb-4b53-e9bd-e3c962db7429"
      },
      "source": [
        "import re\n",
        "# split based on the word only\n",
        "words = re.split(r'\\W+', text)\n",
        "words[:100]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['',\n",
              " 'One',\n",
              " 'morning',\n",
              " 'when',\n",
              " 'Gregor',\n",
              " 'Samsa',\n",
              " 'woke',\n",
              " 'from',\n",
              " 'troubled',\n",
              " 'dreams',\n",
              " 'he',\n",
              " 'found',\n",
              " 'himself',\n",
              " 'transformed',\n",
              " 'in',\n",
              " 'his',\n",
              " 'bed',\n",
              " 'into',\n",
              " 'a',\n",
              " 'horrible',\n",
              " 'vermin',\n",
              " 'He',\n",
              " 'lay',\n",
              " 'on',\n",
              " 'his',\n",
              " 'armour',\n",
              " 'like',\n",
              " 'back',\n",
              " 'and',\n",
              " 'if',\n",
              " 'he',\n",
              " 'lifted',\n",
              " 'his',\n",
              " 'head',\n",
              " 'a',\n",
              " 'little',\n",
              " 'he',\n",
              " 'could',\n",
              " 'see',\n",
              " 'his',\n",
              " 'brown',\n",
              " 'belly',\n",
              " 'slightly',\n",
              " 'domed',\n",
              " 'and',\n",
              " 'divided',\n",
              " 'by',\n",
              " 'arches',\n",
              " 'into',\n",
              " 'stiff',\n",
              " 'sections',\n",
              " 'The',\n",
              " 'bedding',\n",
              " 'was',\n",
              " 'hardly',\n",
              " 'able',\n",
              " 'to',\n",
              " 'cover',\n",
              " 'it',\n",
              " 'and',\n",
              " 'seemed',\n",
              " 'ready',\n",
              " 'to',\n",
              " 'slide',\n",
              " 'off',\n",
              " 'any',\n",
              " 'moment',\n",
              " 'His',\n",
              " 'many',\n",
              " 'legs',\n",
              " 'pitifully',\n",
              " 'thin',\n",
              " 'compared',\n",
              " 'with',\n",
              " 'the',\n",
              " 'size',\n",
              " 'of',\n",
              " 'the',\n",
              " 'rest',\n",
              " 'of',\n",
              " 'him',\n",
              " 'waved',\n",
              " 'about',\n",
              " 'helplessly',\n",
              " 'as',\n",
              " 'he',\n",
              " 'looked',\n",
              " 'What',\n",
              " 's',\n",
              " 'happened',\n",
              " 'to',\n",
              " 'me',\n",
              " 'he',\n",
              " 'thought',\n",
              " 'It',\n",
              " 'wasn',\n",
              " 't',\n",
              " 'a',\n",
              " 'dream',\n",
              " 'His']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FOcySoFlf-5s"
      },
      "source": [
        "###Split by Whitespace and Remove Punctuation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CgjVSrcThx4P",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "514821e7-bd55-44b0-8ab1-43d4fe5ce43e"
      },
      "source": [
        "import string\n",
        "print(string.punctuation)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vmYxE38siz3L"
      },
      "source": [
        "We can use regular expressions to select for the punctuation characters and use the sub()\n",
        "function to replace them with nothing. For example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ojTK8J7Ce3Aw"
      },
      "source": [
        "re_punc = re.compile('[%s]' %re.escape(string.punctuation))\n",
        "# remove punctuation from each word\n",
        "stripped = [re_punc.sub('',w) for w in words]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r_Pq0mwIgZjG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "7ff2c9c4-9720-411c-8c10-a5f6974f64a9"
      },
      "source": [
        "stripped[:100]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['',\n",
              " 'One',\n",
              " 'morning',\n",
              " 'when',\n",
              " 'Gregor',\n",
              " 'Samsa',\n",
              " 'woke',\n",
              " 'from',\n",
              " 'troubled',\n",
              " 'dreams',\n",
              " 'he',\n",
              " 'found',\n",
              " 'himself',\n",
              " 'transformed',\n",
              " 'in',\n",
              " 'his',\n",
              " 'bed',\n",
              " 'into',\n",
              " 'a',\n",
              " 'horrible',\n",
              " 'vermin',\n",
              " 'He',\n",
              " 'lay',\n",
              " 'on',\n",
              " 'his',\n",
              " 'armour',\n",
              " 'like',\n",
              " 'back',\n",
              " 'and',\n",
              " 'if',\n",
              " 'he',\n",
              " 'lifted',\n",
              " 'his',\n",
              " 'head',\n",
              " 'a',\n",
              " 'little',\n",
              " 'he',\n",
              " 'could',\n",
              " 'see',\n",
              " 'his',\n",
              " 'brown',\n",
              " 'belly',\n",
              " 'slightly',\n",
              " 'domed',\n",
              " 'and',\n",
              " 'divided',\n",
              " 'by',\n",
              " 'arches',\n",
              " 'into',\n",
              " 'stiff',\n",
              " 'sections',\n",
              " 'The',\n",
              " 'bedding',\n",
              " 'was',\n",
              " 'hardly',\n",
              " 'able',\n",
              " 'to',\n",
              " 'cover',\n",
              " 'it',\n",
              " 'and',\n",
              " 'seemed',\n",
              " 'ready',\n",
              " 'to',\n",
              " 'slide',\n",
              " 'off',\n",
              " 'any',\n",
              " 'moment',\n",
              " 'His',\n",
              " 'many',\n",
              " 'legs',\n",
              " 'pitifully',\n",
              " 'thin',\n",
              " 'compared',\n",
              " 'with',\n",
              " 'the',\n",
              " 'size',\n",
              " 'of',\n",
              " 'the',\n",
              " 'rest',\n",
              " 'of',\n",
              " 'him',\n",
              " 'waved',\n",
              " 'about',\n",
              " 'helplessly',\n",
              " 'as',\n",
              " 'he',\n",
              " 'looked',\n",
              " 'What',\n",
              " 's',\n",
              " 'happened',\n",
              " 'to',\n",
              " 'me',\n",
              " 'he',\n",
              " 'thought',\n",
              " 'It',\n",
              " 'wasn',\n",
              " 't',\n",
              " 'a',\n",
              " 'dream',\n",
              " 'His']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IvJ80BtQjB1A"
      },
      "source": [
        "Sometimes text data may contain non-printable characters. We can use a similar approach to\n",
        "filter out all non-printable characters by selecting the inverse of the string.printable constant."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1d_PdpTlinTq",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "dd226aa7-969f-4c37-d318-15d5b969ea03"
      },
      "source": [
        "re_print = re.compile('[^%s]' %re.escape(string.printable))\n",
        "result = [re_print.sub('', w) for w in words]\n",
        "result[:100]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['',\n",
              " 'One',\n",
              " 'morning',\n",
              " 'when',\n",
              " 'Gregor',\n",
              " 'Samsa',\n",
              " 'woke',\n",
              " 'from',\n",
              " 'troubled',\n",
              " 'dreams',\n",
              " 'he',\n",
              " 'found',\n",
              " 'himself',\n",
              " 'transformed',\n",
              " 'in',\n",
              " 'his',\n",
              " 'bed',\n",
              " 'into',\n",
              " 'a',\n",
              " 'horrible',\n",
              " 'vermin',\n",
              " 'He',\n",
              " 'lay',\n",
              " 'on',\n",
              " 'his',\n",
              " 'armour',\n",
              " 'like',\n",
              " 'back',\n",
              " 'and',\n",
              " 'if',\n",
              " 'he',\n",
              " 'lifted',\n",
              " 'his',\n",
              " 'head',\n",
              " 'a',\n",
              " 'little',\n",
              " 'he',\n",
              " 'could',\n",
              " 'see',\n",
              " 'his',\n",
              " 'brown',\n",
              " 'belly',\n",
              " 'slightly',\n",
              " 'domed',\n",
              " 'and',\n",
              " 'divided',\n",
              " 'by',\n",
              " 'arches',\n",
              " 'into',\n",
              " 'stiff',\n",
              " 'sections',\n",
              " 'The',\n",
              " 'bedding',\n",
              " 'was',\n",
              " 'hardly',\n",
              " 'able',\n",
              " 'to',\n",
              " 'cover',\n",
              " 'it',\n",
              " 'and',\n",
              " 'seemed',\n",
              " 'ready',\n",
              " 'to',\n",
              " 'slide',\n",
              " 'off',\n",
              " 'any',\n",
              " 'moment',\n",
              " 'His',\n",
              " 'many',\n",
              " 'legs',\n",
              " 'pitifully',\n",
              " 'thin',\n",
              " 'compared',\n",
              " 'with',\n",
              " 'the',\n",
              " 'size',\n",
              " 'of',\n",
              " 'the',\n",
              " 'rest',\n",
              " 'of',\n",
              " 'him',\n",
              " 'waved',\n",
              " 'about',\n",
              " 'helplessly',\n",
              " 'as',\n",
              " 'he',\n",
              " 'looked',\n",
              " 'What',\n",
              " 's',\n",
              " 'happened',\n",
              " 'to',\n",
              " 'me',\n",
              " 'he',\n",
              " 'thought',\n",
              " 'It',\n",
              " 'wasn',\n",
              " 't',\n",
              " 'a',\n",
              " 'dream',\n",
              " 'His']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HwuGWixMjozJ"
      },
      "source": [
        "###Normalizing case\n",
        "\n",
        "It is common to convert all words to one case. This means that the vocabulary will shrink in\n",
        "size, but some distinctions are lost (e.g. Apple the company vs apple the fruit is a commonly\n",
        "used example). We can convert all words to lowercase by calling the lower() function on each\n",
        "word."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SGhXwB-pjk9W",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "45541082-9a84-4052-b6d9-119bbf622527"
      },
      "source": [
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/metamorphosis_clean.txt'\n",
        "file = open(filename, 'rt')\n",
        "text = file.read()\n",
        "file.close()\n",
        "# split into words by white space\n",
        "words = text.split()\n",
        "# convert to lower case\n",
        "words = [word.lower() for word in words]\n",
        "print(words[:100])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['\\ufeffone', 'morning,', 'when', 'gregor', 'samsa', 'woke', 'from', 'troubled', 'dreams,', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin.', 'he', 'lay', 'on', 'his', 'armour-like', 'back,', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly,', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections.', 'the', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment.', 'his', 'many', 'legs,', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him,', 'waved', 'about', 'helplessly', 'as', 'he', 'looked.', '\"what\\'s', 'happened', 'to', 'me?\"', 'he', 'thought.', 'it', \"wasn't\", 'a', 'dream.', 'his', 'room,', 'a', 'proper', 'human']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kY5_xUTYq69k"
      },
      "source": [
        "##Tokenization and Cleaning with NLTK"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z7FjR1wNrCOZ"
      },
      "source": [
        "###Install NLTK"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G8E7RwflqrMH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 411
        },
        "outputId": "10e8f88d-ed8a-4699-fd52-c148347c710a"
      },
      "source": [
        "!pip install -U nltk"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting nltk\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/92/75/ce35194d8e3022203cca0d2f896dbb88689f9b3fce8e9f9cff942913519d/nltk-3.5.zip (1.4MB)\n",
            "\r\u001b[K     |▎                               | 10kB 12.9MB/s eta 0:00:01\r\u001b[K     |▌                               | 20kB 3.0MB/s eta 0:00:01\r\u001b[K     |▊                               | 30kB 3.9MB/s eta 0:00:01\r\u001b[K     |█                               | 40kB 4.2MB/s eta 0:00:01\r\u001b[K     |█▏                              | 51kB 3.4MB/s eta 0:00:01\r\u001b[K     |█▍                              | 61kB 3.8MB/s eta 0:00:01\r\u001b[K     |█▋                              | 71kB 4.0MB/s eta 0:00:01\r\u001b[K     |█▉                              | 81kB 4.5MB/s eta 0:00:01\r\u001b[K     |██                              | 92kB 4.7MB/s eta 0:00:01\r\u001b[K     |██▎                             | 102kB 4.6MB/s eta 0:00:01\r\u001b[K     |██▌                             | 112kB 4.6MB/s eta 0:00:01\r\u001b[K     |██▊                             | 122kB 4.6MB/s eta 0:00:01\r\u001b[K     |███                             | 133kB 4.6MB/s eta 0:00:01\r\u001b[K     |███▏                            | 143kB 4.6MB/s eta 0:00:01\r\u001b[K     |███▍                            | 153kB 4.6MB/s eta 0:00:01\r\u001b[K     |███▋                            | 163kB 4.6MB/s eta 0:00:01\r\u001b[K     |███▉                            | 174kB 4.6MB/s eta 0:00:01\r\u001b[K     |████▏                           | 184kB 4.6MB/s eta 0:00:01\r\u001b[K     |████▍                           | 194kB 4.6MB/s eta 0:00:01\r\u001b[K     |████▋                           | 204kB 4.6MB/s eta 0:00:01\r\u001b[K     |████▉                           | 215kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████                           | 225kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████▎                          | 235kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████▌                          | 245kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████▊                          | 256kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████                          | 266kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████▏                         | 276kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████▍                         | 286kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████▋                         | 296kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████▉                         | 307kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████                         | 317kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████▎                        | 327kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████▌                        | 337kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████▊                        | 348kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████                        | 358kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████▎                       | 368kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████▌                       | 378kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████▊                       | 389kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████                       | 399kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████▏                      | 409kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████▍                      | 419kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████▋                      | 430kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████▉                      | 440kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████                      | 450kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████▎                     | 460kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████▌                     | 471kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████▊                     | 481kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████                     | 491kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████▏                    | 501kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████▍                    | 512kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████▋                    | 522kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████▉                    | 532kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 542kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████▍                   | 552kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████▋                   | 563kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████▉                   | 573kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████                   | 583kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████▎                  | 593kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████▌                  | 604kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████▊                  | 614kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████                  | 624kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████▏                 | 634kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████▍                 | 645kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████▋                 | 655kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████▉                 | 665kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████                 | 675kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████▎                | 686kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████▌                | 696kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████▊                | 706kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████                | 716kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████▎               | 727kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████▌               | 737kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████▊               | 747kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████               | 757kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████▏              | 768kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████▍              | 778kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████▋              | 788kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████▉              | 798kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████              | 808kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████▎             | 819kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████▌             | 829kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████▊             | 839kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████             | 849kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████▏            | 860kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████▍            | 870kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████▋            | 880kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████▉            | 890kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████▏           | 901kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████▍           | 911kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████▋           | 921kB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████▉           | 931kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████           | 942kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████▎          | 952kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████▌          | 962kB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████▊          | 972kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████          | 983kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████▏         | 993kB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████▍         | 1.0MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████▋         | 1.0MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████▉         | 1.0MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████         | 1.0MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████▎        | 1.0MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████▌        | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████▊        | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████        | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████▌       | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████▊       | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████████       | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▏      | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▍      | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▋      | 1.1MB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▉      | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████      | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▎     | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▌     | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▊     | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████████     | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▏    | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▍    | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▋    | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▉    | 1.2MB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▏   | 1.3MB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▍   | 1.3MB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▋   | 1.3MB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▉   | 1.3MB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████   | 1.3MB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▎  | 1.3MB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▌  | 1.3MB 4.6MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▊  | 1.3MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 1.3MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▏ | 1.4MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▍ | 1.4MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▋ | 1.4MB 4.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▉ | 1.4MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████ | 1.4MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▎| 1.4MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▌| 1.4MB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▊| 1.4MB 4.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 1.4MB 4.6MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: click in /usr/local/lib/python3.6/dist-packages (from nltk) (7.1.2)\n",
            "Requirement already satisfied, skipping upgrade: joblib in /usr/local/lib/python3.6/dist-packages (from nltk) (0.16.0)\n",
            "Requirement already satisfied, skipping upgrade: regex in /usr/local/lib/python3.6/dist-packages (from nltk) (2019.12.20)\n",
            "Requirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.6/dist-packages (from nltk) (4.41.1)\n",
            "Building wheels for collected packages: nltk\n",
            "  Building wheel for nltk (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for nltk: filename=nltk-3.5-cp36-none-any.whl size=1434676 sha256=e92fa0e9635f10dd11c57a0a66292ab4c71f598ca5977d7de604c195260f80d9\n",
            "  Stored in directory: /root/.cache/pip/wheels/ae/8c/3f/b1fe0ba04555b08b57ab52ab7f86023639a526d8bc8d384306\n",
            "Successfully built nltk\n",
            "Installing collected packages: nltk\n",
            "  Found existing installation: nltk 3.2.5\n",
            "    Uninstalling nltk-3.2.5:\n",
            "      Successfully uninstalled nltk-3.2.5\n",
            "Successfully installed nltk-3.5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "nltk"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O-ahO_Y7rJ_5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a2f94253-f5a0-4b7f-e6a6-b1ed020e2734"
      },
      "source": [
        "import nltk\n",
        "nltk.download()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "NLTK Downloader\n",
            "---------------------------------------------------------------------------\n",
            "    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit\n",
            "---------------------------------------------------------------------------\n",
            "Downloader> d\n",
            "\n",
            "Download which package (l=list; x=cancel)?\n",
            "  Identifier> l\n",
            "Packages:\n",
            "  [ ] abc................. Australian Broadcasting Commission 2006\n",
            "  [ ] alpino.............. Alpino Dutch Treebank\n",
            "  [ ] averaged_perceptron_tagger Averaged Perceptron Tagger\n",
            "  [ ] averaged_perceptron_tagger_ru Averaged Perceptron Tagger (Russian)\n",
            "  [ ] basque_grammars..... Grammars for Basque\n",
            "  [ ] biocreative_ppi..... BioCreAtIvE (Critical Assessment of Information\n",
            "                           Extraction Systems in Biology)\n",
            "  [ ] bllip_wsj_no_aux.... BLLIP Parser: WSJ Model\n",
            "  [ ] book_grammars....... Grammars from NLTK Book\n",
            "  [ ] brown............... Brown Corpus\n",
            "  [ ] brown_tei........... Brown Corpus (TEI XML Version)\n",
            "  [ ] cess_cat............ CESS-CAT Treebank\n",
            "  [ ] cess_esp............ CESS-ESP Treebank\n",
            "  [ ] chat80.............. Chat-80 Data Files\n",
            "  [ ] city_database....... City Database\n",
            "  [ ] cmudict............. The Carnegie Mellon Pronouncing Dictionary (0.6)\n",
            "  [ ] comparative_sentences Comparative Sentence Dataset\n",
            "  [ ] comtrans............ ComTrans Corpus Sample\n",
            "  [ ] conll2000........... CONLL 2000 Chunking Corpus\n",
            "  [ ] conll2002........... CONLL 2002 Named Entity Recognition Corpus\n",
            "Hit Enter to continue: u\n",
            "  [ ] conll2007........... Dependency Treebanks from CoNLL 2007 (Catalan\n",
            "                           and Basque Subset)\n",
            "  [ ] crubadan............ Crubadan Corpus\n",
            "  [ ] dependency_treebank. Dependency Parsed Treebank\n",
            "  [ ] dolch............... Dolch Word List\n",
            "  [ ] europarl_raw........ Sample European Parliament Proceedings Parallel\n",
            "                           Corpus\n",
            "  [ ] floresta............ Portuguese Treebank\n",
            "  [ ] framenet_v15........ FrameNet 1.5\n",
            "  [ ] framenet_v17........ FrameNet 1.7\n",
            "  [ ] gazetteers.......... Gazeteer Lists\n",
            "  [ ] genesis............. Genesis Corpus\n",
            "  [ ] gutenberg........... Project Gutenberg Selections\n",
            "  [ ] ieer................ NIST IE-ER DATA SAMPLE\n",
            "  [ ] inaugural........... C-Span Inaugural Address Corpus\n",
            "  [ ] indian.............. Indian Language POS-Tagged Corpus\n",
            "  [ ] jeita............... JEITA Public Morphologically Tagged Corpus (in\n",
            "                           ChaSen format)\n",
            "  [ ] kimmo............... PC-KIMMO Data Files\n",
            "  [ ] knbc................ KNB Corpus (Annotated blog corpus)\n",
            "  [ ] large_grammars...... Large context-free and feature-based grammars\n",
            "                           for parser comparison\n",
            "Hit Enter to continue: c\n",
            "  [ ] lin_thesaurus....... Lin's Dependency Thesaurus\n",
            "  [ ] mac_morpho.......... MAC-MORPHO: Brazilian Portuguese news text with\n",
            "                           part-of-speech tags\n",
            "  [ ] machado............. Machado de Assis -- Obra Completa\n",
            "  [ ] masc_tagged......... MASC Tagged Corpus\n",
            "  [ ] maxent_ne_chunker... ACE Named Entity Chunker (Maximum entropy)\n",
            "  [ ] maxent_treebank_pos_tagger Treebank Part of Speech Tagger (Maximum entropy)\n",
            "  [ ] moses_sample........ Moses Sample Models\n",
            "  [ ] movie_reviews....... Sentiment Polarity Dataset Version 2.0\n",
            "  [ ] mte_teip5........... MULTEXT-East 1984 annotated corpus 4.0\n",
            "  [ ] mwa_ppdb............ The monolingual word aligner (Sultan et al.\n",
            "                           2015) subset of the Paraphrase Database.\n",
            "  [ ] names............... Names Corpus, Version 1.3 (1994-03-29)\n",
            "  [ ] nombank.1.0......... NomBank Corpus 1.0\n",
            "  [ ] nonbreaking_prefixes Non-Breaking Prefixes (Moses Decoder)\n",
            "  [ ] nps_chat............ NPS Chat\n",
            "  [ ] omw................. Open Multilingual Wordnet\n",
            "  [ ] opinion_lexicon..... Opinion Lexicon\n",
            "  [ ] panlex_swadesh...... PanLex Swadesh Corpora\n",
            "  [ ] paradigms........... Paradigm Corpus\n",
            "  [ ] pe08................ Cross-Framework and Cross-Domain Parser\n",
            "                           Evaluation Shared Task\n",
            "Hit Enter to continue: q\n",
            "\n",
            "Download which package (l=list; x=cancel)?\n",
            "  Identifier> x\n",
            "\n",
            "---------------------------------------------------------------------------\n",
            "    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit\n",
            "---------------------------------------------------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "izAZt2jXryaz"
      },
      "source": [
        "###Split into Sentences"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WOSaWT-Prsgu",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 731
        },
        "outputId": "8bbeaaa9-5e16-4c36-a5ef-b3ecdc6a044b"
      },
      "source": [
        "from nltk import sent_tokenize\n",
        "#load data\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/metamorphosis_clean.txt'\n",
        "file = open(filename, 'rt')\n",
        "text = file.read()\n",
        "file.close()\n",
        "#Split into tokens\n",
        "sentences = sent_tokenize(text)\n",
        "sentences[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "LookupError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-790088e28031>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m#Split into tokens\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0msentences\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msent_tokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[0msentences\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/nltk/tokenize/__init__.py\u001b[0m in \u001b[0;36msent_tokenize\u001b[0;34m(text, language)\u001b[0m\n\u001b[1;32m    104\u001b[0m     \u001b[0;34m:\u001b[0m\u001b[0mparam\u001b[0m \u001b[0mlanguage\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mPunkt\u001b[0m \u001b[0mcorpus\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m     \"\"\"\n\u001b[0;32m--> 106\u001b[0;31m     \u001b[0mtokenizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"tokenizers/punkt/{0}.pickle\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlanguage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    107\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mtokenizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    108\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/nltk/data.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(resource_url, format, cache, verbose, logic_parser, fstruct_reader, encoding)\u001b[0m\n\u001b[1;32m    750\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    751\u001b[0m     \u001b[0;31m# Load the resource.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 752\u001b[0;31m     \u001b[0mopened_resource\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_url\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    753\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    754\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mformat\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"raw\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/nltk/data.py\u001b[0m in \u001b[0;36m_open\u001b[0;34m(resource_url)\u001b[0m\n\u001b[1;32m    875\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    876\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mprotocol\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"nltk\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 877\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m\"\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    878\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"file\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    879\u001b[0m         \u001b[0;31m# urllib might not use mode='rb', so handle this one ourselves:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    583\u001b[0m     \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"*\"\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m70\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    587\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mpunkt\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('punkt')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mtokenizers/punkt/PY3/english.pickle\u001b[0m\n\n  Searched in:\n    - '/root/nltk_data'\n    - '/usr/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n    - ''\n**********************************************************************\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jxfgEGfuzzFY"
      },
      "source": [
        "##Split into Words"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eHEoR30bzm91",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 731
        },
        "outputId": "6e1af593-5c5a-4f6e-b21b-fb605a571be6"
      },
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "# load data\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/metamorphosis_clean.txt'\n",
        "file = open(filename, 'rt')\n",
        "text = file.read()\n",
        "file.close()\n",
        "# split into words\n",
        "tokens = word_tokenize(text)\n",
        "tokens[0:100]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "LookupError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-2-130a19c9459f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m# split into words\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0mtokens\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mword_tokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[0mtokens\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/nltk/tokenize/__init__.py\u001b[0m in \u001b[0;36mword_tokenize\u001b[0;34m(text, language, preserve_line)\u001b[0m\n\u001b[1;32m    127\u001b[0m     \u001b[0;34m:\u001b[0m\u001b[0mtype\u001b[0m \u001b[0mpreserve_line\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    128\u001b[0m     \"\"\"\n\u001b[0;32m--> 129\u001b[0;31m     \u001b[0msentences\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mpreserve_line\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0msent_tokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlanguage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    130\u001b[0m     return [\n\u001b[1;32m    131\u001b[0m         \u001b[0mtoken\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0msent\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msentences\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtoken\u001b[0m \u001b[0;32min\u001b[0m \u001b[0m_treebank_word_tokenizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/nltk/tokenize/__init__.py\u001b[0m in \u001b[0;36msent_tokenize\u001b[0;34m(text, language)\u001b[0m\n\u001b[1;32m    104\u001b[0m     \u001b[0;34m:\u001b[0m\u001b[0mparam\u001b[0m \u001b[0mlanguage\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mPunkt\u001b[0m \u001b[0mcorpus\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m     \"\"\"\n\u001b[0;32m--> 106\u001b[0;31m     \u001b[0mtokenizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"tokenizers/punkt/{0}.pickle\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlanguage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    107\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mtokenizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    108\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/nltk/data.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(resource_url, format, cache, verbose, logic_parser, fstruct_reader, encoding)\u001b[0m\n\u001b[1;32m    750\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    751\u001b[0m     \u001b[0;31m# Load the resource.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 752\u001b[0;31m     \u001b[0mopened_resource\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_url\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    753\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    754\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mformat\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"raw\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/nltk/data.py\u001b[0m in \u001b[0;36m_open\u001b[0;34m(resource_url)\u001b[0m\n\u001b[1;32m    875\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    876\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mprotocol\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"nltk\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 877\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m\"\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    878\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"file\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    879\u001b[0m         \u001b[0;31m# urllib might not use mode='rb', so handle this one ourselves:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    583\u001b[0m     \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"*\"\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m70\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    587\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mpunkt\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('punkt')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mtokenizers/punkt/PY3/english.pickle\u001b[0m\n\n  Searched in:\n    - '/root/nltk_data'\n    - '/usr/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n    - ''\n**********************************************************************\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nEKCtQ7U0UdC"
      },
      "source": [
        "##Filter Out Punctuations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H5dVX0ls0Ie7"
      },
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "# load data\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/metamorphosis_clean.txt'\n",
        "file = open(filename, 'rt')\n",
        "text = file.read()\n",
        "file.close()\n",
        "# split into words\n",
        "tokens = word_tokenize(text)\n",
        "# remove all tokens that are not alphabetic\n",
        "words = [word for word in tokens if word.isalpha()]\n",
        "words[0:100]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "72M5DhXNYRZb"
      },
      "source": [
        "##Filter out Stop Words (and Pipeline)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6SqtcOpl0334"
      },
      "source": [
        "from nltk.corpus import stopwords\n",
        "stop_words = stopwords.words('english')\n",
        "stop_words"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UFDcn5XPZ9Jr"
      },
      "source": [
        "Let's write whole concept under a single umbrella"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZoUtmmBhZMnl"
      },
      "source": [
        "import string\n",
        "import re\n",
        "from nltk.tokenize import word_tokenize\n",
        "from nltk.corpus import stopwords\n",
        "# load data\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/metamorphosis_clean.txt'\n",
        "file = open(filename, 'rt')\n",
        "text = file.read()\n",
        "file.close()\n",
        "# split into words\n",
        "tokens = word_tokenize(text)\n",
        "# convert to lower case\n",
        "tokens = [w.lower() for w in tokens]\n",
        "# prepare regex for char filtering\n",
        "re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "# remove punctuation from each word\n",
        "stripped = [re_punc.sub('', w) for w in tokens]\n",
        "# remove remaining tokens that are not alphabetic\n",
        "words = [word for word in stripped if word.isalpha()]\n",
        "# filter out stop words\n",
        "stop_words = set(stopwords.words('english'))\n",
        "words = [w for w in words if not w in stop_words]\n",
        "print(words[:100])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FHGGCC8PaFHq"
      },
      "source": [
        "##Stem Words"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9N3Eg1CqaJGL"
      },
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "from nltk.stem.porter import PorterStemmer\n",
        "# load data\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/metamorphosis_clean.txt'\n",
        "file = open(filename, 'rt')\n",
        "text = file.read()\n",
        "file.close()\n",
        "# split into words\n",
        "tokens = word_tokenize(text)\n",
        "# stemming of words\n",
        "porter = PorterStemmer()\n",
        "stemmed = [porter.stem(wor) for word in tokens]\n",
        "stemmed[:100]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "26FT8aQ14V1C"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yqL_eB9SNQ29"
      },
      "source": [
        "#**Chapter 6: How to Prepare Text Data with scikit-learn**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eGX6lN7IN0mh"
      },
      "source": [
        "The following topic will be covered in this chapter:\n",
        "1. CountVectorizer\n",
        "2. TfidfVectorizer\n",
        "3. HashingVectorizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cSasWe4OKDA"
      },
      "source": [
        "##Words Counts with *CountVectorizer*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jf-Q-9XG_RuA"
      },
      "source": [
        "import nltk\n",
        "nltk.download()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S5A1HkKtdcIu"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "# list of text documents\n",
        "text = [\"The quick brown fox jumped over the lazy dog.\"] \n",
        "# create the transform\n",
        "vectorizer = CountVectorizer()\n",
        "#tokenize and build vocab\n",
        "vectorizer.fit(text)\n",
        "# summarize\n",
        "print(vectorizer.vocabulary_)\n",
        "# encode document\n",
        "vector = vectorizer.transform(text)\n",
        "# summarize encoded vector\n",
        "print(vector.shape)\n",
        "print(type(vector))\n",
        "print(vector.toarray())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pOXHmvz5ELCQ"
      },
      "source": [
        "##Word Frequencies with TfidfVectorizer\n",
        "  \n",
        "*   **Term Frequency**: This summarizes how often a given word appears within a document.\n",
        "*   **Inverse Document Frequency**: This downscales words that appear a lot across document"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rf5I36AaOeWn"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "# list of text documents\n",
        "text = [\"The quick brown fox jumped over the lazy dog.\",\n",
        "        \"The dog.\",\n",
        "        \"The fox\"]\n",
        "# create the transform\n",
        "vectorizer = TfidfVectorizer()\n",
        "#tokenize and build vocab\n",
        "vectorizer.fit(text)\n",
        "# summarize\n",
        "print(vectorizer.vocabulary_)\n",
        "print(vectorizer.idf_)\n",
        "# encode document\n",
        "vector = vectorizer.transform([text[0]])\n",
        "# summarize encode vector\n",
        "print(vector.shape)\n",
        "print(vector.toarray())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kEVgNtbOHE3d"
      },
      "source": [
        "##**Hashing** with *HashingVectorizer*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rm2gKojEHPR0"
      },
      "source": [
        "from sklearn.feature_extraction.text import HashingVectorizer\n",
        "# list of text documents\n",
        "text = [\"The quick brown fox jumped over the lazy dog.\"]\n",
        "# create the transform\n",
        "vectorizer = HashingVectorizer(n_features=20)\n",
        "# encode document\n",
        "vector = vectorizer.transform(text)\n",
        "# summarize encoded vector\n",
        "print(vector.shape)\n",
        "print(vector.toarray())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZBH0GyEoHXnV"
      },
      "source": [
        "#**Chapter 7: How to Prepare Text Data With Keras**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dJZLA45dwsKQ"
      },
      "source": [
        "\n",
        "\n",
        "1.   Split words with text_to_word_sequence.\n",
        "2.   Encoding with one_hot.\n",
        "3.   Hash Encoding with hashing_trick.\n",
        "4.   Tokenizer API.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "55aH3abkxLXS"
      },
      "source": [
        "##7.2 Split Words with text_to_word_sequence\n",
        "\n",
        "\n",
        "By default, this function automatically does 3 things:\n",
        "*   Splits words by space\n",
        "*   Filters out punctuation\n",
        "*   Converts text to lowercase\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dz35uINjHWch"
      },
      "source": [
        "from keras.preprocessing.text import text_to_word_sequence\n",
        "# define the document\n",
        "text = 'The quick brown! %^&fox jumped over the lazy dog.'\n",
        "# tokenize the document\n",
        "result = text_to_word_sequence(text)\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l9F4Mwlfy5Pn"
      },
      "source": [
        "##7.3 Encoding with one_hot\n",
        "\n",
        "We can use the text to word sequence() function from the previous section to split the document into words and then use a set to represent only the unique words in the document.\n",
        "The size of this set can be used to estimate the size of the vocabulary for one document."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KmSCB2SfyTGU"
      },
      "source": [
        "from keras.preprocessing.text import text_to_word_sequence\n",
        "# define the document\n",
        "text = 'The quick brown fox jumped over the lazy dog.'\n",
        "# estimate the size of the vocabulary\n",
        "words = set(text_to_word_sequence(text))\n",
        "vocab_size = len(words)\n",
        "print(vocab_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C51yTjOt0P_q"
      },
      "source": [
        "We can put this together with the **one_hot()** function and encode the words in the document. The vocabulary size is increased by one-third to minimize collisions when hashing words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lrQW7LN51CgK"
      },
      "source": [
        "from keras.preprocessing.text import one_hot\n",
        "from keras.preprocessing.text import text_to_word_sequence\n",
        "# define the document\n",
        "text = 'The quick brown fox jumped over the lazy dog.'\n",
        "# estimate the size of the vocabulary\n",
        "words = set(text_to_word_sequence(text))\n",
        "vocab_size = len(words)\n",
        "print(vocab_size)\n",
        "# integer encode the document \n",
        "result = one_hot(text, round(vocab_size*1.3))\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iLva55Sy6g2X"
      },
      "source": [
        "##Hash Encoding with ***hashing_trick***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RE_F06R-2Rhi"
      },
      "source": [
        "from keras.preprocessing.text import hashing_trick\n",
        "from keras.preprocessing.text import text_to_word_sequence\n",
        "# define the document\n",
        "text = 'The quick brown fox jumped over the lazy fog.'\n",
        "# estimate the size of the vocabulary\n",
        "words = set(text_to_word_sequence(text))\n",
        "vocab_size = len(words)\n",
        "print(vocab_size)\n",
        "# integer encode the document\n",
        "result = hashing_trick(text, round(vocab_size*1.3), hash_function='md5')\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f5SW7dX0BcDg"
      },
      "source": [
        "##Tokenizer ***API***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2F-RKrTX9yOh"
      },
      "source": [
        "from keras.preprocessing.text import Tokenizer\n",
        "# define 5 documents\n",
        "docs = ['Well done!',\n",
        "'Good work',\n",
        "'Great effort',\n",
        "'nice work',\n",
        "'Excellent!']\n",
        "# create the tokenizer\n",
        "t = Tokenizer()\n",
        "# fit the tokenizer on the documents\n",
        "t.fit_on_texts(docs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BldVxBvKSTNn"
      },
      "source": [
        "Once fit, the Tokenizer provides 4 attributes that you can use to query what has been\n",
        "learned about your documents:\n",
        "\n",
        "*   **word counts:** A dictionary of words and their counts.\n",
        "\n",
        "*   **word docs:** A dictionary of words and how many documents each appeared in.\n",
        "\n",
        "*   **word index:** A dictionary of words and their uniquely assigned integers.\n",
        "\n",
        "*   **document count:** An integer count of the total number of documents that were used to fit the\n",
        "Tokenizer.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gnYWet75LatN"
      },
      "source": [
        "# summarize what was learned\n",
        "print(t.word_counts)\n",
        "print(t.document_count)\n",
        "print(t.word_index)\n",
        "print(t.word_docs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jtj8pjlfVLxQ"
      },
      "source": [
        "Once the **Tokenizer** has been fit on training data, it can be used to encode documents in the train or test datasets. The **texts_to_matrix()** function on the **Tokenizer** can be used to create one vector per document provided per input. The length of the vector is the total size of the vocabulary.\n",
        "Thw modes available include:\n",
        "\n",
        "\n",
        "1.   **Binary:** Whether or not each word is present in the document. This is the default.\n",
        "\n",
        "2.   **Count:** The count of each word in the document\n",
        "\n",
        "3.   **tfidf:** The Text Frequency-Inverse DocumentFrequency (TF-IDF) scoring for each word\n",
        "in the document.\n",
        "\n",
        "4.   **freq:** The frequency of each word as a ratio of words within each document\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "shCMjbgsVScb"
      },
      "source": [
        "from keras.preprocessing.text import Tokenizer\n",
        "# define 5 document\n",
        "docs = ['Well done!',\n",
        "        'Good work',\n",
        "        'Great effort',\n",
        "        'nice work',\n",
        "        'Excellent']\n",
        "# create the tokenizer\n",
        "t = Tokenizer()\n",
        "# fit the tokenizer on the documents\n",
        "t.fit_on_texts(docs)\n",
        "# summarize what was learned\n",
        "print(t.word_counts)\n",
        "print(t.document_count)\n",
        "print(t.word_index)\n",
        "print(t.word_docs)\n",
        "# integer encode documents\n",
        "encoded_docs = t.texts_to_matrix(docs, mode='count')\n",
        "print(encoded_docs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hBZjU6EghEff"
      },
      "source": [
        "#**Chapter 8: The Bags-of-Words Model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oyc3ZsUTm-TF"
      },
      "source": [
        "##8.3 What is a Bag-of-Words?\n",
        "Step1: Collect Data\n",
        "\n",
        "Step 2: Design the Vocabulary\n",
        "\n",
        "Step 3: Create Document Vectors\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m54kQQSehbYM"
      },
      "source": [
        "##8.6 **Scoring Words**\n",
        "\n",
        "*   **Counts.** Count the number of times each word appears in a document.\n",
        "*   **Frequencies.** Calculate the frequency that each word appears in a document out of all\n",
        "the words in the document.\n",
        "\n",
        "8.6.1: **Word Hashing**\n",
        "\n",
        "8.6.2: **TFIDF**\n",
        "\n",
        "\n",
        "* **Term Frequency:** is a scoring of the frequency of the word in the current document.\n",
        "* **Inverse Document Frequency:** is a scoring of how rare the word is across documents.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D5iiaX8ko7VQ"
      },
      "source": [
        "#**Chapter 9: How to Prepare Movie Review Data for Sentiment Analysis**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3pX7iMu5qBnS"
      },
      "source": [
        " After completing this tutorial, you\n",
        "will know:\n",
        "* How to load text data and clean it to remove punctuation and other non-words.\n",
        "* How to develop a vocabulary, tailor it, and save it to file.\n",
        "* How to prepare movie reviews using cleaning and a pre-defined vocabulary and save them\n",
        "to new files ready for modeling.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IoQqvh0mqMB_"
      },
      "source": [
        "##**9.1 Tutorial Overview**\n",
        "\n",
        "This tutorial is divided into the following parts:\n",
        "\n",
        "\n",
        "\n",
        "1.   Movie Review Dataset\n",
        "\n",
        "2.   Load Text Data\n",
        "\n",
        "3.   Clean Text Data\n",
        "\n",
        "4.   Develop Vocabulary\n",
        "\n",
        "5.   Save Prepared Data\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bBlnfnsirG9C"
      },
      "source": [
        "##**9.2 Movie Review Dataset**\n",
        "\n",
        "You can download the dataset from here:\n",
        " Movie Review Polarity Dataset (review polarity.tar.gz, 3MB).\n",
        "http://www.cs.cornell.edu/people/pabo/movie-review-data/review_polarity.tar.\n",
        "gz\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5FBQ4EeZr8WA"
      },
      "source": [
        "##**9.3 Load Text Data**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iWDp0b3ETRrU"
      },
      "source": [
        "# load one file\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg/cv000_29416.txt'\n",
        "# open the file as read only\n",
        "file = open(filename, 'r')\n",
        "# read all text\n",
        "text = file.read()\n",
        "# close the file\n",
        "file.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qUD3KTD2bxp-"
      },
      "source": [
        "This loads the document as ASCII and preserves any white space, like new lines. We can\n",
        "turn this into a function called load doc() that takes a filename of the document to load and\n",
        "returns the text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZgV6ibJ4PY3e"
      },
      "source": [
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "  # open the file as read only\n",
        "  file = open(filename, 'r')\n",
        "  # read all text\n",
        "  text = file.read()\n",
        "  # close the file\n",
        "  file.close()\n",
        "  return text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KScpMIV1bugh"
      },
      "source": [
        "We have two directories each with 1,000 documents each. We can process each directory in\n",
        "turn by first getting a list of files in the directory using the listdir() function, then loading\n",
        "each file in turn. For example, we can load each document in the negative directory using the\n",
        "load doc() function to do the actual loading.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XBaejfXYQcDO"
      },
      "source": [
        "from os import listdir\n",
        "\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "  # open the file as read only\n",
        "  file = open(filename, 'r')\n",
        "  # read all text\n",
        "  text = file.read()\n",
        "  # close the file\n",
        "  file.close()\n",
        "  return text\n",
        "\n",
        "# specify directory to load\n",
        "directory = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg'\n",
        "# walk through all files in the folder\n",
        "for filename in listdir(directory):\n",
        "  # skip files that do not have the right extension\n",
        "  if not filename.endswith(\".txt\"):\n",
        "    next \n",
        "  # create the path of the file to open\n",
        "  path = directory + '/' + filename\n",
        "  # load document\n",
        "  doc = load_doc(path)\n",
        "  print('Loaded %s' % filename)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0iYZ45PSk0Pi"
      },
      "source": [
        "We can turn the processing of the documents into a function as well and use it as a template\n",
        "later for developing a function to clean all documents in a folder. For example, below we define\n",
        "a process docs() function to do the same thing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gbtGiIxRiu1o"
      },
      "source": [
        "from os import listdir\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "  # open the file as read only\n",
        "  file = open(filename, 'r')\n",
        "  # read all text\n",
        "  text = file.read()\n",
        "  # close the file\n",
        "  file.close()\n",
        "  return text\n",
        "# load all docs in a directory\n",
        "def process_docs(directory):\n",
        "  # walk through all files in the folder\n",
        "  for filename in listdir(directory):\n",
        "  # skip files that do not have the right extension\n",
        "    if not filename.endswith(\".txt\"):\n",
        "      next\n",
        "    # create the full path of the file to open\n",
        "    path = directory + '/' + filename\n",
        "    # load document\n",
        "    doc = load_doc(path)\n",
        "    print('Loaded %s' % filename)\n",
        "# specify directory to load\n",
        "directory = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg'\n",
        "process_docs(directory)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y5Aiyw_DnNhZ"
      },
      "source": [
        "##**9.4 Clean Text Data**\n",
        "\n",
        "**9.4.1 Split into Tokens**\n",
        "\n",
        "First, let’s load one document and look at the raw tokens split by white space. We will use the\n",
        "**load doc()** function developed in the previous section. We can use the **split()** function to\n",
        "split the loaded document into tokens separated by white space."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "76e_c1Gkm1sc"
      },
      "source": [
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "  # open the file as read only\n",
        "  file = open(filename, 'r')\n",
        "  # read all text\n",
        "  text = file.read()\n",
        "  # close the file\n",
        "  file.close()\n",
        "  return text\n",
        "# load the document\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg/cv000_29416.txt'\n",
        "text = load_doc(filename)\n",
        "# split into tokens by white space\n",
        "tokens = text.split()\n",
        "print(tokens)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zsMD0xliumwh"
      },
      "source": [
        "Just looking at the raw tokens can give us a lot of ideas of things to try, such as:\n",
        "\n",
        "\n",
        "*   Remove punctuation from words (e.g. ‘what’s’).\n",
        "\n",
        "*   Removing tokens that are just punctuation (e.g. ‘-’).\n",
        "\n",
        "*   Removing tokens that contain numbers (e.g. ‘10/10’).\n",
        "\n",
        "*   Remove tokens that have one character (e.g. ‘a’).\n",
        "\n",
        "*   Remove tokens that don’t have much meaning (e.g. ‘and’).\n",
        "\n",
        "Some ideas:\n",
        "\n",
        "*   We can filter out punctuation from tokens using **regular expressions**.\n",
        "\n",
        "*   We can remove tokens that are just punctuation or contain numbers by using an **isalpha()**\n",
        "check on each token.\n",
        "\n",
        "*   We can remove English stop words using the list loaded using **NLTK**.\n",
        "\n",
        "\n",
        "*   We can filter out short tokens by checking their length.\n",
        "\n",
        "\n",
        "Below is an updated version of cleaning this review:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1aKpjGSJ0zhW"
      },
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ygxwGQ2Qsq7r"
      },
      "source": [
        "from nltk.corpus import stopwords\n",
        "import string\n",
        "import re\n",
        "\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "  # open the file as read only\n",
        "  file = open(filename, 'r')\n",
        "  # read all text\n",
        "  text = file.read()\n",
        "  # close the file\n",
        "  file.close()\n",
        "  return text\n",
        "\n",
        "# load the document\n",
        "directory = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg/cv000_29416.txt'\n",
        "text = load_doc(directory)\n",
        "# split the tokens by white space\n",
        "tokens = text.split()\n",
        "# prepare regex for char filtering\n",
        "re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "# remove punctuation from each word\n",
        "tokens = [re_punc.sub('', word) for word in tokens]\n",
        "# remove the remaining toens that are not alphanumeric\n",
        "tokens = [word for word in tokens if word.isalpha()]\n",
        "# filter out stopwards\n",
        "stop_words = set(stopwords.words('english'))\n",
        "tokens = [w for w in tokens if not w in stop_words]\n",
        "# filter out short tokens\n",
        "tokens = [word for word in tokens if len(word)>1]\n",
        "print(tokens)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_WohAHhy5kUN"
      },
      "source": [
        "We can put this into a function called **clean doc()** and test it on another review, this time\n",
        "a **positive review**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ra7Ct6I80sv0"
      },
      "source": [
        "from nltk.corpus import stopwords\n",
        "import string\n",
        "import re\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "  # open the file as read only\n",
        "  file = open(filename, 'r')\n",
        "  # read all text\n",
        "  text = file.read()\n",
        "  # close the file\n",
        "  file.close()\n",
        "  return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "  # split into tokens by white space\n",
        "  tokens = doc.split()\n",
        "  # prepare regex for char filtering\n",
        "  re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "  # remove punctuation from each word\n",
        "  tokens = [re_punc.sub('', w) for w in tokens]\n",
        "  # remove remaining tokens that are not alphabetic\n",
        "  tokens = [word for word in tokens if word.isalpha()]\n",
        "  # filter out stop words\n",
        "  stop_words = set(stopwords.words('english'))\n",
        "  tokens = [w for w in tokens if not w in stop_words]\n",
        "  # filter out short tokens\n",
        "  tokens = [word for word in tokens if len(word) > 1]\n",
        "  return tokens\n",
        "# load the document\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos/cv000_29590.txt'\n",
        "text = load_doc(filename)\n",
        "tokens = clean_doc(text)\n",
        "print(tokens)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c_HO1HU49asV"
      },
      "source": [
        "##**9.5 Develop Vocabulary**\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8wGDv33C5vrQ"
      },
      "source": [
        "We can keep track of the vocabulary in a Counter, which is a dictionary of words and their\n",
        "count with some additional convenience functions. We need to develop a new function to process\n",
        "a document and add it to the vocabulary. The function needs to load a document by calling the\n",
        "previously developed load doc() function. It needs to clean the loaded document using the\n",
        "previously developed clean doc() function, then it needs to add all the tokens to the Counter,\n",
        "and update counts. We can do this last step by calling the update() function on the counter\n",
        "object. Below is a function called add doc to vocab() that takes as arguments a document\n",
        "filename and a Counter vocabulary"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tb_91qma77nj"
      },
      "source": [
        "# load doc and add to vocab\n",
        "def add_doc_to_vocab(filename, vocab):\n",
        "  # load doc\n",
        "  doc = load_doc(filename)\n",
        "  # clean doc\n",
        "  tokens = clean_doc(doc)\n",
        "  # update counts\n",
        "  vocab.update(tokens)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "an_Uer_v8ais"
      },
      "source": [
        "Finally, we can use our template above for processing all documents in a directory called\n",
        "**process_docs()** and update it to call **add_doc_to_vocab()**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2XgPusZx54AR"
      },
      "source": [
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab):\n",
        "  # walk through all files in the folder\n",
        "  for filename in listdir(directory):\n",
        "    # skip files that do not have the right extensions\n",
        "    if not filename.endswith(\".txt\"):\n",
        "      next\n",
        "    # create the full path of the file to open\n",
        "    path = directory + '/' + filename\n",
        "    # add doc to vocab\n",
        "    add_doc_to_vocab(path, vocab)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2nv8HFFA_kYW"
      },
      "source": [
        "We can put all of this together and develop a full vocabulary from all documents in the\n",
        "dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dte_Yjry_Wks"
      },
      "source": [
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from collections import Counter\n",
        "from nltk.corpus import stopwords\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "# load doc and add to vocab\n",
        "def add_doc_to_vocab(filename, vocab):\n",
        "    # load doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # update counts\n",
        "    vocab.update(tokens)\n",
        "\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab):\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "      # skip files that do not have the right extension\n",
        "      if not filename.endswith(\".txt\"):\n",
        "        next\n",
        "      # create the full path of the file to open\n",
        "      path = directory + '/' + filename\n",
        "      # add doc to vocab\n",
        "      add_doc_to_vocab(path, vocab)\n",
        "\n",
        "# define vocab\n",
        "vocab = Counter()\n",
        "# add all docs to vocab\n",
        "process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab)\n",
        "process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab)\n",
        "# print the size of the vocab\n",
        "print(len(vocab))\n",
        "# print the top words in the vocab\n",
        "print(vocab.most_common(50))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5uoc_6wJSk0N"
      },
      "source": [
        "Perhaps the least common words, those that only appear once across all reviews, are not\n",
        "predictive. Perhaps some of the most common words are not useful too. Generally, words that\n",
        "only appear once or a few times across 2,000 reviews are probably not predictive and can be\n",
        "removed from the vocabulary. We can do\n",
        "this by stepping through words and their counts and only keeping those with a count above a\n",
        "chosen threshold. Here we will use 5 occurrences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rxiB09mfAVKu"
      },
      "source": [
        "# keep token with >5 occurrence\n",
        "min_occurance = 5\n",
        "tokens = [k for k,c in vocab.items() if c >= min_occurance]\n",
        "print(len(tokens))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GBcBkjI4TzwU"
      },
      "source": [
        "This reduces the vocabulary from 46,557 to 14,803 words, a huge drop. Perhaps a minimum\n",
        "of 5 occurrences is too aggressive; you can experiment with different values. We can then save\n",
        "the chosen vocabulary of words to a new file. I like to save the vocabulary as ASCII with one\n",
        "word per line. Below defines a function called save list() to save a list of items, in this case,\n",
        "tokens to file, one per line."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F4xv9tXITfBg"
      },
      "source": [
        "def save_list(lines, filename):\n",
        "  data = '\\n'.join(lines)\n",
        "  file = open(filename, 'w')\n",
        "  file.write(data)\n",
        "  file.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7_wcrM-s2WY1"
      },
      "source": [
        "The complete example for defining and saving the vocabulary is listed below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tIkUGIPXWzuF"
      },
      "source": [
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from collections import Counter\n",
        "from nltk.corpus import stopwords\n",
        "\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "\n",
        "# load doc and add to vocab\n",
        "def add_doc_to_vocab(filename, vocab):\n",
        "    # load doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # update counts\n",
        "    vocab.update(tokens)\n",
        "\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab):\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "    # skip files that do not have the right extension\n",
        "        if not filename.endswith(\".txt\"):\n",
        "           next\n",
        "        # create the full path of the file to open\n",
        "        path = directory + '/' + filename\n",
        "        # add doc to vocab\n",
        "        add_doc_to_vocab(path, vocab)\n",
        "\n",
        "# save list to file\n",
        "def save_list(lines, filename):\n",
        "    data = '\\n'.join(lines)\n",
        "    file = open(filename, 'w')\n",
        "    file.write(data)\n",
        "    file.close()\n",
        "\n",
        "# define vocab\n",
        "vocab = Counter()\n",
        "# add all docs to vocab\n",
        "process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab)\n",
        "process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab)\n",
        "# print the size of the vocab\n",
        "print(len(vocab))\n",
        "# print the top words in the vocab\n",
        "print(vocab.most_common(50))\n",
        "# keep tokens with > 5 occurrence\n",
        "min_occurane = 5\n",
        "tokens = [k for k,c in vocab.items() if c >= min_occurane]\n",
        "print(len(tokens))\n",
        "# save tokens to a vocabulary file\n",
        "save_list(tokens, '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/vocab.txt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EYF_WEI0PYAf"
      },
      "source": [
        "##**9.6 Save Prepared Data**\n",
        "\n",
        "The complete code listing is provided below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dn2i_2GOEHr-"
      },
      "source": [
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from nltk.corpus import stopwords\n",
        "\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "\n",
        "# save list to file\n",
        "def save_list(lines, filename):\n",
        "    data = '\\n'.join(lines)\n",
        "    file = open(filename, 'w')\n",
        "    file.write(data)\n",
        "    file.close()\n",
        "\n",
        "# load doc, clean and return line of tokens\n",
        "def doc_to_line(filename, vocab):\n",
        "    # load the doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # filter by vocab\n",
        "    tokens = [w for w in tokens if w in vocab]\n",
        "    return ' '.join(tokens)\n",
        "\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab):\n",
        "    lines = list()\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "      # skip files that do not have the right extension\n",
        "      if not filename.endswith(\".txt\"):\n",
        "        next\n",
        "      # create the full path of the file to open\n",
        "      path = directory + '/' + filename\n",
        "      # load and clean the doc\n",
        "      line = doc_to_line(path, vocab)\n",
        "      # add to list\n",
        "      lines.append(line)\n",
        "    return lines\n",
        "\n",
        "# load vocabulary\n",
        "vocab_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/vocab.txt'\n",
        "vocab = load_doc(vocab_filename)\n",
        "vocab = vocab.split()\n",
        "vocab = set(vocab)\n",
        "# prepare negative reviews\n",
        "negative_lines = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab)\n",
        "save_list(negative_lines, '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/negative.txt')\n",
        "# prepare positive reviews\n",
        "positive_lines = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab)\n",
        "save_list(positive_lines, '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/positive.txt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_kj6KzI2G-g3"
      },
      "source": [
        "#**Chapter 10: Project: Develop a Neural Bag-of-Words Model for Sentiment Analysis**\n",
        "\n",
        "After completing this tutorial, you will know:\n",
        "\n",
        "\n",
        "*   How to prepare the review text data for modeling with a restricted vocabulary.\n",
        "*   How to use the bag-of-words model to prepare train and test data.\n",
        "\n",
        "*   How to develop a Multilayer Perceptron bag-of-words model and use it to make predictions\n",
        "on new review text data.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scqsMK0xHuYr"
      },
      "source": [
        "##**10.1 Tutorial Overview**\n",
        "This tutorial is divided into the following parts:\n",
        "1. Movie Review Dataset\n",
        "2. Data Preparation\n",
        "3. Bag-of-Words Representation\n",
        "4. Sentiment Analysis Models\n",
        "5. Comparing Word Scoring Methods\n",
        "6. Predicting Sentiment for New Reviews"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iwr_xIaEH4Wf"
      },
      "source": [
        "##**10.2 Movie Review Dataset**\n",
        "\n",
        "*  Movie Review Polarity Dataset (review polarity.tar.gz, 3MB).\n",
        "http://www.cs.cornell.edu/people/pabo/movie-review-data/review_polarity.tar.\n",
        "gz\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcHiNPEVIGYp"
      },
      "source": [
        "##**10.3 Data Preparation**\n",
        "\n",
        "1. Separation of data into training and test sets.\n",
        "2. Loading and cleaning the data to remove punctuation and numbers.\n",
        "3. Defining a vocabulary of preferred words."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oP89ix1uI7_h"
      },
      "source": [
        "##**10.3.1 Split into Train and Test Sets**\n",
        "\n",
        "This is a 90% train, 10% split of the data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FN8bXgChJE7n"
      },
      "source": [
        "##**10.3.2 Loading and Cleaning Reviews**\n",
        "\n",
        "* Split tokens on white space.\n",
        "* Remove all punctuation from words.\n",
        "* Remove all words that are not purely comprised of alphabetical characters.\n",
        "* Remove all words that are known stop words.\n",
        "* Remove all words that have a length ≤ 1 character."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yw8WwObuJqu0"
      },
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3KeLBooNe-51"
      },
      "source": [
        "from nltk.corpus import stopwords\n",
        "import string\n",
        "import re\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "# load the document\n",
        "filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos/cv000_29590.txt'\n",
        "text = load_doc(filename)\n",
        "tokens = clean_doc(text)\n",
        "print(tokens)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GL7JPu_8KbuH"
      },
      "source": [
        "##**10.3.3 Define a Vocabulary**\n",
        "\n",
        "It is important to define a vocabulary of known words when using a bag-of-words model. The\n",
        "more words, the larger the representation of documents, therefore it is important to constrain\n",
        "the words to only those believed to be predictive. This is difficult to know beforehand and often\n",
        "it is important to test different hypotheses about how to construct a useful vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YO0OvHuaKVRx"
      },
      "source": [
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from collections import Counter\n",
        "from nltk.corpus import stopwords\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "\n",
        "# load doc and add to vocab\n",
        "def add_doc_to_vocab(filename, vocab):\n",
        "    # load doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # update counts\n",
        "    vocab.update(tokens)\n",
        "\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab):\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "      # skip any reviews in the test set\n",
        "      if filename.startswith('cv9'):\n",
        "        continue\n",
        "      # create the full path of the file to open\n",
        "      path = directory + '/' + filename\n",
        "      # add doc to vocab\n",
        "      add_doc_to_vocab(path, vocab)\n",
        "# define vocab\n",
        "vocab = Counter()\n",
        "# add all docs to vocab\n",
        "process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab)\n",
        "process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab)\n",
        "# print the size of the vocab\n",
        "print(len(vocab))\n",
        "# print the top words in the vocab\n",
        "print(vocab.most_common(50))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n7CRoAPX96Pk"
      },
      "source": [
        "Pulling all of this together, the complete example is listed below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NxIDqD4j9lM1"
      },
      "source": [
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from collections import Counter\n",
        "from nltk.corpus import stopwords\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "# load doc and add to vocab\n",
        "def add_doc_to_vocab(filename, vocab):\n",
        "    # load doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # update counts\n",
        "    vocab.update(tokens)\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab):\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "        # skip any reviews in the test set\n",
        "        if filename.startswith('cv9'):\n",
        "            continue\n",
        "        # create the full path of the file to open\n",
        "        path = directory + '/' + filename\n",
        "        # add doc to vocab\n",
        "        add_doc_to_vocab(path, vocab)\n",
        "# save list to file\n",
        "def save_list(lines, filename):\n",
        "    # convert lines to a single blob of text\n",
        "    data = '\\n'.join(lines)\n",
        "    # open file\n",
        "    file = open(filename, 'w')\n",
        "    # write text\n",
        "    file.write(data)\n",
        "    # close file\n",
        "    file.close()\n",
        "\n",
        "# define vocab\n",
        "vocab = Counter()\n",
        "\n",
        "# add all docs to vocab\n",
        "process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab)\n",
        "process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab)\n",
        "# print the size of the vocab\n",
        "print(len(vocab))\n",
        "\n",
        "# keep tokens with a min occurrence\n",
        "min_occurane = 2\n",
        "tokens = [k for k,c in vocab.items() if c >= min_occurane]\n",
        "print(len(tokens))\n",
        "\n",
        "# save tokens to a vocabulary file\n",
        "save_list(tokens, 'vocab.txt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tqk1ZeF3Co1X"
      },
      "source": [
        "##**10.4 Bag-of-Words Representation**\n",
        "\n",
        "In this section, we will look at how we can convert each review into a representation that we\n",
        "can provide to a Multilayer Perceptron model. A bag-of-words model is a way of extracting\n",
        "features from text so the text input can be used with machine learning algorithms like neural\n",
        "networks.\n",
        "\n",
        "\n",
        " This section is divided into 2 steps:\n",
        "1. Converting reviews to lines of tokens.\n",
        "2. Encoding reviews with a bag-of-words model representation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "clpBFWbwJ2JL"
      },
      "source": [
        "##**10.4.1 Reviews to Lines of Tokens**\n",
        "\n",
        "The complete example is listed below, demonstrating how to prepare the\n",
        "positive and negative reviews from the training dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ouQgrxL_CfdK"
      },
      "source": [
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from nltk.corpus import stopwords\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "# load doc, clean and return line of tokens\n",
        "def doc_to_line(filename, vocab):\n",
        "    # load the doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # filter by vocab\n",
        "    tokens = [w for w in tokens if w in vocab]\n",
        "    return ' '.join(tokens)\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab):\n",
        "    lines = list()\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "        # skip any reviews in the test set\n",
        "        if filename.startswith('cv9'):\n",
        "            continue\n",
        "        # create the full path of the file to open\n",
        "        path = directory + '/' + filename\n",
        "        # load and clean the doc\n",
        "        line = doc_to_line(path, vocab)\n",
        "        # add to list\n",
        "        lines.append(line)\n",
        "    return lines\n",
        "# load and clean a dataset\n",
        "def load_clean_dataset(vocab):\n",
        "    # load documents\n",
        "    neg = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab)\n",
        "    pos = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab)\n",
        "    docs = neg + pos\n",
        "    # prepare labels\n",
        "    labels = [0 for _ in range(len(neg))] + [1 for _ in range(len(pos))]\n",
        "    return docs, labels\n",
        "# load the vocabulary\n",
        "vocab_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/vocab.txt'\n",
        "vocab = load_doc(vocab_filename)\n",
        "vocab = vocab.split()\n",
        "vocab = set(vocab)\n",
        "# load all training reviews\n",
        "docs, labels = load_clean_dataset(vocab)\n",
        "# summarize what we have\n",
        "print(len(docs), len(labels))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pmwuxsV3b4sd"
      },
      "source": [
        "##**10.4.2 Movie Reviews to Bag-of-Words Vectors**\n",
        "\n",
        "\n",
        "We will use the Keras API to convert reviews to encoded document vectors. The Tokenizer class is convenient and will easily transform documents into encoded vectors. First, the Tokenizer must be created, then fit on the text documents\n",
        "in the training dataset. In this case, these are the aggregation of the positive lines and\n",
        "negative lines arrays developed in the previous section."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ghDmpt2IaF5b"
      },
      "source": [
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from nltk.corpus import stopwords\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "# load doc, clean and return line of tokens\n",
        "def doc_to_line(filename, vocab):\n",
        "    # load the doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # filter by vocab\n",
        "    tokens = [w for w in tokens if w in vocab]\n",
        "    return ' '.join(tokens)\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab, is_train):\n",
        "    lines = list()\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "        # skip any reviews in the test set\n",
        "        if is_train and filename.startswith('cv9'):\n",
        "            continue\n",
        "        if not is_train and not filename.startswith('cv9'):\n",
        "            continue\n",
        "        # create the full path of the file to open\n",
        "        path = directory + '/' + filename\n",
        "        # load and clean the doc\n",
        "        line = doc_to_line(path, vocab)\n",
        "        # add to list\n",
        "        lines.append(line)\n",
        "    return lines\n",
        "\n",
        "# load and clean a dataset\n",
        "def load_clean_dataset(vocab, is_train):\n",
        "    # load documents\n",
        "    neg = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab, is_train)\n",
        "    pos = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab, is_train)\n",
        "    docs = neg + pos\n",
        "    # prepare labels\n",
        "    labels = [0 for _ in range(len(neg))] + [1 for _ in range(len(pos))]\n",
        "    return docs, labels\n",
        "# fit a tokenizer\n",
        "def create_tokenizer(lines):\n",
        "    tokenizer = Tokenizer()\n",
        "    tokenizer.fit_on_texts(lines)\n",
        "    return tokenizer\n",
        "# load the vocabulary\n",
        "vocab_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/vocab.txt'\n",
        "vocab = load_doc(vocab_filename)\n",
        "vocab = set(vocab.split())\n",
        "# load all reviews\n",
        "train_docs, ytrain = load_clean_dataset(vocab, True)\n",
        "test_docs, ytest = load_clean_dataset(vocab, False)\n",
        "# create the tokenizer\n",
        "tokenizer = create_tokenizer(train_docs)\n",
        "# encode data\n",
        "Xtrain = tokenizer.texts_to_matrix(train_docs, mode='freq')\n",
        "Xtest = tokenizer.texts_to_matrix(test_docs, mode='freq')\n",
        "print(Xtrain.shape, Xtest.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mLE-nLvnteSZ"
      },
      "source": [
        "##**10.5 Sentiment Analysis Models**\n",
        "\n",
        "In this section, we will develop Multilayer Perceptron (MLP) models to classify encoded\n",
        "documents as either positive or negative. The models will be simple feedforward network models with fully connected layers called Dense in the Keras deep learning library. This section is\n",
        "divided into 3 sections:\n",
        "\n",
        "1. First sentiment analysis model\n",
        "2. Comparing word scoring modes\n",
        "3. Making a prediction for new reviews"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kxuz-VNAt6JF"
      },
      "source": [
        "###**10.5.1 First Sentiment Analysis Model**\n",
        "\n",
        "The complete example is listed below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KH1jm7N-tiGG"
      },
      "source": [
        "import string\n",
        "import numpy as np\n",
        "import re\n",
        "from os import listdir\n",
        "from nltk.corpus import stopwords\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.utils.vis_utils import plot_model\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "\n",
        "# load doc, clean and return line of tokens\n",
        "def doc_to_line(filename, vocab):\n",
        "    # load the doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # filter by vocab\n",
        "    tokens = [w for w in tokens if w in vocab]\n",
        "    return ' '.join(tokens)\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab, is_train):\n",
        "    lines = list()\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "        # skip any reviews in the test set\n",
        "        if is_train and filename.startswith('cv9'):\n",
        "            continue\n",
        "        if not is_train and not filename.startswith('cv9'):\n",
        "            continue\n",
        "        # create the full path of the file to open\n",
        "        path = directory + '/' + filename\n",
        "        # load and clean the doc\n",
        "        line = doc_to_line(path, vocab)\n",
        "        # add to list\n",
        "        lines.append(line)\n",
        "    return lines\n",
        "\n",
        "# load and clean a dataset\n",
        "def load_clean_dataset(vocab, is_train):\n",
        "    # load documents\n",
        "    neg = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab, is_train)\n",
        "    pos = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab, is_train)\n",
        "    docs = neg + pos\n",
        "    # prepare labels\n",
        "    labels = [0 for _ in range(len(neg))] + [1 for _ in range(len(pos))]\n",
        "    return docs, labels\n",
        "\n",
        "# fit a tokenizer\n",
        "def create_tokenizer(lines):\n",
        "    tokenizer = Tokenizer()\n",
        "    tokenizer.fit_on_texts(lines)\n",
        "    return tokenizer\n",
        "\n",
        "# define the model\n",
        "def define_model(n_words):\n",
        "    # define network\n",
        "    model = Sequential()\n",
        "    model.add(Dense(50, input_shape=(n_words,), activation='relu'))\n",
        "    model.add(Dense(1, activation='sigmoid'))\n",
        "    # compile network\n",
        "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "    # summarize defined model\n",
        "    model.summary()\n",
        "    plot_model(model, to_file='model.png', show_shapes=True)\n",
        "    return model\n",
        "\n",
        "# load the vocabulary\n",
        "vocab_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/vocab.txt'\n",
        "vocab = load_doc(vocab_filename)\n",
        "vocab = set(vocab.split())\n",
        "# load all reviews\n",
        "train_docs, ytrain = load_clean_dataset(vocab, True)\n",
        "test_docs, ytest = load_clean_dataset(vocab, False)\n",
        "# create the tokenizer\n",
        "tokenizer = create_tokenizer(train_docs)\n",
        "# encode data\n",
        "Xtrain = np.array(tokenizer.texts_to_matrix(train_docs, mode='freq'))\n",
        "Xtest = np.array(tokenizer.texts_to_matrix(test_docs, mode='freq'))\n",
        "# define the model\n",
        "n_words = Xtest.shape[1]\n",
        "model = define_model(n_words)\n",
        "# fit network\n",
        "model.fit(np.array(Xtrain, dtype='float64'), np.array(ytrain, dtype='float64'), epochs=10, verbose=2)\n",
        "# evaluate\n",
        "loss, acc = model.evaluate(np.array(Xtest, dtype='float64'), np.array(ytest, dtype='float64'), verbose=0)\n",
        "print('Test Accuracy: %f' % (acc*100))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5rtcG_JFz7LB"
      },
      "source": [
        "##**10.6 Comparing Word Scoring Methods**\n",
        "\n",
        "We are now going to evaluate the performance of the 4 different word scoring methods.\n",
        "Pulling all of this together, the complete example is listed below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IgfROEe8mec-"
      },
      "source": [
        "import string\n",
        "import numpy as np\n",
        "import re\n",
        "from os import listdir\n",
        "from nltk.corpus import stopwords\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from pandas import DataFrame\n",
        "from matplotlib import pyplot\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "# load doc, clean and return line of tokens\n",
        "def doc_to_line(filename, vocab):\n",
        "    # load the doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # filter by vocab\n",
        "    tokens = [w for w in tokens if w in vocab]\n",
        "    return ' '.join(tokens)\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab, is_train):\n",
        "    lines = list()\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "        # skip any reviews in the test set\n",
        "        if is_train and filename.startswith('cv9'):\n",
        "            continue\n",
        "        if not is_train and not filename.startswith('cv9'):\n",
        "            continue\n",
        "    # create the full path of the file to open\n",
        "    path = directory + '/' + filename\n",
        "    # load and clean the doc\n",
        "    line = doc_to_line(path, vocab)\n",
        "    # add to list\n",
        "    lines.append(line)\n",
        "    return lines\n",
        "# load and clean a dataset\n",
        "def load_clean_dataset(vocab, is_train):\n",
        "    # load documents\n",
        "    neg = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab, is_train)\n",
        "    pos = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab, is_train)\n",
        "    docs = neg + pos\n",
        "    # prepare labels\n",
        "    labels = [0 for _ in range(len(neg))] + [1 for _ in range(len(pos))]\n",
        "    return docs, labels\n",
        "# define the model\n",
        "def define_model(n_words):\n",
        "    # define network\n",
        "    model = Sequential()\n",
        "    model.add(Dense(50, input_shape=(n_words,), activation='relu'))\n",
        "    model.add(Dense(1, activation='sigmoid'))\n",
        "    # compile network\n",
        "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "    return model\n",
        "# evaluate a neural network model\n",
        "def evaluate_mode(Xtrain, ytrain, Xtest, ytest):\n",
        "    scores = list()\n",
        "    n_repeats = 10\n",
        "    n_words = Xtest.shape[1]\n",
        "    for i in range(n_repeats):\n",
        "        # define network\n",
        "        model = define_model(n_words)\n",
        "        # fit network\n",
        "        model.fit(Xtrain, ytrain, epochs=10, verbose=0)\n",
        "        # evaluate\n",
        "        _, acc = model.evaluate(Xtest, ytest, verbose=0)\n",
        "        scores.append(acc)\n",
        "        print('%d accuracy: %s' % ((i+1), acc))\n",
        "    return scores\n",
        "# prepare bag of words encoding of docs\n",
        "def prepare_data(train_docs, test_docs, mode):\n",
        "    # create the tokenizer\n",
        "    tokenizer = Tokenizer()\n",
        "    # fit the tokenizer on the documents\n",
        "    tokenizer.fit_on_texts(train_docs)\n",
        "    # encode training data set\n",
        "    Xtrain = tokenizer.texts_to_matrix(train_docs, mode=mode)\n",
        "    # encode training data set\n",
        "    Xtest = tokenizer.texts_to_matrix(test_docs, mode=mode)\n",
        "    return Xtrain, Xtest\n",
        "# load the vocabulary\n",
        "vocab_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/vocab.txt'\n",
        "vocab = load_doc(vocab_filename)\n",
        "vocab = set(vocab.split())\n",
        "# load all reviews\n",
        "train_docs, ytrain = load_clean_dataset(vocab, True)\n",
        "test_docs, ytest = load_clean_dataset(vocab, False)\n",
        "# run experiment\n",
        "modes = ['binary', 'count', 'tfidf', 'freq']\n",
        "results = DataFrame()\n",
        "for mode in modes:\n",
        "    # prepare data for mode\n",
        "    Xtrain, Xtest = prepare_data(train_docs, test_docs, mode)\n",
        "    # evaluate model on data for mode\n",
        "    results[mode] = evaluate_mode(np.array(Xtrain,dtype='float64'), np.array(ytrain,dtype='float'), np.array(Xtest,dtype='float'), np.array(ytest,dtype='float64'))\n",
        "# summarize results\n",
        "print(results.describe())\n",
        "# plot results\n",
        "results.boxplot()\n",
        "pyplot.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m9OD-a2Y67_a"
      },
      "source": [
        "##**10.7 Predicting Sentiment for New Reviews**\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Predicting the sentiment of new reviews involves following the same steps used to prepare\n",
        "the test data. Specifically, loading the text, cleaning the document, filtering tokens by the\n",
        "chosen vocabulary, converting the remaining tokens to a line, encoding it using the Tokenizer,\n",
        "and making a prediction. We can make a prediction of a class value directly with the fit model\n",
        "by calling predict() that will return an integer of 0 for a negative review and 1 for a positive\n",
        "review. All of these steps can be put into a new function called predict sentiment() that\n",
        "requires the review text, the vocabulary, the tokenizer, and the fit model and returns the\n",
        "predicted sentiment and an associated percentage or confidence-like output."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RFosEBM16fZ8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 663
        },
        "outputId": "404ca02b-d6b1-4d89-e323-62fedad5a52e"
      },
      "source": [
        "import nltk\n",
        "import numpy as np\n",
        "nltk.download('stopwords')\n",
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from nltk.corpus import stopwords\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.utils.vis_utils import plot_model\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    return tokens\n",
        "# load doc, clean and return line of tokens\n",
        "def doc_to_line(filename, vocab):\n",
        "    # load the doc\n",
        "    doc = load_doc(filename)\n",
        "    # clean doc\n",
        "    tokens = clean_doc(doc)\n",
        "    # filter by vocab\n",
        "    tokens = [w for w in tokens if w in vocab]\n",
        "    return ' '.join(tokens)\n",
        "\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab):\n",
        "    lines = list()\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "    # create the full path of the file to open\n",
        "      path = directory + '/' + filename\n",
        "      # load and clean the doc\n",
        "      line = doc_to_line(path, vocab)\n",
        "      # add to list\n",
        "      lines.append(line)\n",
        "    return lines\n",
        "\n",
        "# load and clean a dataset\n",
        "def load_clean_dataset(vocab):\n",
        "    # load documents\n",
        "    neg = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab)\n",
        "    pos = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab)\n",
        "    docs = neg + pos\n",
        "    # prepare labels\n",
        "    labels = [0 for _ in range(len(neg))] + [1 for _ in range(len(pos))]\n",
        "    return docs, labels\n",
        "\n",
        "# fit a tokenizer\n",
        "def create_tokenizer(lines):\n",
        "    tokenizer = Tokenizer()\n",
        "    tokenizer.fit_on_texts(lines)\n",
        "    return tokenizer\n",
        "# define the model\n",
        "def define_model(n_words):\n",
        "    # define network\n",
        "    model = Sequential()\n",
        "    model.add(Dense(50, input_shape=(n_words,), activation='relu'))\n",
        "    model.add(Dense(1, activation='sigmoid'))\n",
        "    # compile network\n",
        "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "    # summarize defined model\n",
        "    model.summary()\n",
        "    plot_model(model, to_file='model.png', show_shapes=True)\n",
        "    return model\n",
        "# classify a review as negative or positive\n",
        "def predict_sentiment(review, vocab, tokenizer, model):\n",
        "    # clean\n",
        "    tokens = clean_doc(review)\n",
        "    # filter by vocab\n",
        "    tokens = [w for w in tokens if w in vocab]\n",
        "    # convert to line\n",
        "    line = ' '.join(tokens)\n",
        "    # encode\n",
        "    encoded = tokenizer.texts_to_matrix([line], mode='binary')\n",
        "    # predict sentiment\n",
        "    yhat = model.predict(encoded, verbose=0)\n",
        "    # retrieve predicted percentage and label\n",
        "    percent_pos = yhat[0,0]\n",
        "    if round(percent_pos) == 0:\n",
        "      return (1-percent_pos), 'NEGATIVE'\n",
        "    return percent_pos, 'POSITIVE'\n",
        "\n",
        "# load the vocabulary\n",
        "vocab_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/vocab.txt'\n",
        "vocab = load_doc(vocab_filename)\n",
        "vocab = set(vocab.split())\n",
        "# load all reviews\n",
        "train_docs, ytrain = load_clean_dataset(vocab)\n",
        "test_docs, ytest = load_clean_dataset(vocab)\n",
        "# create the tokenizer\n",
        "tokenizer = create_tokenizer(train_docs)\n",
        "# encode data\n",
        "Xtrain = tokenizer.texts_to_matrix(train_docs, mode='binary')\n",
        "Xtest = tokenizer.texts_to_matrix(test_docs, mode='binary')\n",
        "# define network\n",
        "n_words = Xtrain.shape[1]\n",
        "model = define_model(n_words)     \n",
        "# fit network\n",
        "model.fit(np.array(Xtrain, dtype='float64'), np.array(ytrain, dtype='float64'), epochs=10, verbose=2)\n",
        "# test positive text\n",
        "text = 'Best movie ever! It was great, I recommend it.'\n",
        "percent, sentiment = predict_sentiment(text, vocab, tokenizer, model)\n",
        "print('Review: [%s]\\nSentiment: %s (%.3f%%)' % (text, sentiment, percent*100))\n",
        "# test negative text\n",
        "text = 'This is a bad movie.'\n",
        "percent, sentiment = predict_sentiment(text, vocab, tokenizer, model)\n",
        "print('Review: [%s]\\nSentiment: %s (%.3f%%)' % (text, sentiment, percent*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_4 (Dense)              (None, 50)                653650    \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 1)                 51        \n",
            "=================================================================\n",
            "Total params: 653,701\n",
            "Trainable params: 653,701\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/10\n",
            "51/51 - 0s - loss: 0.4683 - accuracy: 0.7716\n",
            "Epoch 2/10\n",
            "51/51 - 0s - loss: 0.0806 - accuracy: 0.9913\n",
            "Epoch 3/10\n",
            "51/51 - 0s - loss: 0.0247 - accuracy: 1.0000\n",
            "Epoch 4/10\n",
            "51/51 - 0s - loss: 0.0107 - accuracy: 1.0000\n",
            "Epoch 5/10\n",
            "51/51 - 0s - loss: 0.0057 - accuracy: 1.0000\n",
            "Epoch 6/10\n",
            "51/51 - 0s - loss: 0.0035 - accuracy: 1.0000\n",
            "Epoch 7/10\n",
            "51/51 - 0s - loss: 0.0024 - accuracy: 1.0000\n",
            "Epoch 8/10\n",
            "51/51 - 0s - loss: 0.0018 - accuracy: 1.0000\n",
            "Epoch 9/10\n",
            "51/51 - 0s - loss: 0.0013 - accuracy: 1.0000\n",
            "Epoch 10/10\n",
            "51/51 - 0s - loss: 0.0010 - accuracy: 1.0000\n",
            "Review: [Best movie ever! It was great, I recommend it.]\n",
            "Sentiment: POSITIVE (66.414%)\n",
            "Review: [This is a bad movie.]\n",
            "Sentiment: NEGATIVE (59.677%)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tgLOAfZI1DFa"
      },
      "source": [
        "#**Part V Word Embeddings**\n",
        "\n",
        "**Chapter 11\n",
        "The Word Embedding Model**\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y0VVefXN1kU6"
      },
      "source": [
        "#**Chapter 12 How to Develop Word Embeddings with Gensim**\n",
        "\n",
        "Word embeddings are a modern approach for representing text in natural language processing.\n",
        "Embedding algorithms like Word2Vec and GloVe are key to the state-of-the-art results achieved\n",
        "by neural network models on natural language processing problems like machine translation.\n",
        "In this tutorial, you will discover how to train and load word embedding models for natural\n",
        "language processing applications in Python using Gensim. After completing this tutorial, you\n",
        "will know:\n",
        "\n",
        "* How to train your own Word2Vec word embedding model on text data.\n",
        "* How to visualize a trained word embedding model using Principal Component Analysis.\n",
        "* How to load pre-trained Word2Vec and GloVe word embedding models from Google and\n",
        "Stanford."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FTvvQY4bpdQx"
      },
      "source": [
        "##**12.1 Tutorial Overview**\n",
        "This tutorial is divided into the following parts:\n",
        "1. Word Embeddings\n",
        "2. Gensim Library\n",
        "3. Develop Word2Vec Embedding\n",
        "4. Visualize Word Embedding\n",
        "5. Load Google’s Word2Vec Embedding\n",
        "6. Load Stanford’s GloVe Embedding\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "89cBkPi5p1z7"
      },
      "source": [
        "##**12.2 Word Embeddings**\n",
        "\n",
        "A word embedding is an approach to provide a dense vector representation of words that capture\n",
        "something about their meaning. Word embeddings are an improvement over simpler bag-of-word\n",
        "model word encoding schemes like word counts and frequencies that result in large and sparse\n",
        "vectors (mostly 0 values) that describe documents but not the meaning of the words.\n",
        "\n",
        "   Word embeddings work by using an algorithm to train a set of fixed-length dense and\n",
        "continuous-valued vectors based on a large corpus of text. Each word is represented by a\n",
        "point in the embedding space and these points are learned and moved around based on the\n",
        "words that surround the target word. It is defining a word by the company that it keeps that\n",
        "allows the word embedding to learn something about the meaning of words. The vector space\n",
        "representation of the words provides a projection where words with similar meanings are locally\n",
        "clustered within the space.\n",
        "\n",
        "  The use of word embeddings over other text representations is one of the key methods that\n",
        "has led to breakthrough performance with deep neural networks on problems like machine\n",
        "translation. In this tutorial, we are going to look at how to use two different word embedding\n",
        "methods called Word2Vec by researchers at Google and GloVe by researchers at Stanford."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HmY3yFKdsrdh"
      },
      "source": [
        "##**12.3 Gensim Python Library**\n",
        "\n",
        "Gensim is an open source Python library for natural language processing, with a focus on topic\n",
        "modeling. It is billed as “topic modeling for humans”. Gensim was developed and is maintained\n",
        "by the Czech natural language processing researcher Radim Rehurek and his company RaRe\n",
        "Technologies. It is not an everything-including-the-kitchen-sink NLP research library (like\n",
        "NLTK); instead, Gensim is a mature, focused, and efficient suite of NLP tools for topic modeling.\n",
        "Most notably for this tutorial, it supports an implementation of the Word2Vec word embedding\n",
        "for learning new word vectors from text.\n",
        "\n",
        "\n",
        "It also provides tools for loading pre-trained word embeddings in a few formats and for\n",
        "making use and querying a loaded embedding. We will use the Gensim library in this tutorial.\n",
        "Gensim can be installed easily using pip or easy install. For example, you can install Gensim\n",
        "with pip by typing the following on your command line:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y0R5VbzUpcZK",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 326
        },
        "outputId": "ecd933e4-a7a5-4eb3-eebe-120f706be18b"
      },
      "source": [
        "!pip install gensim"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: gensim in /usr/local/lib/python3.6/dist-packages (3.6.0)\n",
            "Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.18.5)\n",
            "Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.4.1)\n",
            "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (2.1.0)\n",
            "Requirement already satisfied: six>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.15.0)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim) (1.14.37)\n",
            "Requirement already satisfied: boto in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim) (2.49.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim) (2.23.0)\n",
            "Requirement already satisfied: botocore<1.18.0,>=1.17.37 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim) (1.17.37)\n",
            "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim) (0.3.3)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim) (0.10.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (2020.6.20)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (1.24.3)\n",
            "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.18.0,>=1.17.37->boto3->smart-open>=1.2.1->gensim) (2.8.1)\n",
            "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.18.0,>=1.17.37->boto3->smart-open>=1.2.1->gensim) (0.15.2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YFJiFVKquAtt"
      },
      "source": [
        "##**12.4 Develop Word2Vec Embedding**\n",
        "\n",
        "Word2Vec is one algorithm for learning a word embedding from a text corpus. There are two\n",
        "main training algorithms that can be used to learn the embedding from text; they are Continuous\n",
        "Bag-of-Words (CBOW) and skip grams. \n",
        "\n",
        " Gensim provides the\n",
        "Word2Vec class for working with a Word2Vec model.\n",
        "\n",
        "\n",
        "There are many parameters on this constructor; a few noteworthy arguments you may\n",
        "wish to configure are:\n",
        "* size: (default 100) The number of dimensions of the embedding, e.g. the length of the\n",
        "dense vector to represent each token (word).\n",
        "* window: (default 5) The maximum distance between a target word and words around the\n",
        "target word.\n",
        "* min count: (default 5) The minimum count of words to consider when training the model;\n",
        "words with an occurrence less than this count will be ignored.\n",
        "* workers: (default 3) The number of threads to use while training.\n",
        "* sg: (default 0 or CBOW) The training algorithm, either CBOW (0) or skip gram (1).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w0WKSHoK1X7R"
      },
      "source": [
        "from gensim.models import Word2Vec\n",
        "# define training data\n",
        "sentences = [['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec'],\n",
        "['this', 'is', 'the', 'second', 'sentence'],\n",
        "['yet', 'another', 'sentence'],\n",
        "['one', 'more', 'sentence'],\n",
        "['and', 'the', 'final', 'sentence']]\n",
        "# train model\n",
        "model = Word2Vec(sentences, min_count=1)\n",
        "# summarize the loaded model\n",
        "print(model)\n",
        "# summarize vocabulary\n",
        "words = list(model.wv.vocab)\n",
        "print(words)\n",
        "# access vector for one word\n",
        "print(model['sentence'])\n",
        "# save model\n",
        "model.save('model.bin')\n",
        "# load model\n",
        "new_model = Word2Vec.load('model.bin')\n",
        "print(new_model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cLW1xcbL7hLR"
      },
      "source": [
        "##**12.5 Visualize Word Embedding**\n",
        "\n",
        "After you learn word embedding for your text data, it can be nice to explore it with visualization.\n",
        "You can use classical projection methods to reduce the high-dimensional word vectors to twodimensional plots and plot them on a graph. The visualizations can provide a qualitative\n",
        "diagnostic for your learned model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u8A1Tkb679gb"
      },
      "source": [
        "###**12.5.1 Plot Word Vectors Using PCA**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zujSvApx68uj",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 557
        },
        "outputId": "eade7c96-de36-44e7-a0d3-e213ce2a0589"
      },
      "source": [
        "from gensim.models import Word2Vec\n",
        "from sklearn.decomposition import PCA\n",
        "from matplotlib import pyplot\n",
        "# define training data\n",
        "sentences = [['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec'],\n",
        "['this', 'is', 'the', 'second', 'sentence'],\n",
        "['yet', 'another', 'sentence'],\n",
        "['one', 'more', 'sentence'],\n",
        "['and', 'the', 'final', 'sentence']]\n",
        "# train model\n",
        "model = Word2Vec(sentences, min_count=1)\n",
        "# fit a 2d PCA model to the vectors\n",
        "X = model[model.wv.vocab]\n",
        "pca = PCA(n_components=2)\n",
        "result = pca.fit_transform(X)\n",
        "print(result)\n",
        "# create a scatter plot of the projection\n",
        "pyplot.scatter(result[:, 0], result[:, 1])\n",
        "words = list(model.wv.vocab)\n",
        "for i, word in enumerate(words):\n",
        "   pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))\n",
        "pyplot.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[-0.00788266 -0.01642513]\n",
            " [ 0.00031284 -0.00812224]\n",
            " [-0.0031435  -0.01019156]\n",
            " [ 0.01674983  0.00287382]\n",
            " [ 0.00410513  0.00561506]\n",
            " [-0.00069731 -0.00159713]\n",
            " [ 0.00164515 -0.00633078]\n",
            " [-0.0005942  -0.00353954]\n",
            " [-0.00413411 -0.00503733]\n",
            " [ 0.00578829  0.01553079]\n",
            " [-0.01899787  0.00572339]\n",
            " [-0.01304885  0.00960203]\n",
            " [-0.00036064  0.01652364]\n",
            " [ 0.02025789 -0.00462501]]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:13: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).\n",
            "  del sys.path[0]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAD4CAYAAAAUymoqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3xU1bn/8c9jCCGC3AQpNwtVBCEJCQREKIgoBBTlov6kchSkeDmtiraNxqKnXqqmxVMVqVoUqOWooKhAvYBc5aJiggkIyJ2oRLQR5BIETML6/TE7cYgTyGSGTEK+79drXtl7zVprnr0zyTN77TV7m3MOERGRYJwW6QBERKT6UfIQEZGgKXmIiEjQlDxERCRoSh4iIhK0WpEOIJyaNGni2rRpE+kwRESqldWrV3/rnGsaTJtTKnm0adOGzMzMSIchIlKtmNnnwbbRsJWIiARNyUOkiqhXr16kQxApNyUPEREJmpKHSBgNHTqUrl270qlTJyZPngz4jijGjx9P586d6dGjB9988w0AO3bs4MILLyQ+Pp777rsvkmGLBE3JQySMpk6dyurVq8nMzGTixIns3r2bgwcP0qNHD9asWUOfPn14/vnnARg3bhz//d//zaeffkrz5s0jHLlIcE6p2VYikTA7K5cJ8zfx1d5DFGa+Sq0vMqgfG82XX37Jli1bqF27NoMHDwaga9euLFiwAICVK1fy+uuvA3D99ddzzz33RGwbRIKlIw+REMzOyuXeNz4ld+8hDn2xlryNmdQe/igP/vMdkpKSOHz4MNHR0ZgZAFFRURQWFpa0Ly4XqW6UPERCMGH+Jg4VFAFw9Mj3nFanLkeI5sHpC/joo4+O27ZXr17MmDEDgJdeeumkx1pZZs+ezYYNG0rW+/btq+9fnYKUPERC8NXeQyXLsW274o4eJff5W9n01j/o0aPHcds+9dRT/P3vfyc+Pp7c3NyTHWqlKZ08QuF/lCZVi51KN4NKTk52+oQjlalX+mJy/RJIsZYNY1mZ1i8CEYVm6NChfPnllxw+fJhx48Zx8803U69ePcaNG8dbb71FbGwsc+bMoVmzZuTk5DBmzBi+/fZbmjZtyrRp09i5cyeDBw+mQYMGNGjQgNdff51f//rXXHDBBSxZsoS9e/cyZcoUevfuTVFREWlpaSxdupQjR47w29/+lltuuYWlS5dy//3306hRIzZu3MjmzZsjvVtOeWa22jmXHEwbHXmIhCA1pT2x0VHHlMVGR5Ga0j5CEYUmmNlit99+O6NGjWLt2rWMHDmSO+64g549e3LllVcyYcIEsrOzOeeccwDfEcTHH3/Mk08+yYMPPgjAlClTaNCgARkZGWRkZPD888+zY8cOAD755BOeeuopJY4qTLOtREIwNKklQMlsqxYNY0lNaV9SXtX5zxRr0TCW1jve4rOPFgGccLbYhx9+yBtvvAH4ZovdfffdZb7O8OHDS9rn5OQA8N5777F27VpmzZoFwL59+0per3v37rRt2/akbLOEh5KHSIiGJrWsNsnCX/FMseIT/tvWriJr+XymzZzDtT3PpW/fviecLVZeMTExP2nvnOPpp58mJSXlmLpLly6lbt26oWyaVAINW4nUUP4zxcA3W4yYukxc9gUbN2484Wyxnj17HjNbrHfv3gCcccYZHDhw4ISvn5KSwrPPPktBQQEAmzdv5uDBgxXdHKlkYUkeZjbQzDaZ2VYzSwvwfIyZzfSeX2VmbbzyM81siZnlm9mkUm2Wen1me4+zwhGriPh8VepEf/FssYwJo0hLSzvhbLGnn36aadOmkZCQwPTp03nqqacAGDFiBBMmTCApKYlt27aV2X7s2LF07NiRLl26EBcXxy233KLZVdVIyLOtzCwK2Az0B3YCGcCvnHMb/Or8Bkhwzt1qZiOAYc65a82sLpAExAFxzrnb/NosBf7gnCv39CnNthIpv1NtpphUXKRmW3UHtjrntjvnfgBmAENK1RkCvOgtzwIuMTNzzh10zq0ADochDhEJwqk2U0wqVziSR0vgS7/1nV5ZwDrOuUJgH3BmOfqe5g1Z3W9lXMfBzG42s0wzy8zLyws+epEaamhSSx4bHk/LhrEYviOOx4bHV8uT/1L5qvJsq5HOuVwzOwN4Hbge+FfpSs65ycBk8A1bVW6IItVbdZ0pJpEXjiOPXKC133orryxgHTOrBTQAdh+vU+dcrvfzAPAyvuExERGpAsKRPDKAdmbW1sxqAyOAuaXqzAVGectXA4vdcc7Um1ktM2viLUcDg4F1YYhVRETCIORhK+dcoZndBswHooCpzrn1ZvYQkOmcmwtMAaab2VZgD74EA4CZ5QD1gdpmNhQYAHwOzPcSRxSwEHg+1FhFRCQ8dGFEEZEaThdGFBGRSqHkISIiQVPyEBGRoCl5iIhI0JQ8REQkaEoeIiISNCUPEREJmpKHiIgETclDRESCpuQhIiJBU/IQEZGgKXmIiEjQlDxERCRoSh4iIhI0JQ8REQmakoeIiARNyUNERIKm5CEiIkFT8hARkaApeYiISNCUPEREJGhhSR5mNtDMNpnZVjNLC/B8jJnN9J5fZWZtvPIzzWyJmeWb2aRSbbqa2adem4lmZuGIVcpWWFgY6RBEpJoIOXmYWRTwd2AQ0BH4lZl1LFXt18B3zrlzgSeAv3jlh4H7gT8E6PpZ4CagnfcYGGqsp6qcnBw6dOjA6NGjOe+88xg5ciQLFy6kV69etGvXjo8//pg9e/YwdOhQEhIS6NGjB2vXrgXggQce4Prrr6dXr15cf/315OXlcdVVV9GtWze6devGypUrI7x1IlIV1QpDH92Brc657QBmNgMYAmzwqzMEeMBbngVMMjNzzh0EVpjZuf4dmllzoL5z7iNv/V/AUODdMMR7Stq6dSuvvfYaU6dOpVu3brz88susWLGCuXPn8uijj9K6dWuSkpKYPXs2ixcv5oYbbiA7OxuADRs2sGLFCmJjY7nuuuu46667+OUvf8kXX3xBSkoKn332WYS3TkSqmnAkj5bAl37rO4ELyqrjnCs0s33AmcC3x+lzZ6k+WwaqaGY3AzcDnH322cHGXm3NzsplwvxNfLX3EI3dPs5q0Zr4+HgAOnXqxCWXXIKZER8fT05ODp9//jmvv/46AP369WP37t3s378fgCuvvJLY2FgAFi5cyIYNP+b9/fv3k5+fT7169Sp5C0WkKgtH8ogo59xkYDJAcnKyi3A4lWJ2Vi73vvEphwqKAPhm/2F2H3bMzsplaFJLTjvtNGJiYgA47bTTKCwsJDo6usz+6tatW7J89OhRPvroI+rUqXNyN0JEqrVwnDDPBVr7rbfyygLWMbNaQANg9wn6bHWCPmusCfM3lSSOYs45JszfVGab3r1789JLLwGwdOlSmjRpQv369X9Sb8CAATz99NMl68VDWyIi/sKRPDKAdmbW1sxqAyOAuaXqzAVGectXA4udc2UeJTjndgH7zayHN8vqBmBOGGI9JXy191BQ5eA7Mb569WoSEhJIS0vjxRdfDFhv4sSJZGZmkpCQQMeOHXnuuefCErOInFrsOP/Dy9+J2WXAk0AUMNU594iZPQRkOufmmlkdYDqQBOwBRvidYM8B6gO1gb3AAOfcBjNLBv4JxOI7UX778RIO+IatMjMzQ96eqq5X+mJyAySKlg1jWZnWLwIRiUh1ZmarnXPJwbQJyzkP59w7wDulyv7Hb/kwcE0ZbduUUZ4JxIUjvlNNakr7Y855AMRGR5Ga0j6CUYlITVLtT5jXREOTfBPPimdbtWgYS2pK+5JyEZGTTcmjmhqa1FLJQkQiRte2EhGRoCl5iIhI0JQ8REQkaEoeIiISNCUPEREJmpKHiIgETclDRESCpuQhIiJBU/IQEZGgKXmIiEjQlDxERCRoSh4iIhI0JQ8RCSg7O5t33nnnxBWlRlLyKOVvf/sbcXFxxMXF8eSTT5KTk8P555/PTTfdRKdOnRgwYACHDvluxLRt2zYGDhxI165d6d27Nxs3boxw9CLho+Qhx+WcO2UeXbt2daHIzMx0cXFxLj8/3x04cMB17NjRffLJJy4qKsplZWU555y75ppr3PTp051zzvXr189t3rzZOefcRx995C6++OKQXl8kXPLz891ll13mEhISXKdOndyMGTNcZmam69Onj+vSpYsbMGCA++qrr5xzzl100UXu7rvvdt26dXPt2rVzy5Ytc0eOHHGtW7d2TZo0cZ07d3YzZsxw+fn57sYbb3TdunVziYmJbvbs2c4556ZNm+aGDRvmUlJS3LnnnutSU1NL4nj33XddUlKSS0hIcP369SuJLVA/Ejn47voa1P/bGn8/j9lZuSU3VWL9O3S78BLq1q0LwPDhw1m+fDlt27YlMTERgK5du5KTk0N+fj4ffPAB11zz4w0Sjxw5EpFtEClt3rx5tGjRgrfffhuAffv2MWjQIObMmUPTpk2ZOXMm48ePZ+rUqQAUFhby8ccf88477/Dggw+ycOFCHnroITIzM5k0aRIAf/zjH+nXrx9Tp05l7969dO/enUsvvRTwHaVkZWURExND+/btuf3226lTpw433XQTy5Yto23btuzZsweARx55JGA/xX93Uj3U6OQxOyv3mNu57j9UwOLPvmN2Vu4xN1qKiYkpWY6KiuLQoUMcPXqUhg0bkp2dXelxiwTi/0GoUUE+O9+eR+N77mHw4ME0atSIdevW0b9/fwCKiopo3rx5Sdvhw4cDP344CuS9995j7ty5PP744wAcPnyYL774AoBLLrmEBg0aANCxY0c+//xzvvvuO/r06UPbtm0BaNy48XH7Of/888O8R+RkqtHJY8L8TcfcBzymVSd2v/Mk6f9eQ//zGvLmm28yffp0Jk+e/JO29evXp23btrz22mtcc801OOdYu3YtnTt3rsxNEAF++kFoT3QTGl73N46csYv77ruPfv360alTJz788MOA7Ys/IEVFRVFYWBiwjnOO119/nfbt2x9TvmrVqp98wCqrj+P1I9VLjT5h/tXeQ8esx/zsXOrFXcInT/83F1xwAWPHjqVRo0Zltn/ppZeYMmUKnTt3plOnTsyZM+dkhywSUOkPQoUHdnOEWmTUiiM1NZVVq1aRl5dXkjwKCgpYv379cfs844wzOHDgQMl6SkoKTz/9NL4hcsjKyjpu+x49erBs2TJ27NgBUDJsFWw/UjWF5cjDzAYCTwFRwAvOufRSz8cA/wK6AruBa51zOd5z9wK/BoqAO5xz873yHOCAV17onEsOR6z+WjSMJbdUAqnffRjnD7iOlWn9SsrWrVtXsvyHP/yhZLlt27bMmzcv3GGJBK30B6GCvBz+s3Qau8x48OwzefbZZ6lVqxZ33HEH+/bto7CwkDvvvJNOnTqV2efFF19Meno6iYmJ3Hvvvdx///3ceeedJCQkcPToUdq2bctbb71VZvumTZsyefJkhg8fztGjRznrrLNYsGBB0P1I1WTF2b/CHZhFAZuB/sBOIAP4lXNug1+d3wAJzrlbzWwEMMw5d62ZdQReAboDLYCFwHnOuSIveSQ7574tbyzJyckuMzOz3LGXPtQHiI2O4rHh8cec8xCp6nqlL/7JByGAlg1jj/kgJBKIma0O9gN6OIatugNbnXPbnXM/ADOAIaXqDAFe9JZnAZeYmXnlM5xzR5xzO4CtXn+VYmhSSx4bHk/LhrEYvj80JQ6pjlJT2hMbHXVMWWx0FKkpOq8gJ0c4hq1aAl/6re8ELiirjnOu0Mz2AWd65R+Valv8n9sB75mZA/7hnPvpWWvAzG4GbgY4++yzgw5+aFJLJQup9orfw8WzrVo0jCU1pb3e23LSVOXZVr90zuWa2VnAAjPb6JxbVrqSl1Qmg2/YqrKDFKkq9EFIKlM4hq1ygdZ+6628soB1zKwW0ADfifMy2zrnin/+B3iTShzOEhGR4wtH8sgA2plZWzOrDYwA5paqMxcY5S1fDSz2vhI/FxhhZjFm1hZoB3xsZnXN7AwAM6sLDADWISIiVULIw1beOYzbgPn4pupOdc6tN7OH8F0vZS4wBZhuZluBPfgSDF69V4ENQCHwW2+mVTPgTd85dWoBLzvnNCdWRKSKCHmqblUS7FRdERGJ3FRdEREpw8SJEzn//PNp1KgR6enpJ27gycnJ4eWXXz6JkYWmKs+2EhGp9p555hkWLlxIq1atAj5fWFhIrVo//VdcnDyuu+66kx1ihSh5iIicJLfeeivbt29n0KBBjBkzhm3btjFp0iRGjx5NnTp1yMrKolevXgwZMoRx48YBYGYsW7aMtLQ0PvvsMxITExk1ahR33XVXhLfmWEoeIiInyXPPPce8efNYsmTJT67ftXPnTj744AOioqK44oor+Pvf/06vXr3Iz8+nTp06pKen8/jjj1fZ634peYiIhJH/fVVaNIzl+x+KAta75ppriIryXVKmV69e/O53v2PkyJEMHz68zCGuqkQnzEVEwqT4Yqu5ew/hgNy9h/ju+x94Z+2un9T1v3NiWloaL7zwAocOHaJXr15s3LixEqOuGB15iIiESen7qgA4B5OWbOU355Tdbtu2bcTHxxMfH09GRgYbN26kdevWx9xPparRkYeISJiUvq9Ksa/3BS4v9uSTTxIXF0dCQgLR0dEMGjSIhIQEoqKi6Ny5M0888cTJCDck+pKgiEiYVNf7quhLgiIiEVST7quicx4iImFSk+6rouQhIhJGNeW+Khq2EhGRoCl5iIhI0JQ8REQkaEoeIiISNCUPEREJmpKHiIgETclDRESCpuQhIiJBU/IQEZGghSV5mNlAM9tkZlvNLC3A8zFmNtN7fpWZtfF77l6vfJOZpZS3TxERiZyQk4eZRQF/BwYBHYFfmVnHUtV+DXznnDsXeAL4i9e2IzAC6AQMBJ4xs6hy9ikiIhESjiOP7sBW59x259wPwAxgSKk6Q4AXveVZwCVmZl75DOfcEefcDmCr1195+hQRkQgJR/JoCXzpt77TKwtYxzlXCOwDzjxO2/L0CYCZ3WxmmWaWmZeXF8JmiIhIeVX7E+bOucnOuWTnXHLTpk0jHY6ISI0QjuSRC7T2W2/llQWsY2a1gAbA7uO0LU+fIiISIeFIHhlAOzNra2a18Z0An1uqzlxglLd8NbDY+e5/OxcY4c3Gagu0Az4uZ58iVcLEiRM5//zzGTlyZKRDEak0Id8MyjlXaGa3AfOBKGCqc269mT0EZDrn5gJTgOlmthXYgy8Z4NV7FdgAFAK/dc4VAQTqM9RYRU6GZ555hoULF9KqVasT1i0sLKRWLd2DTao/8x0AnBqSk5NdZmZmpMOQGuTWW29l6tSptG/fntGjR7N8+XK2b9/O6aefzuTJk0lISOCBBx5g27ZtbN++nbPPPptXXnkl0mGLHMPMVjvnkoNpU+1PmItE0nPPPUeLFi1YsmQJOTk5JCUlsXbtWh599FFuuOGGknobNmxg4cKFShxyytDxs0gFzM7KZcL8TXy19xBf7zvMO2t3sWLFCl5//XUA+vXrx+7du9m/fz8AV155JbGxsZEMWSSslDxEgjQ7K5d73/iUQwVFABQedTz89gaKDhWU2aZu3bqVFZ5IpdCwlUiQJszfVJI4ih0uKOJQ4/N46aWXAFi6dClNmjShfv36kQhR5KTTkYdIkL7aeyhgeXS3/8fq1TNISEjg9NNP58UXXwxYT+RUoNlWIkHqlb6Y3AAJpGXDWFam9YtARCKh0WwrkUqQmtKe2OioY8pio6NITWkfoYhEKp+GrUSCNDTJd43O4tlWLRrGkprSvqRcpCZQ8hCpgKFJLatUsli6dCmPP/44b731VqRDkRpCw1YiIhI0JQ+RMDh48CCXX345nTt3Ji4ujpkzZ7J69WouuugiunbtSkpKCrt27QJg69atXHrppXTu3JkuXbqwbds2nHOkpqYSFxdHfHw8M2fOBHxHFH379uXqq6+mQ4cOjBw5kuJJLvPmzaNDhw506dKFN954I2LbLjWThq1EwmDevHm0aNGCt99+G4B9+/YxaNAg5syZQ9OmTZk5cybjx49n6tSpjBw5krS0NIYNG8bhw4c5evQob7zxBtnZ2axZs4Zvv/2Wbt260adPHwCysrJYv349LVq0oFevXqxcuZLk5GRuuukmFi9ezLnnnsu1114byc2XGkjJQyQM4uPj+f3vf88999zD4MGDadSoEevWraN///4AFBUV0bx5cw4cOEBubi7Dhg0DoE6dOgCsWLGCX/3qV0RFRdGsWTMuuugiMjIyqF+/Pt27dy+5Ym9iYiI5OTnUq1ePtm3b0q5dOwD+67/+i8mTJ0dgy6WmUvIQqSD/61u1aBjLw/98C9uZzX333Ue/fv3o1KkTH3744TFtDhw4EPTrxMTElCxHRUVRWFgYcuwiodI5D5EKKL6+Ve7eQzjg8y938uf526nX6WJSU1NZtWoVeXl5JcmjoKCA9evXc8YZZ9CqVStmz54NwJEjR/j+++/p3bs3M2fOpKioiLy8PJYtW0b37t3LfP0OHTqQk5PDtm3bAHS1Xql0Sh4iFVD6+lYFeTnsmDKOkZdfxIMPPshDDz3ErFmzuOeee+jcuTOJiYl88MEHAEyfPp2JEyeSkJBAz549+frrrxk2bBgJCQl07tyZfv368de//pWf/exnZb5+nTp1mDx5MpdffjldunThrLPOOunbLFVf8V0tGzVqRHp6eoX7MbP8E9bR5UlEgtc27W0C/eUYsCP98soORwTwHZGW966W/kpfnsTM8p1z9Y7XRkceUuPt3buXZ555Jqg2LRoGvjdHWeUiJ9utt97K9u3bGTRoEE888QS33XYbAKNHj+aOO+6gZ8+e/OIXv2DWrFkA5Ofnc8kll9ClSxeAjmY2JJjXU/KQGq8iyUPXt5Kqxv+ulo0aNTrmuV27fDcre+utt0hLSwN8Q59vvvkmn3zyCcBm4H/NzMr7epptJaek//mf/6Fx48bceeedAIwfP56zzjqLH374gVdffZUjR44wbNgwHnzwQdLS0ti2bRuJiYn079+fCRMmnLB/Xd9KqopAd7UsbejQoZx22ml07NiRb775BgDnHH/84x9ZtmwZwHn4Rl2bAV+X53WVPOSUNGbMGIYPH86dd97J0aNHmTFjBo8++iiLFi3i448/xjnHlVdeybJly0hPT2fdunVkZ2cH9RpV7fpWUvOUdVfLQfW/O6ae/3Tv4vPcL730Enl5eaxevZratWtvAJoAdcr72iENW5lZYzNbYGZbvJ+Nyqg3yquzxcxG+ZV3NbNPzWyrmU0sPmQyswfMLNfMsr3HZaHEKTVPmzZtOPPMM8nKyuK9994jKSmJjIyMkuUuXbqwceNGtmzZEulQRSqsrLtavrvup0cfpe3bt4+zzjqL6OhogDOAnwfz2qEeeaQBi5xz6WaW5q3f41/BzBoDfwKSAQesNrO5zrnvgGeBm4BVwDvAQOBdr+kTzrnHQ4xPahj/Q/jaTXtw/4RJ1C3KZ8yYMSxatIh7772XW2655Zg2OTk5kQlWJERl3dXyu+8LTth25MiRXHHFFcTHxwOcCWwM5rVDmqprZpuAvs65XWbWHFjqnGtfqs6vvDq3eOv/AJZ6jyXOuQ6l65nZA0B+sMlDU3VrttKH8K6ogK+n3kaj2ChyP9/OokWLuP/++1m0aBH16tUjNzeX6OhooqKi6NKlC59//nmEt0AkOOG6q2Uk7iTYzDlXfHz0Nb6TLaW1BL70W9/plbX0lkuXF7vNzNaa2dSyhsMAzOxmM8s0s8y8vLwKbYScGkofwltUNLXPjqfWuT2JiopiwIABXHfddVx44YXEx8dz9dVXc+DAAc4880x69epFXFwcqampEdwCkeBEctbfCYetzGwhEOirruP9V5xzzszC9Y3DZ4GH8Q1zPQz8LzAmUEXn3GRgMviOPML0+lINlT6Ed+4oR77aBN3SSsrGjRvHuHHjftL25ZdfPunxRcI///lPMjMzmTRpEn/729944YUXqFWrFk2bNmXq1Kn8/OdBDXNLFRPJWX8nTB7OuUvLes7MvjGz5n7DVv8JUC0X6Ou33grfkFWut+xfnuu95jd+r/E8oNujyQm1aBhbcgj/w7dfkDfrQWLPu5Cf/+LcCEdWeYqKioiKigr4XFJSEpmZmZx++uk8++yz3H333SX3DZHqK1Kz/kIdtpoLFM+eGgXMCVBnPjDAzBp5w08DgPnecNd+M+vhzbK6obi9l4iKDQPWhRin1AD+h/C1m5xNy1un0DLllmrzxb0JEyYwceJEAO666y769fONWS9evJiRI0fyyiuvEB8fT1xcHPfc8+O8lHr16vH73/+ezp078+GHHzJt2jTOO+88unfvzsqVK0vqXXzxxZx++ukA9OjRg507faPGI0aMKLkPCfi+kTxr1iyKiopITU2lW7duJCQk8I9//KOkzl/+8hfi4+Pp3LlzyZfOpGYJNXmkA/3NbAtwqbeOmSWb2QsAzrk9+IaeMrzHQ14ZwG+AF4CtwDZ+nGn1V28K71rgYuCuEOOUGmBoUkseGx5Py4axGL6Tho8Nj68238Xo3bs3y5cvByAzM5P8/HwKCgpYvnw55513Hvfccw+LFy8mOzubjIyMkivzHjx4kAsuuIA1a9Zwzjnn8Kc//YmVK1eyYsUKNmzYEPC1pkyZwqBBgwC49tprefXVVwH44YcfWLRoEZdffjlTpkyhQYMGZGRkkJGRwfPPP8+OHTt49913mTNnDqtWrWLNmjXcfffdlbB3pKoJaaquc243cEmA8kxgrN/6VGBqGfXiApRfH0pcUnNVty/u+U8t/tkZ0ez48GP2799PTEwMXbp0ITMzk+XLl3PFFVfQt29fmjZtCvimWS5btoyhQ4cSFRXFVVddBcCqVauOqXfttdeyefPmY17z//7v/8jMzOT9998HYNCgQYwbN44jR44wb948+vTpQ2xsLO+99x5r164tuRbSvn372LJlCwsXLuTGG28sOYpp3LhxpewrqVr0DXORCCk9tXjXgQIORDfid39+kp49e5KQkMCSJUvYunUrbdq0YfXq1QH7qVOnTpnnOUpbuHAhjzzyCO+//37Jt47r1KlD3759mT9/PjNnzmTEiBGA75vITz/9NCkpKcf0MX/+/Cb+L44AAA98SURBVIpuspxCdGFEkQgJ9O3g6JYdmT757/Tp04fevXvz3HPPkZSURPfu3Xn//ff59ttvKSoq4pVXXuGiiy76SZ8XXHAB77//Prt376agoIDXXnut5LmsrCxuueUW5s6d+5P7f1x77bVMmzaN5cuXM3DgQABSUlJ49tlnKSjwfeFs8+bNHDx4kP79+zNt2jS+//57APbs2YPUPDryEImQQN8OjmnViX0fvsqFF15I3bp1qVOnDr1796Z58+akp6dz8cUX45zj8ssvZ8iQn15Bu3nz5jzwwANceOGFNGzYkMTExJLnUlNTyc/P55prrgHg7LPPZu7cuQAMGDCA66+/niFDhlC7dm0Axo4dS05ODl26dME5R9OmTZk9ezYDBw4kOzub5ORkateuzWWXXcajjz56MnaRVGG6GZRIhITr28EioYrEN8xFpIJ0TxCpzjRsJRIhuieIVGdKHiInWc+ePfnggw8CPlfdphaLFNOwlchJVlbiEKnOlDxETrJ69eoBvvtI9+nTh8TEROLi4kq+TS5SHWnYSqSSvPzyy6SkpDB+/HiKiopKvichUh0peYicBP6XHTlUUMTsrFy6devGmDFjKCgoYOjQocd8B0OkutGwlUiYFV92JHfvIRzgHNz7xqfsOeMcli1bRsuWLRk9ejT/+te/Ih2qSIUpeYiEWaDLjhwqKOLhGcto1qwZN910E2PHjuWTTz6JUIQiodOwlUiYBbrsCMAX6zLo3PkRoqOjqVevno48pFpT8hAJM/87GgKc/TvfJc3P6z2YlW//LVJhiYSVhq1EwkyXHZGaQEceImGmy45ITaDkIXIS6LIjcqrTsJXUOHv37uWZZ54BYOnSpQwePDjCEYlUP0oeUuP4Jw8RqRglD6lx0tLS2LZtG4mJiSV317v66qvp0KEDI0eOpPgGaatXr+aiiy6ia9eupKSksGvXrghHLlJ1hJQ8zKyxmS0wsy3ez0Zl1Bvl1dliZqP8yh8xsy/NLL9U/Rgzm2lmW81slZm1CSVOEX/p6emcc845ZGdnM2HCBLKysnjyySfZsGED27dvZ+XKlRQUFHD77bcza9YsVq9ezZgxYxg/fnykQxepMkI9YZ4GLHLOpZtZmrd+j38FM2sM/AlIBhyw2szmOue+A/4NTAK2lOr318B3zrlzzWwE8Bfg2hBjFQmoe/futGrVCoDExERycnJo2LAh69ato3///gAUFRXRvHnzSIYpUqWEmjyGAH295ReBpZRKHkAKsMA5twfAzBYAA4FXnHMfeWWB+n3AW54FTDIzc6fSDdel0hVfrPDzz3PY8+1BZmfl0hCIiYkpqRMVFUVhYSHOOTp16sSHH34YuYBFqrBQz3k0c84VDwR/DTQLUKcl8KXf+k6v7HhK2jjnCoF9wJmBKprZzWaWaWaZeXl5wcQuNYj/xQqtdiw/HDrIvW98yootgd8z7du3Jy8vryR5FBQUsH79+soMWaRKO+GRh5ktBH4W4KljBoCdc87MKv3IwDk3GZgMkJycrCMTCcj/YoVRsfWJadmRbc/dQnpMLH0Tz/1J/dq1azNr1izuuOMO9u3bR2FhIXfeeSedOnWq7NBFqqQTJg/n3KVlPWdm35hZc+fcLjNrDvwnQLVcfhzaAmiFb3jreHKB1sBOM6sFNAB2nyhWkbKUvlhh0ytTATDgrfTLS8onTZpUspyYmMiyZcsqJT6R6ibUYau5QPHsqVHAnAB15gMDzKyRNxtrgFdW3n6vBhbrfIeEokXD2KDKReT4Qk0e6UB/M9sCXOqtY2bJZvYCgHei/GEgw3s85Hfy/K9mthM43cx2mtkDXr9TgDPNbCvwO3yzuEQqTBcrFAkvO5U+0CcnJ7vMzMxIhyFVlP+tYXWxQpEfmdlq51xyMG10YUSpMXSxQpHw0eVJREQkaEoeIiISNCUPEREJmpKHiIgETclDRESCpuQhIiJBU/IQEZGgKXmIiEjQlDxERCRoSh4iIhI0JQ8REQmakoeIiARNyUNERIKm5CEiIkFT8hARkaApeYiISNCUPEREJGhKHiIiEjQlDxERCZqSh4iIBC2k5GFmjc1sgZlt8X42KqPeKK/OFjMb5Vf+iJl9aWb5peqPNrM8M8v2HmNDiVNERMIr1COPNGCRc64dsMhbP4aZNQb+BFwAdAf+5Jdk/u2VBTLTOZfoPV4IMU4REQmjUJPHEOBFb/lFYGiAOinAAufcHufcd8ACYCCAc+4j59yuEGMQEZFKFmryaOb3z/9roFmAOi2BL/3Wd3plJ3KVma01s1lm1rqsSmZ2s5llmllmXl5euQMXEZGKO2HyMLOFZrYuwGOIfz3nnANcmOL6N9DGOZeA70jlxbIqOucmO+eSnXPJTZs2DdPLi4jI8ZwweTjnLnXOxQV4zAG+MbPmAN7P/wToIhfwP3Jo5ZUd7zV3O+eOeKsvAF3LszFSNe3du5dnnnkGgKVLlzJ48OCA9caOHcuGDRsqMzQRqaBQh63mAsWzp0YBcwLUmQ8MMLNG3onyAV5ZmYoTkudK4LMQ45QI8k8ex/PCCy/QsWPHSohIREIVavJIB/qb2RbgUm8dM0s2sxcAnHN7gIeBDO/xkFeGmf3VzHYCp5vZTjN7wOv3DjNbb2ZrgDuA0SHGKRGUlpbGtm3bSExMJDU1lfz8fK6++mo6dOjAyJEj8Y14Qt++fcnMzKSoqIjRo0cTFxdHfHw8TzzxRIS3QERKqxVKY+fcbuCSAOWZwFi/9anA1AD17gbuDlB+L3BvKLFJ1ZGens66devIzs5m6dKlDBkyhPXr19OiRQt69erFypUr+eUvf1lSPzs7m9zcXNatWwf4jlxEpGoJKXmIlGV2Vi4T5m/iq72HaOz2sf9wYclz3bt3p1WrVgAkJiaSk5NzTPL4xS9+wfbt27n99tu5/PLLGTBgQKXHLyLHp8uTSNjNzsrl3jc+JXfvIRzwzf7DfLP/MLOzfPMkYmJiSupGRUVRWFh4TPtGjRqxZs0a+vbty3PPPcfYsbrAgEhVo+QhYTdh/iYOFRSVrFvtWIqOfM+E+ZvK1f7bb7/l6NGjXHXVVfz5z3/mk08+OVmhikgFadhKwu6rvYeOWY+KrU9My45k/O+NpLY5i2bNAn2X9Ee5ubnceOONHD16FIDHHnvspMUqIhVjxTNdTgXJyckuMzMz0mHUeL3SF5NbKoEAtGwYy8q0fhGISESOx8xWO+eSg2mjYSsJu9SU9sRGRx1TFhsdRWpK+whFJCLhpmErCbuhSb5LlxXPtmrRMJbUlPYl5SJS/Sl5yEkxNKmlkoXIKUzDViIiEjQlDxERCZqSh4iIBE3JQ0REgqbkISIiQTulviRoZnnA58ep0gT4tpLCCUZVjQsUW0VV1diqalyg2CoqHLH93DkX1K1YT6nkcSJmlhnstygrQ1WNCxRbRVXV2KpqXKDYKipSsWnYSkREgqbkISIiQatpyWNypAMoQ1WNCxRbRVXV2KpqXKDYKioisdWocx4iIhIeNe3IQ0REwkDJQ0REglbtk4eZNTazBWa2xfvZqIx6o7w6W8xslFd2upm9bWYbzWy9maX71Y8xs5lmttXMVplZm8qMzSt/xMy+NLP8UvVHm1memWV7j6Bv8n0SY6sK+62rmX3qxTDRzMwrf8DMcv3222XljGegmW3y+ksL8HyZ22xm93rlm8wspbx9ltdJii3H23/ZZlbhu6tVNDYzO9PMlphZvplNKtUm4O+2CsS11Ouz+L11VrBxhRhbfzNb7e2b1WbWz69NyPssIOdctX4AfwXSvOU04C8B6jQGtns/G3nLjYDTgYu9OrWB5cAgb/03wHPe8ghgZmXG5j3XA2gO5JdqMxqYFKn9doLYqsJ++9iLz4B3/X6nDwB/CDKWKGAb8AvvPbIG6FiebQY6evVjgLZeP1Hl6TNSsXnP5QBNQnx/hRJbXeCXwK2l3+dl/W6rQFxLgeQI7rMkoIW3HAfkhmuflfWo9kcewBDgRW/5RWBogDopwALn3B7n3HfAAmCgc+5759wSAOfcD8AnQKsA/c4CLqlAxq5wbF5MHznndgX5mpGOLaL7zcyaA/W9+BzwrzLal1d3YKtzbrv3HpnhxVdWvP7bPASY4Zw74pzbAWz1+itPn5GKLVwqHJtz7qBzbgVw2L9ymH63YY8rjEKJLcs595VXvh6I9Y5Swv33UOJUSB7N/P6JfQ00C1CnJfCl3/pOr6yEmTUErgAWlW7jnCsE9gFnRiK2MlxlZmvNbJaZtQ4yrpMZW6T3W0tvuayYb/P221QrYzisnK8TsE6pbT5ejBX5nVdGbAAOeM8b/ri5AnGFGtvx+jze7zZScRWb5g1Z3V/BoaFwxXYV8Ilz7gjh2WcBVYs7CZrZQuBnAZ4a77/inHNmFvTcYzOrBbwCTHTOba9KsZXh38ArzrkjZnYLvk8i/UpXilBs5RKh2J4FHsb3z/Fh4H+BMWHq+1TyS+dcrjduv8DMNjrnlkU6qCpupLfPzgBeB67H9ym/UplZJ+AvwICT/VrVInk45y4t6zkz+8bMmjvndnmHaP8JUC0X6Ou33grfGGWxycAW59yTpdq0BnZ6yaUBsDsCsf2Ec84/jhfwnSMIVK/SYyPy+y2XH4cei8tzvdf8xu81ngfeOsG2+G/PT/oLUKf0Nh+v7Yn6LI+TEptzrvjnf8zsTXzDKcEmj1BiO16fAX+3EY7Lf58dMLOX8e2zYJNHSLGZWSvgTeAG59w2v/qh7rOAToVhq7lA8UybUcCcAHXmAwPMrJE3VDHAK8PM/ozvF3Dncfq9GljsjRlWWmxl8f6hFrsS+CzIuE5abER4v3nDXfvNrIc3dHBDcftS+20YsK4csWQA7cysrZnVxneScu5x4vXf5rnACG/suS3QDt/Jy/L0WR5hj83M6nqfnjGzuvj2a3n2UzhjC+h4v9tIxmVmtcysibccDQymkveZN+z+Nr6JJiuLK4dpnwUWjrPukXzgG+9bBGwBFgKNvfJk4AW/emPwnRTcCtzolbXCN4TxGZDtPcZ6z9UBXvPqfwz8ojJj88r/im+M8qj38wGv/DF8J8XWAEuADlUotqqw35Lx/fFuAybx45UUpgOfAmvx/RE2L2c8lwGbvf7Ge2UPAVeeaJvxDcNtAzbhN8slUJ8VfP+HNTZ8M33WeI/1EYwtB9gD5Hvvr47H+91GMi58s7BWe++r9cBTeDPXKis24D7gID/+H8sGzgrXPgv00OVJREQkaKfCsJWIiFQyJQ8REQmakoeIiARNyUNERIKm5CEiIkFT8hARkaApeYiISND+P4DSrshOxdV3AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EUToYfONCzp2"
      },
      "source": [
        "###**12.6 Load Google’s Word2Vec Embedding**\n",
        "\n",
        "Training your own word vectors may be the best approach for a given NLP problem. But it\n",
        "can take a long time, a fast computer with a lot of RAM and disk space, and perhaps some\n",
        "expertise in finessing the input data and training algorithm. An alternative is to simply use an\n",
        "existing pre-trained word embedding. Along with the paper and code for Word2Vec, Google\n",
        "also published a pre-trained Word2Vec model on the Word2Vec Google Code Project.\n",
        "\n",
        "\n",
        "A pre-trained model is nothing more than a file containing tokens and their associated word\n",
        "vectors. The pre-trained Google Word2Vec model was trained on Google news data (about 100\n",
        "billion words); it contains 3 million words and phrases and was fit using 300-dimensional word\n",
        "vectors. It is a 1.53 Gigabyte file. You can download it from here:\n",
        "\n",
        " GoogleNews-vectors-negative300.bin.gz.\n",
        "https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing\n",
        "\n",
        "We can put all of this together as follows."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7xnpxdIH-iL8"
      },
      "source": [
        "from gensim.models import KeyedVectors\n",
        "# load the google word2vec model\n",
        "filename = 'GoogleNews-vectors-negative300.bin'\n",
        "model = KeyedVectors.load_word2vec_format(filename, binary=True)\n",
        "# calculate: (king - man) + woman = ?\n",
        "result = model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oUmQBdraEtTM"
      },
      "source": [
        "##**12.7 Load Stanford’s GloVe Embedding**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z3Nw527xEya7"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DnGF_Ft2vQxn"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8wUGvQk-vRV0"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C_3llBbqvSDz"
      },
      "source": [
        "#**Chapter 13 How to Learn and Load Word Embeddings in Keras**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-d-FiKVXvzfw"
      },
      "source": [
        "Word embeddings provide a dense representation of words and their relative meanings. They are\n",
        "an improvement over sparse representations used in simpler bag of word model representations.\n",
        "Word embeddings can be learned from text data and reused among projects. They can also be\n",
        "learned as part of fitting a neural network on text data. In this tutorial, you will discover how\n",
        "to use word embeddings for deep learning in Python with Keras. After completing this tutorial,\n",
        "you will know:\n",
        "\n",
        "* About word embeddings and that Keras supports word embeddings via the Embedding\n",
        "layer.\n",
        "* How to learn a word embedding while fitting a neural network.\n",
        "* How to use a pre-trained word embedding in a neural network.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lfBaA2xmv_bD"
      },
      "source": [
        "##**13.1 Tutorial Overview**\n",
        "This tutorial is divided into the following parts:\n",
        "1. Word Embedding\n",
        "2. Keras Embedding Layer\n",
        "3. Example of Learning an Embedding\n",
        "4. Example of Using Pre-Trained GloVe Embedding\n",
        "5. Tips for Cleaning Text for Word Embedding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0IysEO41wMXD"
      },
      "source": [
        "##**13.2 Word Embedding**\n",
        "\n",
        "The position of a word in the learned vector space is referred to as its embedding.\n",
        "\n",
        "\n",
        "Two popular examples of methods of learning word embeddings from text include:\n",
        "* Word2Vec.\n",
        "* GloVe."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDKO2Pf3xeoa"
      },
      "source": [
        "##**13.3 Keras Embedding Layer**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kJjPtAMs2inW"
      },
      "source": [
        "The Embedding layer is initialized with random weights and will learn an embedding for all\n",
        "of the words in the training dataset. It is a flexible layer that can be used in a variety of ways,\n",
        "such as:\n",
        "\n",
        "* It can be used alone to learn a word embedding that can be saved and used in another\n",
        "model later.\n",
        "\n",
        "* It can be used as part of a deep learning model where the embedding is learned along\n",
        "with the model itself.\n",
        "\n",
        "* It can be used to load a pre-trained word embedding model, a type of transfer learning.\n",
        "\n",
        "\n",
        "The Embedding layer is defined as the first hidden layer of a network. It must specify 3\n",
        "arguments:\n",
        "\n",
        "* input dim: This is the size of the vocabulary in the text data. For example, if your data\n",
        "is integer encoded to values between 0-10, then the size of the vocabulary would be 11\n",
        "words.\n",
        "\n",
        "* output dim: This is the size of the vector space in which words will be embedded. It\n",
        "defines the size of the output vectors from this layer for each word. For example, it could\n",
        "be 32 or 100 or even larger. Test different values for your problem.\n",
        "\n",
        "* input length: This is the length of input sequences, as you would define for any input\n",
        "layer of a Keras model. For example, if all of your input documents are comprised of 1000\n",
        "words, this would be 1000.\n",
        "\n",
        "\n",
        "For example, below we define an Embedding layer with a vocabulary of 200 (e.g. integer\n",
        "encoded words from 0 to 199, inclusive), a vector space of 32 dimensions in which words will be\n",
        "embedded, and input documents that have 50 words each."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rp_-By6Yvg8S"
      },
      "source": [
        "#e = Embedding(200, 32, input_length=50)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gpVvTiX04D6s"
      },
      "source": [
        "##**13.4 Example of Learning an Embedding**\n",
        "\n",
        "\n",
        "First, we will define the documents and their class labels.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iesf2kLB30ye"
      },
      "source": [
        "# define documents\n",
        "docs = ['Well done!',\n",
        "        'Good work',\n",
        "        'Great effort',\n",
        "        'nice work',\n",
        "        'Excellent!',\n",
        "        'Weak',\n",
        "        'Poor effort!',\n",
        "        'not good',\n",
        "        'poor work',\n",
        "        'Could have done better.']\n",
        "# define class labels\n",
        "labels = [1,1,1,1,1,0,0,0,0,0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5iH4AvXI5OQC"
      },
      "source": [
        "Next, we can integer encode each document. This means that as input the Embedding layer\n",
        "will have sequences of integers. We could experiment with other more sophisticated bag of word\n",
        "model encoding like **counts** or **TF-IDF**. Keras provides the one **hot()** function that creates a\n",
        "hash of each word as an efficient integer encoding. We will estimate the vocabulary size of 50,\n",
        "which is much larger than needed to reduce the probability of collisions from the hash function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OJAyrdAq4kUd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "15134490-b1d9-4796-bf62-5bc27e2cd144"
      },
      "source": [
        "# integer encode the documents\n",
        "from keras.preprocessing.text import one_hot\n",
        "vocab_size = 50\n",
        "encoded_docs = [one_hot(d, vocab_size) for d in docs]\n",
        "print(encoded_docs)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[36, 23], [32, 44], [17, 45], [11, 44], [21], [14], [29, 45], [14, 32], [29, 44], [2, 48, 23, 22]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KI8tPBQFB1Fc"
      },
      "source": [
        "The sequences have different lengths and Keras prefers inputs to be vectorized and all inputs\n",
        "to have the same length. We will **pad** all input sequences to have the length of 4. Again, we can do this with a built in Keras function, in this case the **pad_sequences()** function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yubcJP6U5jHX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "e8a9648d-faa7-4024-d0d4-577bd2402c2d"
      },
      "source": [
        "# pad documents to a max length of 4 words\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "max_length = 4\n",
        "padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')\n",
        "print(padded_docs), type(padded_docs)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[36 23  0  0]\n",
            " [32 44  0  0]\n",
            " [17 45  0  0]\n",
            " [11 44  0  0]\n",
            " [21  0  0  0]\n",
            " [14  0  0  0]\n",
            " [29 45  0  0]\n",
            " [14 32  0  0]\n",
            " [29 44  0  0]\n",
            " [ 2 48 23 22]]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(None, numpy.ndarray)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6tFVb597DDYa"
      },
      "source": [
        "We are now ready to define our **Embedding layer** as part of our neural network model.\n",
        "The Embedding layer has a vocabulary of **50** and an input length of 4. We will choose a\n",
        "small embedding space of 8 dimensions. The model is a simple binary classification model.\n",
        "Importantly, the output from the Embedding layer will be 4 vectors of 8 dimensions each, one\n",
        "for each word. We flatten this to a one 32-element vector to pass on to the Dense output layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TcGoedMnDVYZ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "outputId": "53ee9ba0-6223-4181-8270-502ceb2e3214"
      },
      "source": [
        "# define the model\n",
        "import numpy as np\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import Flatten\n",
        "from keras.layers.embeddings import Embedding\n",
        "model = Sequential()\n",
        "model.add(Embedding(vocab_size, 8, input_length=max_length))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "# compile the model\n",
        "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])\n",
        "# summarize the model\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_2 (Embedding)      (None, 4, 8)              400       \n",
            "_________________________________________________________________\n",
            "flatten_2 (Flatten)          (None, 32)                0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 1)                 33        \n",
            "=================================================================\n",
            "Total params: 433\n",
            "Trainable params: 433\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ar8ow3kCEYfU"
      },
      "source": [
        "Finally, we can fit and evaluate the classification model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zTMW8doqEK8b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e82ce487-704d-494a-cba5-fa1949d6f167"
      },
      "source": [
        "# fit the model\n",
        "labels = np.array(labels)\n",
        "model.fit(padded_docs, labels, epochs=50, verbose=0)\n",
        "# evaluate the model\n",
        "loss, accuracy = model.evaluate(padded_docs, labels, verbose=0)\n",
        "print('Accuracy: %f' % (accuracy*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 89.999998\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kC74WW-hEop2"
      },
      "source": [
        "The complete code listing is provided below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LUQ2tZCNEn5n",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 459
        },
        "outputId": "77e50779-858a-4c04-d460-46630817e7a1"
      },
      "source": [
        "from keras.preprocessing.text import one_hot\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import Flatten\n",
        "from keras.layers.embeddings import Embedding\n",
        "# define documents\n",
        "docs = ['Well done!',\n",
        "        'Good work',\n",
        "        'Great effort',\n",
        "        'nice work',\n",
        "        'Excellent!',\n",
        "        'Weak',\n",
        "        'Poor effort!',\n",
        "        'not good',\n",
        "        'poor work',\n",
        "        'Could have done better.']\n",
        "# define class labels\n",
        "labels = np.array([1,1,1,1,1,0,0,0,0,0])\n",
        "# integer encode the documents\n",
        "vocab_size = 50\n",
        "encoded_docs = [one_hot(d, vocab_size) for d in docs]\n",
        "print(encoded_docs)\n",
        "# pad documents to a max length of 4 words\n",
        "max_length = 4\n",
        "padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')\n",
        "print(padded_docs)\n",
        "# define the model\n",
        "model = Sequential()\n",
        "model.add(Embedding(vocab_size, 8, input_length=max_length))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "# compile the model\n",
        "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])\n",
        "# summarize the model\n",
        "model.summary()\n",
        "# fit the model\n",
        "model.fit(padded_docs, labels, epochs=50, verbose=0)\n",
        "# evaluate the model\n",
        "loss, accuracy = model.evaluate(padded_docs, labels, verbose=0)\n",
        "print('Accuracy: %f' % (accuracy*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[36, 23], [32, 44], [17, 45], [11, 44], [21], [14], [29, 45], [14, 32], [29, 44], [2, 48, 23, 22]]\n",
            "[[36 23  0  0]\n",
            " [32 44  0  0]\n",
            " [17 45  0  0]\n",
            " [11 44  0  0]\n",
            " [21  0  0  0]\n",
            " [14  0  0  0]\n",
            " [29 45  0  0]\n",
            " [14 32  0  0]\n",
            " [29 44  0  0]\n",
            " [ 2 48 23 22]]\n",
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_3 (Embedding)      (None, 4, 8)              400       \n",
            "_________________________________________________________________\n",
            "flatten_3 (Flatten)          (None, 32)                0         \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 1)                 33        \n",
            "=================================================================\n",
            "Total params: 433\n",
            "Trainable params: 433\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Accuracy: 89.999998\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mlH06caGGOh6"
      },
      "source": [
        "##**13.5 Example of Using Pre-Trained GloVe Embedding**\n",
        "\n",
        "\n",
        "The Keras Embedding layer can also use a word embedding learned elsewhere.  It is common\n",
        "in the field of Natural Language Processing to learn, save, and make freely available word\n",
        "embeddings. For example, the researchers behind GloVe method provide a suite of pre-trained\n",
        "word embeddings on their website released under a public domain license.\n",
        "\n",
        "The smallest package of embeddings is 822 Megabytes, called glove.6B.zip. It was trained\n",
        "on a dataset of one billion tokens (words) with a vocabulary of 400 thousand words. There\n",
        "are a few different embedding vector sizes, including 50, 100, 200 and 300 dimensions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mg5AOe_wvdAK"
      },
      "source": [
        "Keras provides a **Tokenizer** class that can be fit on the\n",
        "training data, can convert text to sequences consistently by calling the **texts_to_sequences()**\n",
        "method on the **Tokenizer** class, and provides access to the dictionary mapping of words to\n",
        "integers in a **word_index** attribute.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UDDGY3h4F_ls"
      },
      "source": [
        "# define documents\n",
        "docs = ['Well done!',\n",
        "'Good work',\n",
        "'Great effort',\n",
        "'nice work',\n",
        "'Excellent!',\n",
        "'Weak',\n",
        "'Poor effort!',\n",
        "'not good',\n",
        "'poor work',\n",
        "'Could have done better.']\n",
        "# define class labels\n",
        "labels = [1,1,1,1,1,0,0,0,0,0]\n",
        "# prepare tokenizer\n",
        "t = Tokenizer()\n",
        "t.fit_on_texts(docs)\n",
        "vocab_size = len(t.word_index) + 1\n",
        "# integer encode the documents\n",
        "encoded_docs = t.texts_to_sequences(docs)\n",
        "print(encoded_docs)\n",
        "# pad documents to a max length of 4 words\n",
        "max_length = 4\n",
        "padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')\n",
        "print(padded_docs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6FnENZuPwEEy"
      },
      "source": [
        "Next, we need to load the entire **GloVe** word embedding file into memory as a dictionary of\n",
        "word to embedding array."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7aVLCxuQvKOx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 231
        },
        "outputId": "6f988c02-787d-407c-9bb6-853d2dcdb2d1"
      },
      "source": [
        "# load the whole embedding into memory\n",
        "embeddings_index = dict()\n",
        "f = open('glove.6B.100d.txt')\n",
        "for line in f:\n",
        "    values = line.split()\n",
        "    word = values[0]\n",
        "    coefs = asarray(values[1:], dtype='float32')\n",
        "    embeddings_index[word] = coefs\n",
        "f.close()\n",
        "print('Loaded %s word vectors.' % len(embeddings_index))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "FileNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-23-60077735686a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# load the whole embedding into memory\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0membeddings_index\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'glove.6B.100d.txt'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0mvalues\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'glove.6B.100d.txt'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vgs9expux2Px"
      },
      "source": [
        "We can do that by enumerating all unique words in the **Tokenizer.word** index and\n",
        "locating the embedding weight vector from the loaded GloVe embedding. The result is a matrix\n",
        "of weights only for words we will see during training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9Y2AHTf2wacK"
      },
      "source": [
        "# create a weight matrix for words in training docs\n",
        "embedding_matrix = zeros((vocab_size, 100))\n",
        "for word, i in t.word_index.items():\n",
        "embedding_vector = embeddings_index.get(word)\n",
        "if embedding_vector is not None:\n",
        "embedding_matrix[i] = embedding_vector"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oLr5dTbwy4j3"
      },
      "source": [
        "We chose the **100-dimensional** version, therefore the Embedding layer must be defined with output dim set to **100**. Finally, we do not want to update the learned word weights in this model, therefore we will set the trainable attribute for the model to be **False**.\n",
        "\n",
        "The complete worked example is listed below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sP1wwmpEzE-3"
      },
      "source": [
        "from numpy import asarray\n",
        "from numpy import zeros\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import Flatten\n",
        "from keras.layers import Embedding\n",
        "# define documents\n",
        "docs = ['Well done!',\n",
        "        'Good work',\n",
        "        'Great effort',\n",
        "        'nice work',\n",
        "        'Excellent!',\n",
        "        'Weak',\n",
        "        'Poor effort!',\n",
        "        'not good',\n",
        "        'poor work',\n",
        "        'Could have done better.']\n",
        "# define class labels\n",
        "labels = [1,1,1,1,1,0,0,0,0,0]\n",
        "# prepare tokenizer\n",
        "t = Tokenizer()\n",
        "t.fit_on_texts(docs)\n",
        "vocab_size = len(t.word_index) + 1\n",
        "# integer encode the documents\n",
        "encoded_docs = t.texts_to_sequences(docs)\n",
        "print(encoded_docs)\n",
        "# pad documents to a max length of 4 words\n",
        "max_length = 4\n",
        "padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')\n",
        "print(padded_docs)\n",
        "# load the whole embedding into memory\n",
        "embeddings_index = dict()\n",
        "f = open('glove.6B.100d.txt', mode='rt', encoding='utf-8')\n",
        "for line in f:\n",
        "values = line.split()\n",
        "word = values[0]\n",
        "coefs = asarray(values[1:], dtype='float32')\n",
        "embeddings_index[word] = coefs\n",
        "f.close()\n",
        "print('Loaded %s word vectors.' % len(embeddings_index))\n",
        "# create a weight matrix for words in training docs\n",
        "embedding_matrix = zeros((vocab_size, 100))\n",
        "for word, i in t.word_index.items():\n",
        "embedding_vector = embeddings_index.get(word)\n",
        "if embedding_vector is not None:\n",
        "embedding_matrix[i] = embedding_vector\n",
        "# define model\n",
        "model = Sequential()\n",
        "e = Embedding(vocab_size, 100, weights=[embedding_matrix], input_length=4, trainable=False)\n",
        "model.add(e)\n",
        "model.add(Flatten()\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "# compile the model\n",
        "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])\n",
        "# summarize the model\n",
        "model.summary()\n",
        "# fit the model\n",
        "model.fit(padded_docs, labels, epochs=50, verbose=0)\n",
        "# evaluate the model\n",
        "loss, accuracy = model.evaluate(padded_docs, labels, verbose=0)\n",
        "print('Accuracy: %f' % (accuracy*100))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CmU78w0d0zZ_"
      },
      "source": [
        "#**Part VI Text Classification**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dBcC7MMi4wYP"
      },
      "source": [
        "#**Chapter 14: Neural Models for Document Classification**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HBnHPDjTmO6J"
      },
      "source": [
        "##**14.1 Overview**\n",
        "This tutorial is divided into the following parts:\n",
        "1. Word Embeddings + CNN = Text Classification\n",
        "2. Use a Single Layer CNN Architecture\n",
        "3. Dial in CNN Hyperparameters\n",
        "4. Consider Character-Level CNNs\n",
        "5. Consider Deeper CNNs for Classification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPHfoYJmsW4N"
      },
      "source": [
        "#***Chapter 15 Project: Develop an Embedding + CNN Model for Sentiment Analysis***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GPAuIkGXsqkp"
      },
      "source": [
        "##**15.1 Tutorial Overview**\n",
        "This tutorial is divided into the following parts:\n",
        "1. Movie Review Dataset\n",
        "2. Data Preparation\n",
        "3. Train CNN With Embedding Layer\n",
        "4. Evaluate Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ioLBME3DtVsn"
      },
      "source": [
        "##**15.2 Movie Review Dataset**\n",
        "\n",
        "* Movie Review Polarity Dataset (review polarity.tar.gz, 3MB).\n",
        "http://www.cs.cornell.edu/people/pabo/movie-review-data/review_polarity.tar.\n",
        "gz\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jv3jvWTQtk1R"
      },
      "source": [
        "##**15.3 Data Preparation**\n",
        "Note: The preparation of the movie review dataset was first described in Chapter 9. In this\n",
        "section, we will look at 3 things:\n",
        "1. Separation of data into training and test sets.\n",
        "2. Loading and cleaning the data to remove punctuation and numbers.\n",
        "3. Defining a vocabulary of preferred words."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kcqoLdghuCa2"
      },
      "source": [
        "##15.4 Train CNN With Embedding Layer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j5NhRNCOtkXn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "245eec05-30b1-4320-93a6-6b718f0ebf06"
      },
      "source": [
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from numpy import array\n",
        "from keras import models\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from keras.utils.vis_utils import plot_model\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import Flatten\n",
        "from keras.layers import Embedding\n",
        "from keras.layers.convolutional import Conv1D\n",
        "from keras.layers.convolutional import MaxPooling1D\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc, vocab):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # filter out tokens not in vocab\n",
        "    tokens = [w for w in tokens if w in vocab]\n",
        "    tokens = ' '.join(tokens)\n",
        "    return tokens\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, vocab, is_train):\n",
        "    documents = list()\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "        # skip any reviews in the test set\n",
        "        if is_train and filename.startswith('cv9'):\n",
        "            continue\n",
        "        if not is_train and not filename.startswith('cv9'):\n",
        "            continue\n",
        "        # create the full path of the file to open\n",
        "        path = directory + '/' + filename\n",
        "        # load the doc\n",
        "        doc = load_doc(path)\n",
        "        # clean doc\n",
        "        tokens = clean_doc(doc, vocab)\n",
        "        # add to list\n",
        "        documents.append(tokens)\n",
        "    return documents\n",
        "\n",
        "# load and clean a dataset\n",
        "def load_clean_dataset(vocab, is_train):\n",
        "    # load documents\n",
        "    neg = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', vocab, is_train)\n",
        "    pos = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', vocab, is_train)\n",
        "    docs = neg + pos\n",
        "    # prepare labels\n",
        "    labels = array([0 for _ in range(len(neg))] + [1 for _ in range(len(pos))])\n",
        "    return docs, labels\n",
        "\n",
        "# fit a tokenizer\n",
        "def create_tokenizer(lines):\n",
        "    tokenizer = Tokenizer()\n",
        "    tokenizer.fit_on_texts(lines)\n",
        "    return tokenizer\n",
        "\n",
        "# integer encode and pad documents\n",
        "def encode_docs(tokenizer, max_length, docs):\n",
        "    # integer encode\n",
        "    encoded = tokenizer.texts_to_sequences(docs)\n",
        "    # pad sequences\n",
        "    padded = pad_sequences(encoded, maxlen=max_length, padding='post')\n",
        "    return padded\n",
        "\n",
        "# define the model\n",
        "def define_model(vocab_size, max_length):\n",
        "    model = Sequential()\n",
        "    model.add(Embedding(vocab_size, 100, input_length=max_length))\n",
        "    model.add(Conv1D(filters=32, kernel_size=8, activation='relu'))\n",
        "    model.add(MaxPooling1D(pool_size=2))\n",
        "    model.add(Flatten())\n",
        "    model.add(Dense(10, activation='relu'))\n",
        "    model.add(Dense(1, activation='sigmoid'))\n",
        "    # compile network\n",
        "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "    # summarize defined model\n",
        "    model.summary()\n",
        "    plot_model(model, to_file='model.png', show_shapes=True)\n",
        "    return model\n",
        "\n",
        "# load the vocabulary\n",
        "vocab_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/vocab.txt'\n",
        "vocab = load_doc(vocab_filename)\n",
        "vocab = set(vocab.split())\n",
        "# load training data\n",
        "train_docs, ytrain = load_clean_dataset(vocab, True)\n",
        "# create the tokenizer\n",
        "tokenizer = create_tokenizer(train_docs)\n",
        "# define vocabulary size\n",
        "vocab_size = len(tokenizer.word_index) + 1\n",
        "print('Vocabulary size: %d' % vocab_size)\n",
        "# calculate the maximum sequence length\n",
        "max_length = max([len(s.split()) for s in train_docs])\n",
        "print('Maximum length: %d' % max_length)\n",
        "# encode data\n",
        "Xtrain = encode_docs(tokenizer, max_length, train_docs)\n",
        "# define model\n",
        "model = define_model(vocab_size, max_length)\n",
        "# fit network\n",
        "model.fit(Xtrain, ytrain, epochs=10, verbose=1)\n",
        "# save the model\n",
        "model.save('model.h5')\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulary size: 13060\n",
            "Maximum length: 1224\n",
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_3 (Embedding)      (None, 1224, 100)         1306000   \n",
            "_________________________________________________________________\n",
            "conv1d_3 (Conv1D)            (None, 1217, 32)          25632     \n",
            "_________________________________________________________________\n",
            "max_pooling1d_3 (MaxPooling1 (None, 608, 32)           0         \n",
            "_________________________________________________________________\n",
            "flatten_3 (Flatten)          (None, 19456)             0         \n",
            "_________________________________________________________________\n",
            "dense_6 (Dense)              (None, 10)                194570    \n",
            "_________________________________________________________________\n",
            "dense_7 (Dense)              (None, 1)                 11        \n",
            "=================================================================\n",
            "Total params: 1,526,213\n",
            "Trainable params: 1,526,213\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/10\n",
            "48/48 [==============================] - 11s 227ms/step - loss: 0.6699 - accuracy: 0.5972\n",
            "Epoch 2/10\n",
            "48/48 [==============================] - 11s 227ms/step - loss: 0.5198 - accuracy: 0.6496\n",
            "Epoch 3/10\n",
            "48/48 [==============================] - 11s 229ms/step - loss: 0.3170 - accuracy: 0.9456\n",
            "Epoch 4/10\n",
            "48/48 [==============================] - 11s 230ms/step - loss: 0.2736 - accuracy: 0.9867\n",
            "Epoch 5/10\n",
            "48/48 [==============================] - 11s 226ms/step - loss: 0.2506 - accuracy: 0.9980\n",
            "Epoch 6/10\n",
            "48/48 [==============================] - 11s 226ms/step - loss: 0.2399 - accuracy: 0.9987\n",
            "Epoch 7/10\n",
            "48/48 [==============================] - 11s 227ms/step - loss: 0.2297 - accuracy: 1.0000\n",
            "Epoch 8/10\n",
            "48/48 [==============================] - 11s 225ms/step - loss: 0.2208 - accuracy: 1.0000\n",
            "Epoch 9/10\n",
            "48/48 [==============================] - 11s 225ms/step - loss: 0.2128 - accuracy: 1.0000\n",
            "Epoch 10/10\n",
            "48/48 [==============================] - 11s 227ms/step - loss: 0.2052 - accuracy: 1.0000\n",
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_3 (Embedding)      (None, 1224, 100)         1306000   \n",
            "_________________________________________________________________\n",
            "conv1d_3 (Conv1D)            (None, 1217, 32)          25632     \n",
            "_________________________________________________________________\n",
            "max_pooling1d_3 (MaxPooling1 (None, 608, 32)           0         \n",
            "_________________________________________________________________\n",
            "flatten_3 (Flatten)          (None, 19456)             0         \n",
            "_________________________________________________________________\n",
            "dense_6 (Dense)              (None, 10)                194570    \n",
            "_________________________________________________________________\n",
            "dense_7 (Dense)              (None, 1)                 11        \n",
            "=================================================================\n",
            "Total params: 1,526,213\n",
            "Trainable params: 1,526,213\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzvlLktNUTxv"
      },
      "source": [
        "##**15.5 Evaluate Model**\n",
        "\n",
        "In this section, we will evaluate the trained model and use it to make predictions on new data.\n",
        "First, we can use the built-in evaluate() function to estimate the skill of the model on both\n",
        "the training and test dataset. This requires that we load and encode both the training and test\n",
        "datasets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KNIoXNEJsWR7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "03469a21-1e8e-4111-e649-6b92142d5285"
      },
      "source": [
        "# load all reviews\n",
        "train_docs, ytrain = load_clean_dataset(vocab, True)\n",
        "test_docs, ytest = load_clean_dataset(vocab, False)\n",
        "# create the tokenizer\n",
        "tokenizer = create_tokenizer(train_docs)\n",
        "# define vocabulary size\n",
        "vocab_size = len(tokenizer.word_index) + 1\n",
        "print('Vocabulary size: %d' % vocab_size)\n",
        "# calculate the maximum sequence length\n",
        "max_length = max([len(s.split()) for s in train_docs])\n",
        "print('Maximum length: %d' % max_length)\n",
        "# encode data\n",
        "Xtrain = encode_docs(tokenizer, max_length, train_docs)\n",
        "Xtest = encode_docs(tokenizer, max_length, test_docs)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulary size: 13060\n",
            "Maximum length: 1224\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TfPt4HrLdRiD"
      },
      "source": [
        "We can then load the model and evaluate it on both datasets and print the accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RGY5dVnNdO-V",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "1f02595f-25be-4f5a-e7c5-f6c18464a73a"
      },
      "source": [
        "# load the model\n",
        "model = models.load_model('model.h5')\n",
        "# evaluate model on training dataset\n",
        "_, acc = model.evaluate(Xtrain, ytrain, verbose=0)\n",
        "print('Train Accuracy: %f' % (acc*100))\n",
        "# evaluate model on test dataset\n",
        "_, acc = model.evaluate(Xtest, ytest, verbose=0)\n",
        "print('Test Accuracy: %f' % (acc*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train Accuracy: 100.000000\n",
            "Test Accuracy: 97.000003\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q4lnad7cgrH7"
      },
      "source": [
        "New data must then be prepared using the same text encoding and encoding schemes as was\n",
        "used on the training dataset. Once prepared, a prediction can be made by calling the predict()\n",
        "function on the model. The function below named predict sentiment() will encode and pad\n",
        "a given movie review text and return a prediction in terms of both the percentage and a label."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sfQux3Hu4hM0"
      },
      "source": [
        "# classify a review as negative or positive\n",
        "def predict_sentiment(review, vocab, tokenizer, max_length, model):\n",
        "    # clean review\n",
        "    line = clean_doc(review, vocab)\n",
        "    # encode and pad review\n",
        "    padded = encode_docs(tokenizer, max_length, [line])\n",
        "    # predict sentiment\n",
        "    yhat = model.predict(padded, verbose=0)\n",
        "    # retrieve predicted percentage and label\n",
        "    percent_pos = yhat[0,0]\n",
        "    if round(percent_pos) == 0:\n",
        "        return (1-percent_pos), 'NEGATIVE'\n",
        "    return percent_pos, 'POSITIVE'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f56tFVxOiNtA"
      },
      "source": [
        "## Test Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XxSA-v2Ahxyg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "cee79be3-78d3-4b51-919b-9cd3074b713f"
      },
      "source": [
        "# test positive text\n",
        "text = 'Everyone will enjoy this film. I love it, recommended!'\n",
        "percent, sentiment = predict_sentiment(text, vocab, tokenizer, max_length, model)\n",
        "print('Review: [%s]\\nSentiment: %s (%.3f%%)' % (text, sentiment, percent*100))\n",
        "# test negative text\n",
        "text = 'This is a bad movie. Do not watch it. It sucks.'\n",
        "percent, sentiment = predict_sentiment(text, vocab, tokenizer, max_length, model)\n",
        "print('Review: [%s]\\nSentiment: %s (%.3f%%)' % (text, sentiment, percent*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Review: [Everyone will enjoy this film. I love it, recommended!]\n",
            "Sentiment: NEGATIVE (59.373%)\n",
            "Review: [This is a bad movie. Do not watch it. It sucks.]\n",
            "Sentiment: NEGATIVE (60.713%)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rv6PVmsCxH7A"
      },
      "source": [
        "#**Chapter 16 Project: Develop an n-gram CNN Model for Sentiment Analysis**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fo2pnicuyC6a"
      },
      "source": [
        "##**16.3.4 Complete Example**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xazb6BQNy0bm",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "8dc3d1f3-f9bb-4f8e-e39c-f6501da511b9"
      },
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "import string\n",
        "import re\n",
        "from os import listdir\n",
        "from nltk.corpus import stopwords\n",
        "from pickle import dump\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# turn a doc into clean tokens\n",
        "def clean_doc(doc):\n",
        "    # split into tokens by white space\n",
        "    tokens = doc.split()\n",
        "    # prepare regex for char filtering\n",
        "    re_punc = re.compile('[%s]' % re.escape(string.punctuation))\n",
        "    # remove punctuation from each word\n",
        "    tokens = [re_punc.sub('', w) for w in tokens]\n",
        "    # remove remaining tokens that are not alphabetic\n",
        "    tokens = [word for word in tokens if word.isalpha()]\n",
        "    # filter out stop words\n",
        "    stop_words = set(stopwords.words('english'))\n",
        "    tokens = [w for w in tokens if not w in stop_words]\n",
        "    # filter out short tokens\n",
        "    tokens = [word for word in tokens if len(word) > 1]\n",
        "    tokens = ' '.join(tokens)\n",
        "    return tokens\n",
        "# load all docs in a directory\n",
        "def process_docs(directory, is_train):\n",
        "    documents = list()\n",
        "    # walk through all files in the folder\n",
        "    for filename in listdir(directory):\n",
        "        # skip any reviews in the test set\n",
        "        if is_train and filename.startswith('cv9'):\n",
        "            continue\n",
        "        if not is_train and not filename.startswith('cv9'):\n",
        "            continue\n",
        "        # create the full path of the file to open\n",
        "        path = directory + '/' + filename\n",
        "        # load the doc\n",
        "        doc = load_doc(path)\n",
        "        # clean doc\n",
        "        tokens = clean_doc(doc)\n",
        "        # add to list\n",
        "        documents.append(tokens)\n",
        "    return documents\n",
        "# load and clean a dataset\n",
        "def load_clean_dataset(is_train):\n",
        "      # load documents\n",
        "      neg = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/neg', is_train)\n",
        "      pos = process_docs('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/txt_sentoken/pos', is_train)\n",
        "      docs = neg + pos\n",
        "      # prepare labels\n",
        "      labels = [0 for _ in range(len(neg))] + [1 for _ in range(len(pos))]\n",
        "      return docs, labels\n",
        "# save a dataset to file\n",
        "def save_dataset(dataset, filename):\n",
        "    dump(dataset, open(filename, 'wb'))\n",
        "    print('Saved: %s' % filename)\n",
        "# load and clean all reviews\n",
        "train_docs, ytrain = load_clean_dataset(True)\n",
        "test_docs, ytest = load_clean_dataset(False)\n",
        "# save training datasets\n",
        "save_dataset([train_docs, ytrain], '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/train.pkl')\n",
        "save_dataset([test_docs, ytest], '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/test.pkl')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "Saved: /content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/train.pkl\n",
            "Saved: /content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/test.pkl\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dpcg0s7B2o-6"
      },
      "source": [
        "##**16.4 Develop Multichannel Model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4QMNk_PvyZ1"
      },
      "source": [
        "###**16.4.2 Define Model**\n",
        "\n",
        "In Keras, a multiple-input model can be defined using the functional API. We will define a\n",
        "model with three input channels for processing 4-grams, 6-grams, and 8-grams of movie review\n",
        "text. Each channel is comprised of the following elements:\n",
        "\n",
        "* Input layer that defines the length of input sequences.\n",
        "* Embedding layer set to the size of the vocabulary and 100-dimensional real-valued representations.\n",
        "* Conv1D layer with 32 filters and a kernel size set to the number of words to read at once.\n",
        "* MaxPooling1D layer to consolidate the output from the convolutional layer.\n",
        "* Flatten layer to reduce the three-dimensional output to two dimensional for concatenation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TLOoeiSsxN4z",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "31d9fa0f-4d45-4925-d358-bfd9c78a8619"
      },
      "source": [
        "from pickle import load\n",
        "import numpy as np\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from keras.utils.vis_utils import plot_model\n",
        "from keras.models import Model\n",
        "from keras.layers import Input\n",
        "from keras.layers import Dense\n",
        "from keras.layers import Flatten\n",
        "from keras.layers import Dropout\n",
        "from keras.layers import Embedding\n",
        "from keras.layers.convolutional import Conv1D\n",
        "from keras.layers.convolutional import MaxPooling1D\n",
        "from keras.layers.merge import concatenate\n",
        "# load a clean dataset\n",
        "def load_dataset(filename):\n",
        "    return load(open(filename, 'rb'))\n",
        "# fit a tokenizer\n",
        "def create_tokenizer(lines):\n",
        "    tokenizer = Tokenizer()\n",
        "    tokenizer.fit_on_texts(lines)\n",
        "    return tokenizer\n",
        "# calculate the maximum document length\n",
        "def max_length(lines):\n",
        "    return max([len(s.split()) for s in lines])\n",
        "# encode a list of lines\n",
        "def encode_text(tokenizer, lines, length):\n",
        "    # integer encode\n",
        "    encoded = tokenizer.texts_to_sequences(lines)\n",
        "    # pad encoded sequences\n",
        "    padded = pad_sequences(encoded, maxlen=length, padding='post')\n",
        "    return padded\n",
        "# define the model\n",
        "def define_model(length, vocab_size):\n",
        "    # channel 1\n",
        "    inputs1 = Input(shape=(length,))\n",
        "    embedding1 = Embedding(vocab_size, 100)(inputs1)\n",
        "    conv1 = Conv1D(filters=32, kernel_size=4, activation='relu')(embedding1)\n",
        "    drop1 = Dropout(0.5)(conv1)\n",
        "    pool1 = MaxPooling1D(pool_size=2)(drop1)\n",
        "    flat1 = Flatten()(pool1)\n",
        "    # channel 2\n",
        "    inputs2 = Input(shape=(length,))\n",
        "    embedding2 = Embedding(vocab_size, 100)(inputs2)\n",
        "    conv2 = Conv1D(filters=32, kernel_size=6, activation='relu')(embedding2)\n",
        "    drop2 = Dropout(0.5)(conv2)\n",
        "    pool2 = MaxPooling1D(pool_size=2)(drop2)\n",
        "    flat2 = Flatten()(pool2)\n",
        "    # channel 3\n",
        "    inputs3 = Input(shape=(length,))\n",
        "    embedding3 = Embedding(vocab_size, 100)(inputs3)\n",
        "    conv3 = Conv1D(filters=32, kernel_size=8, activation='relu')(embedding3)\n",
        "    drop3 = Dropout(0.5)(conv3)\n",
        "    pool3 = MaxPooling1D(pool_size=2)(drop3)\n",
        "    flat3 = Flatten()(pool3)\n",
        "    # merge\n",
        "    merged = concatenate([flat1, flat2, flat3])\n",
        "    # interpretation\n",
        "    dense1 = Dense(10, activation='relu')(merged)\n",
        "    outputs = Dense(1, activation='sigmoid')(dense1)\n",
        "    model = Model(inputs=[inputs1, inputs2, inputs3], outputs=outputs)\n",
        "    # compile\n",
        "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "    # summarize\n",
        "    model.summary()\n",
        "    plot_model(model, show_shapes=True, to_file='model.png')\n",
        "    return model\n",
        "# load training dataset\n",
        "trainLines, trainLabels = load_dataset('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/train.pkl')\n",
        "# create tokenizer\n",
        "tokenizer = create_tokenizer(trainLines)\n",
        "# calculate max document length\n",
        "length = max_length(trainLines)\n",
        "print('Max document length: %d' % length)\n",
        "# calculate vocabulary size\n",
        "vocab_size = len(tokenizer.word_index) + 1\n",
        "print('Vocabulary size: %d' % vocab_size)\n",
        "# encode data\n",
        "trainX = encode_text(tokenizer, trainLines, length)\n",
        "# define model\n",
        "model = define_model(length, vocab_size)\n",
        "# fit model\n",
        "model.fit([np.array(trainX, dtype='float64'),np.array(trainX, dtype='float64'),np.array(trainX, dtype='float64')], np.array(trainLabels), epochs=7, batch_size=16)\n",
        "# save the model\n",
        "model.save('model.h5')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Max document length: 1380\n",
            "Vocabulary size: 40852\n",
            "Model: \"functional_7\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_10 (InputLayer)           [(None, 1380)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_11 (InputLayer)           [(None, 1380)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_12 (InputLayer)           [(None, 1380)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding_9 (Embedding)         (None, 1380, 100)    4085200     input_10[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "embedding_10 (Embedding)        (None, 1380, 100)    4085200     input_11[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "embedding_11 (Embedding)        (None, 1380, 100)    4085200     input_12[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_9 (Conv1D)               (None, 1377, 32)     12832       embedding_9[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_10 (Conv1D)              (None, 1375, 32)     19232       embedding_10[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv1d_11 (Conv1D)              (None, 1373, 32)     25632       embedding_11[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "dropout_9 (Dropout)             (None, 1377, 32)     0           conv1d_9[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "dropout_10 (Dropout)            (None, 1375, 32)     0           conv1d_10[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "dropout_11 (Dropout)            (None, 1373, 32)     0           conv1d_11[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "max_pooling1d_9 (MaxPooling1D)  (None, 688, 32)      0           dropout_9[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "max_pooling1d_10 (MaxPooling1D) (None, 687, 32)      0           dropout_10[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "max_pooling1d_11 (MaxPooling1D) (None, 686, 32)      0           dropout_11[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "flatten_9 (Flatten)             (None, 22016)        0           max_pooling1d_9[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "flatten_10 (Flatten)            (None, 21984)        0           max_pooling1d_10[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "flatten_11 (Flatten)            (None, 21952)        0           max_pooling1d_11[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "concatenate_3 (Concatenate)     (None, 65952)        0           flatten_9[0][0]                  \n",
            "                                                                 flatten_10[0][0]                 \n",
            "                                                                 flatten_11[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "dense_6 (Dense)                 (None, 10)           659530      concatenate_3[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "dense_7 (Dense)                 (None, 1)            11          dense_6[0][0]                    \n",
            "==================================================================================================\n",
            "Total params: 12,972,837\n",
            "Trainable params: 12,972,837\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "Epoch 1/7\n",
            "95/95 [==============================] - 39s 416ms/step - loss: 0.6702 - accuracy: 0.5965\n",
            "Epoch 2/7\n",
            "95/95 [==============================] - 39s 414ms/step - loss: 0.4026 - accuracy: 0.8222\n",
            "Epoch 3/7\n",
            "95/95 [==============================] - 40s 416ms/step - loss: 0.2678 - accuracy: 0.9861\n",
            "Epoch 4/7\n",
            "95/95 [==============================] - 40s 419ms/step - loss: 0.2390 - accuracy: 0.9980\n",
            "Epoch 5/7\n",
            "95/95 [==============================] - 40s 416ms/step - loss: 0.2201 - accuracy: 0.9987\n",
            "Epoch 6/7\n",
            "95/95 [==============================] - 40s 420ms/step - loss: 0.2051 - accuracy: 0.9993\n",
            "Epoch 7/7\n",
            "95/95 [==============================] - 39s 414ms/step - loss: 0.1907 - accuracy: 1.0000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JvBDbxpS22u9"
      },
      "source": [
        "##**16.5 Evaluate Model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bLQmv2AWygc0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "e77b96c9-6f23-4ee3-c2ba-837ace15c599"
      },
      "source": [
        "from pickle import load\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from keras.models import load_model\n",
        "# load a clean dataset\n",
        "def load_dataset(filename):\n",
        "    return load(open(filename, 'rb'))\n",
        "# fit a tokenizer\n",
        "def create_tokenizer(lines):\n",
        "    tokenizer = Tokenizer()\n",
        "    tokenizer.fit_on_texts(lines)\n",
        "    return tokenizer\n",
        "# calculate the maximum document length\n",
        "def max_length(lines):\n",
        "    return max([len(s.split()) for s in lines])\n",
        "# encode a list of lines\n",
        "def encode_text(tokenizer, lines, length):\n",
        "    # integer encode\n",
        "    encoded = tokenizer.texts_to_sequences(lines)\n",
        "    # pad encoded sequences\n",
        "    padded = pad_sequences(encoded, maxlen=length, padding='post')\n",
        "    return padded\n",
        "# load datasets\n",
        "trainLines, trainLabels = load_dataset('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/train.pkl')\n",
        "testLines, testLabels = load_dataset('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/Review_moving_polarity/test.pkl')\n",
        "# create tokenizer\n",
        "tokenizer = create_tokenizer(trainLines)\n",
        "# calculate max document length\n",
        "length = max_length(trainLines)\n",
        "print('Max document length: %d' % length)\n",
        "# calculate vocabulary size\n",
        "vocab_size = len(tokenizer.word_index) + 1\n",
        "print('Vocabulary size: %d' % vocab_size)\n",
        "# encode data\n",
        "trainX = encode_text(tokenizer, trainLines, length)\n",
        "testX = encode_text(tokenizer, testLines, length)\n",
        "# load the model\n",
        "model = load_model('model.h5')\n",
        "# evaluate model on training dataset\n",
        "_, acc = model.evaluate([trainX,trainX,trainX], np.array(trainLabels), verbose=0)\n",
        "print('Train Accuracy: %.2f' % (acc*100))\n",
        "# evaluate model on test dataset dataset\n",
        "_, acc = model.evaluate([testX,testX,testX], np.array(testLabels), verbose=0)\n",
        "print('Test Accuracy: %.2f' % (acc*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Max document length: 1380\n",
            "Vocabulary size: 40852\n",
            "Train Accuracy: 100.00\n",
            "Test Accuracy: 95.00\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bgsr5Gx_9AQ-"
      },
      "source": [
        "#**Part VII Language Modeling**\n",
        "#**Chapter 17 Neural Language Modeling**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P0ZhEpT8_p0Z"
      },
      "source": [
        "#**Chapter 18 How to Develop a Character-Based Neural Language Model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ugu6ONmFAzxV"
      },
      "source": [
        "A language model predicts the next word in the sequence based on the specific words that have\n",
        "come before it in the sequence. It is also possible to develop language models at the character\n",
        "level using neural networks. The benefit of character-based language models is their small\n",
        "vocabulary and flexibility in handling any words, punctuation, and other document structure.\n",
        "This comes at the cost of requiring larger models that are slower to train. Nevertheless, in the\n",
        "field of neural language models, character-based models offer a lot of promise for a general,\n",
        "flexible and powerful approach to language modeling. In this tutorial, you will discover how to\n",
        "develop a character-based neural language model. After completing this tutorial, you will know:\n",
        "\n",
        "* How to prepare text for character-based language modeling.\n",
        "* How to develop a character-based language model using LSTMs.\n",
        "* How to use a trained character-based language model to generate text."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBQprSnLBD7B"
      },
      "source": [
        "##**18.1 Tutorial Overview**\n",
        "This tutorial is divided into the following parts:\n",
        "1. Sing a Song of Sixpence\n",
        "2. Data Preparation\n",
        "3. Train Language Model\n",
        "4. Generate Text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qjQZ_W0dCrOG"
      },
      "source": [
        "##**18.3.6 Complete Example**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sSZ2SuEb8LHV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        },
        "outputId": "3cfad21c-930c-4cd5-ba83-5448816e584e"
      },
      "source": [
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# save tokens to file, one dialog per line\n",
        "def save_doc(lines, filename):\n",
        "    data = '\\n'.join(lines)\n",
        "    file = open(filename, 'w')\n",
        "    file.write(data)\n",
        "    file.close()\n",
        "# load text\n",
        "raw_text = load_doc('/content/drive/My Drive/Colab Notebooks/NLP/Text Files/rhyme.txt')\n",
        "print(raw_text)\n",
        "# clean\n",
        "tokens = raw_text.split()\n",
        "raw_text = ' '.join(tokens)\n",
        "print(len(raw_text))\n",
        "# organize into sequences of characters\n",
        "length = 10\n",
        "sequences = list()\n",
        "for i in range(length, len(raw_text)):\n",
        "    # select sequence of tokens\n",
        "    seq = raw_text[i-length:i+1]\n",
        "    # store\n",
        "    sequences.append(seq)\n",
        "print('Total Sequences: %d' % len(sequences))\n",
        "# save sequences to file\n",
        "out_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/char_sequences.txt'\n",
        "save_doc(sequences,out_filename)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sing a song of sixpence,\n",
            "A pocket full of rye.\n",
            "Four and twenty blackbirds,\n",
            "Baked in a pie.\n",
            "\n",
            "When the pie was opened\n",
            "The birds began to sing;\n",
            "Wasn't that a dainty dish,\n",
            "To set before the king.\n",
            "\n",
            "The king was in his counting house,\n",
            "Counting out his money;\n",
            "The queen was in the parlour,\n",
            "Eating bread and honey.\n",
            "\n",
            "The maid was in the garden,\n",
            "Hanging out the clothes,\n",
            "When down came a blackbird\n",
            "And pecked off her nose.\n",
            "409\n",
            "Total Sequences: 399\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KCtxISngNC25"
      },
      "source": [
        "##**18.4 Train Language Model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MyAAsQs0P7tC"
      },
      "source": [
        "###**18.4.1 Load Data**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "boG_-BPjIddA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "12bf3d85-ab6f-4f92-d71e-a020ec78b7d6"
      },
      "source": [
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# load\n",
        "in_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/char_sequences.txt'\n",
        "raw_text = load_doc(in_filename)\n",
        "#print(raw_text)\n",
        "lines = raw_text.split('\\n')\n",
        "lines"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Sing a song',\n",
              " 'ing a song ',\n",
              " 'ng a song o',\n",
              " 'g a song of',\n",
              " ' a song of ',\n",
              " 'a song of s',\n",
              " ' song of si',\n",
              " 'song of six',\n",
              " 'ong of sixp',\n",
              " 'ng of sixpe',\n",
              " 'g of sixpen',\n",
              " ' of sixpenc',\n",
              " 'of sixpence',\n",
              " 'f sixpence,',\n",
              " ' sixpence, ',\n",
              " 'sixpence, A',\n",
              " 'ixpence, A ',\n",
              " 'xpence, A p',\n",
              " 'pence, A po',\n",
              " 'ence, A poc',\n",
              " 'nce, A pock',\n",
              " 'ce, A pocke',\n",
              " 'e, A pocket',\n",
              " ', A pocket ',\n",
              " ' A pocket f',\n",
              " 'A pocket fu',\n",
              " ' pocket ful',\n",
              " 'pocket full',\n",
              " 'ocket full ',\n",
              " 'cket full o',\n",
              " 'ket full of',\n",
              " 'et full of ',\n",
              " 't full of r',\n",
              " ' full of ry',\n",
              " 'full of rye',\n",
              " 'ull of rye.',\n",
              " 'll of rye. ',\n",
              " 'l of rye. F',\n",
              " ' of rye. Fo',\n",
              " 'of rye. Fou',\n",
              " 'f rye. Four',\n",
              " ' rye. Four ',\n",
              " 'rye. Four a',\n",
              " 'ye. Four an',\n",
              " 'e. Four and',\n",
              " '. Four and ',\n",
              " ' Four and t',\n",
              " 'Four and tw',\n",
              " 'our and twe',\n",
              " 'ur and twen',\n",
              " 'r and twent',\n",
              " ' and twenty',\n",
              " 'and twenty ',\n",
              " 'nd twenty b',\n",
              " 'd twenty bl',\n",
              " ' twenty bla',\n",
              " 'twenty blac',\n",
              " 'wenty black',\n",
              " 'enty blackb',\n",
              " 'nty blackbi',\n",
              " 'ty blackbir',\n",
              " 'y blackbird',\n",
              " ' blackbirds',\n",
              " 'blackbirds,',\n",
              " 'lackbirds, ',\n",
              " 'ackbirds, B',\n",
              " 'ckbirds, Ba',\n",
              " 'kbirds, Bak',\n",
              " 'birds, Bake',\n",
              " 'irds, Baked',\n",
              " 'rds, Baked ',\n",
              " 'ds, Baked i',\n",
              " 's, Baked in',\n",
              " ', Baked in ',\n",
              " ' Baked in a',\n",
              " 'Baked in a ',\n",
              " 'aked in a p',\n",
              " 'ked in a pi',\n",
              " 'ed in a pie',\n",
              " 'd in a pie.',\n",
              " ' in a pie. ',\n",
              " 'in a pie. W',\n",
              " 'n a pie. Wh',\n",
              " ' a pie. Whe',\n",
              " 'a pie. When',\n",
              " ' pie. When ',\n",
              " 'pie. When t',\n",
              " 'ie. When th',\n",
              " 'e. When the',\n",
              " '. When the ',\n",
              " ' When the p',\n",
              " 'When the pi',\n",
              " 'hen the pie',\n",
              " 'en the pie ',\n",
              " 'n the pie w',\n",
              " ' the pie wa',\n",
              " 'the pie was',\n",
              " 'he pie was ',\n",
              " 'e pie was o',\n",
              " ' pie was op',\n",
              " 'pie was ope',\n",
              " 'ie was open',\n",
              " 'e was opene',\n",
              " ' was opened',\n",
              " 'was opened ',\n",
              " 'as opened T',\n",
              " 's opened Th',\n",
              " ' opened The',\n",
              " 'opened The ',\n",
              " 'pened The b',\n",
              " 'ened The bi',\n",
              " 'ned The bir',\n",
              " 'ed The bird',\n",
              " 'd The birds',\n",
              " ' The birds ',\n",
              " 'The birds b',\n",
              " 'he birds be',\n",
              " 'e birds beg',\n",
              " ' birds bega',\n",
              " 'birds began',\n",
              " 'irds began ',\n",
              " 'rds began t',\n",
              " 'ds began to',\n",
              " 's began to ',\n",
              " ' began to s',\n",
              " 'began to si',\n",
              " 'egan to sin',\n",
              " 'gan to sing',\n",
              " 'an to sing;',\n",
              " 'n to sing; ',\n",
              " ' to sing; W',\n",
              " 'to sing; Wa',\n",
              " 'o sing; Was',\n",
              " ' sing; Wasn',\n",
              " \"sing; Wasn'\",\n",
              " \"ing; Wasn't\",\n",
              " \"ng; Wasn't \",\n",
              " \"g; Wasn't t\",\n",
              " \"; Wasn't th\",\n",
              " \" Wasn't tha\",\n",
              " \"Wasn't that\",\n",
              " \"asn't that \",\n",
              " \"sn't that a\",\n",
              " \"n't that a \",\n",
              " \"'t that a d\",\n",
              " 't that a da',\n",
              " ' that a dai',\n",
              " 'that a dain',\n",
              " 'hat a daint',\n",
              " 'at a dainty',\n",
              " 't a dainty ',\n",
              " ' a dainty d',\n",
              " 'a dainty di',\n",
              " ' dainty dis',\n",
              " 'dainty dish',\n",
              " 'ainty dish,',\n",
              " 'inty dish, ',\n",
              " 'nty dish, T',\n",
              " 'ty dish, To',\n",
              " 'y dish, To ',\n",
              " ' dish, To s',\n",
              " 'dish, To se',\n",
              " 'ish, To set',\n",
              " 'sh, To set ',\n",
              " 'h, To set b',\n",
              " ', To set be',\n",
              " ' To set bef',\n",
              " 'To set befo',\n",
              " 'o set befor',\n",
              " ' set before',\n",
              " 'set before ',\n",
              " 'et before t',\n",
              " 't before th',\n",
              " ' before the',\n",
              " 'before the ',\n",
              " 'efore the k',\n",
              " 'fore the ki',\n",
              " 'ore the kin',\n",
              " 're the king',\n",
              " 'e the king.',\n",
              " ' the king. ',\n",
              " 'the king. T',\n",
              " 'he king. Th',\n",
              " 'e king. The',\n",
              " ' king. The ',\n",
              " 'king. The k',\n",
              " 'ing. The ki',\n",
              " 'ng. The kin',\n",
              " 'g. The king',\n",
              " '. The king ',\n",
              " ' The king w',\n",
              " 'The king wa',\n",
              " 'he king was',\n",
              " 'e king was ',\n",
              " ' king was i',\n",
              " 'king was in',\n",
              " 'ing was in ',\n",
              " 'ng was in h',\n",
              " 'g was in hi',\n",
              " ' was in his',\n",
              " 'was in his ',\n",
              " 'as in his c',\n",
              " 's in his co',\n",
              " ' in his cou',\n",
              " 'in his coun',\n",
              " 'n his count',\n",
              " ' his counti',\n",
              " 'his countin',\n",
              " 'is counting',\n",
              " 's counting ',\n",
              " ' counting h',\n",
              " 'counting ho',\n",
              " 'ounting hou',\n",
              " 'unting hous',\n",
              " 'nting house',\n",
              " 'ting house,',\n",
              " 'ing house, ',\n",
              " 'ng house, C',\n",
              " 'g house, Co',\n",
              " ' house, Cou',\n",
              " 'house, Coun',\n",
              " 'ouse, Count',\n",
              " 'use, Counti',\n",
              " 'se, Countin',\n",
              " 'e, Counting',\n",
              " ', Counting ',\n",
              " ' Counting o',\n",
              " 'Counting ou',\n",
              " 'ounting out',\n",
              " 'unting out ',\n",
              " 'nting out h',\n",
              " 'ting out hi',\n",
              " 'ing out his',\n",
              " 'ng out his ',\n",
              " 'g out his m',\n",
              " ' out his mo',\n",
              " 'out his mon',\n",
              " 'ut his mone',\n",
              " 't his money',\n",
              " ' his money;',\n",
              " 'his money; ',\n",
              " 'is money; T',\n",
              " 's money; Th',\n",
              " ' money; The',\n",
              " 'money; The ',\n",
              " 'oney; The q',\n",
              " 'ney; The qu',\n",
              " 'ey; The que',\n",
              " 'y; The quee',\n",
              " '; The queen',\n",
              " ' The queen ',\n",
              " 'The queen w',\n",
              " 'he queen wa',\n",
              " 'e queen was',\n",
              " ' queen was ',\n",
              " 'queen was i',\n",
              " 'ueen was in',\n",
              " 'een was in ',\n",
              " 'en was in t',\n",
              " 'n was in th',\n",
              " ' was in the',\n",
              " 'was in the ',\n",
              " 'as in the p',\n",
              " 's in the pa',\n",
              " ' in the par',\n",
              " 'in the parl',\n",
              " 'n the parlo',\n",
              " ' the parlou',\n",
              " 'the parlour',\n",
              " 'he parlour,',\n",
              " 'e parlour, ',\n",
              " ' parlour, E',\n",
              " 'parlour, Ea',\n",
              " 'arlour, Eat',\n",
              " 'rlour, Eati',\n",
              " 'lour, Eatin',\n",
              " 'our, Eating',\n",
              " 'ur, Eating ',\n",
              " 'r, Eating b',\n",
              " ', Eating br',\n",
              " ' Eating bre',\n",
              " 'Eating brea',\n",
              " 'ating bread',\n",
              " 'ting bread ',\n",
              " 'ing bread a',\n",
              " 'ng bread an',\n",
              " 'g bread and',\n",
              " ' bread and ',\n",
              " 'bread and h',\n",
              " 'read and ho',\n",
              " 'ead and hon',\n",
              " 'ad and hone',\n",
              " 'd and honey',\n",
              " ' and honey.',\n",
              " 'and honey. ',\n",
              " 'nd honey. T',\n",
              " 'd honey. Th',\n",
              " ' honey. The',\n",
              " 'honey. The ',\n",
              " 'oney. The m',\n",
              " 'ney. The ma',\n",
              " 'ey. The mai',\n",
              " 'y. The maid',\n",
              " '. The maid ',\n",
              " ' The maid w',\n",
              " 'The maid wa',\n",
              " 'he maid was',\n",
              " 'e maid was ',\n",
              " ' maid was i',\n",
              " 'maid was in',\n",
              " 'aid was in ',\n",
              " 'id was in t',\n",
              " 'd was in th',\n",
              " ' was in the',\n",
              " 'was in the ',\n",
              " 'as in the g',\n",
              " 's in the ga',\n",
              " ' in the gar',\n",
              " 'in the gard',\n",
              " 'n the garde',\n",
              " ' the garden',\n",
              " 'the garden,',\n",
              " 'he garden, ',\n",
              " 'e garden, H',\n",
              " ' garden, Ha',\n",
              " 'garden, Han',\n",
              " 'arden, Hang',\n",
              " 'rden, Hangi',\n",
              " 'den, Hangin',\n",
              " 'en, Hanging',\n",
              " 'n, Hanging ',\n",
              " ', Hanging o',\n",
              " ' Hanging ou',\n",
              " 'Hanging out',\n",
              " 'anging out ',\n",
              " 'nging out t',\n",
              " 'ging out th',\n",
              " 'ing out the',\n",
              " 'ng out the ',\n",
              " 'g out the c',\n",
              " ' out the cl',\n",
              " 'out the clo',\n",
              " 'ut the clot',\n",
              " 't the cloth',\n",
              " ' the clothe',\n",
              " 'the clothes',\n",
              " 'he clothes,',\n",
              " 'e clothes, ',\n",
              " ' clothes, W',\n",
              " 'clothes, Wh',\n",
              " 'lothes, Whe',\n",
              " 'othes, When',\n",
              " 'thes, When ',\n",
              " 'hes, When d',\n",
              " 'es, When do',\n",
              " 's, When dow',\n",
              " ', When down',\n",
              " ' When down ',\n",
              " 'When down c',\n",
              " 'hen down ca',\n",
              " 'en down cam',\n",
              " 'n down came',\n",
              " ' down came ',\n",
              " 'down came a',\n",
              " 'own came a ',\n",
              " 'wn came a b',\n",
              " 'n came a bl',\n",
              " ' came a bla',\n",
              " 'came a blac',\n",
              " 'ame a black',\n",
              " 'me a blackb',\n",
              " 'e a blackbi',\n",
              " ' a blackbir',\n",
              " 'a blackbird',\n",
              " ' blackbird ',\n",
              " 'blackbird A',\n",
              " 'lackbird An',\n",
              " 'ackbird And',\n",
              " 'ckbird And ',\n",
              " 'kbird And p',\n",
              " 'bird And pe',\n",
              " 'ird And pec',\n",
              " 'rd And peck',\n",
              " 'd And pecke',\n",
              " ' And pecked',\n",
              " 'And pecked ',\n",
              " 'nd pecked o',\n",
              " 'd pecked of',\n",
              " ' pecked off',\n",
              " 'pecked off ',\n",
              " 'ecked off h',\n",
              " 'cked off he',\n",
              " 'ked off her',\n",
              " 'ed off her ',\n",
              " 'd off her n',\n",
              " ' off her no',\n",
              " 'off her nos',\n",
              " 'ff her nose',\n",
              " 'f her nose.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eQ-nUaRhooSI"
      },
      "source": [
        "##**18.4.2 Encode Sequences**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RRIdB2zkANo6"
      },
      "source": [
        "The sequences of characters must be encoded as integers. This means that each unique character\n",
        "will be assigned a specific integer value and each sequence of characters will be encoded as a\n",
        "sequence of integers. We can create the mapping given a sorted set of unique characters in the\n",
        "raw input data. The mapping is a dictionary of character values to integer values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0e_VUnGHovTv"
      },
      "source": [
        "chars = sorted(list(set(raw_text)))\n",
        "mapping = dict((c, i) for i, c in enumerate(chars))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RGPP1BpzADNU"
      },
      "source": [
        "Next, we can process each sequence of characters one at a time and use the dictionary\n",
        "mapping to look up the integer value for each character."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wNmhWyWN6Ie_"
      },
      "source": [
        "sequences = list()\n",
        "for line in lines:\n",
        "    # integer encode line\n",
        "    encoded_seq = [mapping[char] for char in line]\n",
        "    # store\n",
        "    sequences.append(encoded_seq)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OoPUK94OGZzr"
      },
      "source": [
        "The result is a list of integer lists. We need to know the size of the vocabulary later. We can\n",
        "retrieve this as the size of the dictionary mapping."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bSqM5GU7B0wd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5c4c6863-b91c-4f42-d016-70d68a0c2550"
      },
      "source": [
        "# vocabulary size\n",
        "vocab_size = len(mapping)\n",
        "print('Vocabulary Size: %d' % vocab_size)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulary Size: 38\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LoRnlxEFIVr4"
      },
      "source": [
        "###**18.4.3 Split Inputs and Output**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FX47WTDSItNw"
      },
      "source": [
        "Now that the sequences have been integer encoded, we can separate the columns into input and\n",
        "output sequences of characters. We can do this using a simple array slice."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "itwFVT1JGfjW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "054bd107-4e72-4764-c3d4-17c696284ba8"
      },
      "source": [
        "import numpy as np\n",
        "sequences = np.array(sequences)\n",
        "X, y = sequences[:,:-1], sequences[:,-1]\n",
        "sequences[0:5], X[0:5], y[0:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([[12, 23, 27, 21,  1, 15,  1, 32, 28, 27, 21],\n",
              "        [23, 27, 21,  1, 15,  1, 32, 28, 27, 21,  1],\n",
              "        [27, 21,  1, 15,  1, 32, 28, 27, 21,  1, 28],\n",
              "        [21,  1, 15,  1, 32, 28, 27, 21,  1, 28, 20],\n",
              "        [ 1, 15,  1, 32, 28, 27, 21,  1, 28, 20,  1]]),\n",
              " array([[12, 23, 27, 21,  1, 15,  1, 32, 28, 27],\n",
              "        [23, 27, 21,  1, 15,  1, 32, 28, 27, 21],\n",
              "        [27, 21,  1, 15,  1, 32, 28, 27, 21,  1],\n",
              "        [21,  1, 15,  1, 32, 28, 27, 21,  1, 28],\n",
              "        [ 1, 15,  1, 32, 28, 27, 21,  1, 28, 20]]),\n",
              " array([21,  1, 28, 20,  1]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZE6iDZ6jMHRU"
      },
      "source": [
        "We can\n",
        "use the to categorical() function in the Keras API to one hot encode the input and output\n",
        "sequences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NdoY_MGyI4-R"
      },
      "source": [
        "import numpy as np\n",
        "from keras.utils import to_categorical\n",
        "sequences = [to_categorical(x, num_classes=vocab_size) for x in X]\n",
        "X = np.array(sequences)\n",
        "y = to_categorical(y, num_classes=vocab_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KYUvlZjWNKUb"
      },
      "source": [
        "###**18.4.4 Fit Model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "giZGvdVHNBis"
      },
      "source": [
        "from numpy import array\n",
        "from pickle import dump\n",
        "from keras.utils.vis_utils import plot_model\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import LSTM\n",
        "\n",
        "# define the model\n",
        "def define_model(X):\n",
        "    model = Sequential()\n",
        "    model.add(LSTM(75, input_shape=(X.shape[1], X.shape[2])))\n",
        "    model.add(Dense(vocab_size, activation='softmax'))\n",
        "    # compile model\n",
        "    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "    # summarize defined model\n",
        "    model.summary()\n",
        "    plot_model(model, to_file='model.png', show_shapes=True)\n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zkKq7pl7OhvQ"
      },
      "source": [
        "###**18.4.5 Save Model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DQo5sqb5Ofxm"
      },
      "source": [
        "# save the model to file\n",
        "model.save('model.h5')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "798zmEaqO7rF"
      },
      "source": [
        "###**18.4.6 Complete Example**\n",
        "Tying all of this together, the complete code listing for fitting the character-based neural\n",
        "language model is listed below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_KDzsxwlOT6P",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "ed0191c8-c5ca-4c39-9cdc-b81ccef49853"
      },
      "source": [
        "from numpy import array\n",
        "from pickle import dump\n",
        "from keras.utils import to_categorical\n",
        "from keras.utils.vis_utils import plot_model\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import LSTM\n",
        "# load doc into memory\n",
        "def load_doc(filename):\n",
        "    # open the file as read only\n",
        "    file = open(filename, 'r')\n",
        "    # read all text\n",
        "    text = file.read()\n",
        "    # close the file\n",
        "    file.close()\n",
        "    return text\n",
        "# define the model\n",
        "def define_model(X):\n",
        "    model = Sequential()\n",
        "    model.add(LSTM(75, input_shape=(X.shape[1], X.shape[2])))\n",
        "    model.add(Dense(vocab_size, activation='softmax'))\n",
        "    # compile model\n",
        "    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "    # summarize defined model\n",
        "    model.summary()\n",
        "    plot_model(model, to_file='model.png', show_shapes=True)\n",
        "    return model\n",
        "\n",
        "# load\n",
        "in_filename = '/content/drive/My Drive/Colab Notebooks/NLP/Text Files/char_sequences.txt'\n",
        "raw_text = load_doc(in_filename)\n",
        "lines = raw_text.split('\\n')\n",
        "\n",
        "# integer encode sequences of characters\n",
        "chars = sorted(list(set(raw_text)))\n",
        "mapping = dict((c, i) for i, c in enumerate(chars))\n",
        "sequences = list()\n",
        "\n",
        "for line in lines:\n",
        "    # integer encode line\n",
        "    encoded_seq = [mapping[char] for char in line]\n",
        "    # store\n",
        "    sequences.append(encoded_seq)\n",
        "    # vocabulary size\n",
        "    vocab_size = len(mapping)\n",
        "print('Vocabulary Size: %d' % vocab_size)\n",
        "\n",
        "# separate into input and output\n",
        "sequences = array(sequences)\n",
        "X, y = sequences[:,:-1], sequences[:,-1]\n",
        "sequences = [to_categorical(x, num_classes=vocab_size) for x in X]\n",
        "X = array(sequences)\n",
        "y = to_categorical(y, num_classes=vocab_size)\n",
        "\n",
        "# define model\n",
        "model = define_model(X)\n",
        "\n",
        "# fit model\n",
        "model.fit(X, y, epochs=100, verbose=2)\n",
        "\n",
        "# save the model to file\n",
        "model.save('model.h5')\n",
        "\n",
        "# save the mapping\n",
        "dump(mapping, open('mapping.pkl', 'wb'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulary Size: 38\n",
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "lstm_1 (LSTM)                (None, 75)                34200     \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 38)                2888      \n",
            "=================================================================\n",
            "Total params: 37,088\n",
            "Trainable params: 37,088\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/100\n",
            "13/13 - 0s - loss: 3.6200 - accuracy: 0.1003\n",
            "Epoch 2/100\n",
            "13/13 - 0s - loss: 3.5178 - accuracy: 0.1905\n",
            "Epoch 3/100\n",
            "13/13 - 0s - loss: 3.2400 - accuracy: 0.1905\n",
            "Epoch 4/100\n",
            "13/13 - 0s - loss: 3.0602 - accuracy: 0.1905\n",
            "Epoch 5/100\n",
            "13/13 - 0s - loss: 3.0106 - accuracy: 0.1905\n",
            "Epoch 6/100\n",
            "13/13 - 0s - loss: 2.9843 - accuracy: 0.1905\n",
            "Epoch 7/100\n",
            "13/13 - 0s - loss: 2.9666 - accuracy: 0.1905\n",
            "Epoch 8/100\n",
            "13/13 - 0s - loss: 2.9444 - accuracy: 0.1905\n",
            "Epoch 9/100\n",
            "13/13 - 0s - loss: 2.9220 - accuracy: 0.1905\n",
            "Epoch 10/100\n",
            "13/13 - 0s - loss: 2.9035 - accuracy: 0.1905\n",
            "Epoch 11/100\n",
            "13/13 - 0s - loss: 2.8807 - accuracy: 0.1905\n",
            "Epoch 12/100\n",
            "13/13 - 0s - loss: 2.8534 - accuracy: 0.2080\n",
            "Epoch 13/100\n",
            "13/13 - 0s - loss: 2.8240 - accuracy: 0.1980\n",
            "Epoch 14/100\n",
            "13/13 - 0s - loss: 2.7836 - accuracy: 0.2206\n",
            "Epoch 15/100\n",
            "13/13 - 0s - loss: 2.7478 - accuracy: 0.2331\n",
            "Epoch 16/100\n",
            "13/13 - 0s - loss: 2.7018 - accuracy: 0.2256\n",
            "Epoch 17/100\n",
            "13/13 - 0s - loss: 2.6640 - accuracy: 0.2707\n",
            "Epoch 18/100\n",
            "13/13 - 0s - loss: 2.6162 - accuracy: 0.2732\n",
            "Epoch 19/100\n",
            "13/13 - 0s - loss: 2.5603 - accuracy: 0.2732\n",
            "Epoch 20/100\n",
            "13/13 - 0s - loss: 2.5219 - accuracy: 0.3133\n",
            "Epoch 21/100\n",
            "13/13 - 0s - loss: 2.4712 - accuracy: 0.2957\n",
            "Epoch 22/100\n",
            "13/13 - 0s - loss: 2.4415 - accuracy: 0.3108\n",
            "Epoch 23/100\n",
            "13/13 - 0s - loss: 2.4135 - accuracy: 0.3183\n",
            "Epoch 24/100\n",
            "13/13 - 0s - loss: 2.3469 - accuracy: 0.3584\n",
            "Epoch 25/100\n",
            "13/13 - 0s - loss: 2.3080 - accuracy: 0.3634\n",
            "Epoch 26/100\n",
            "13/13 - 0s - loss: 2.2714 - accuracy: 0.3559\n",
            "Epoch 27/100\n",
            "13/13 - 0s - loss: 2.2268 - accuracy: 0.3784\n",
            "Epoch 28/100\n",
            "13/13 - 0s - loss: 2.1757 - accuracy: 0.3860\n",
            "Epoch 29/100\n",
            "13/13 - 0s - loss: 2.1359 - accuracy: 0.4085\n",
            "Epoch 30/100\n",
            "13/13 - 0s - loss: 2.0971 - accuracy: 0.4060\n",
            "Epoch 31/100\n",
            "13/13 - 0s - loss: 2.0485 - accuracy: 0.4361\n",
            "Epoch 32/100\n",
            "13/13 - 0s - loss: 1.9970 - accuracy: 0.4536\n",
            "Epoch 33/100\n",
            "13/13 - 0s - loss: 1.9747 - accuracy: 0.4361\n",
            "Epoch 34/100\n",
            "13/13 - 0s - loss: 1.9474 - accuracy: 0.4762\n",
            "Epoch 35/100\n",
            "13/13 - 0s - loss: 1.8810 - accuracy: 0.5013\n",
            "Epoch 36/100\n",
            "13/13 - 0s - loss: 1.8363 - accuracy: 0.4937\n",
            "Epoch 37/100\n",
            "13/13 - 0s - loss: 1.8069 - accuracy: 0.4887\n",
            "Epoch 38/100\n",
            "13/13 - 0s - loss: 1.7465 - accuracy: 0.5439\n",
            "Epoch 39/100\n",
            "13/13 - 0s - loss: 1.7111 - accuracy: 0.5363\n",
            "Epoch 40/100\n",
            "13/13 - 0s - loss: 1.6691 - accuracy: 0.5439\n",
            "Epoch 41/100\n",
            "13/13 - 0s - loss: 1.6283 - accuracy: 0.5890\n",
            "Epoch 42/100\n",
            "13/13 - 0s - loss: 1.5792 - accuracy: 0.5890\n",
            "Epoch 43/100\n",
            "13/13 - 0s - loss: 1.5408 - accuracy: 0.5789\n",
            "Epoch 44/100\n",
            "13/13 - 0s - loss: 1.5115 - accuracy: 0.6266\n",
            "Epoch 45/100\n",
            "13/13 - 0s - loss: 1.4671 - accuracy: 0.6216\n",
            "Epoch 46/100\n",
            "13/13 - 0s - loss: 1.4219 - accuracy: 0.6441\n",
            "Epoch 47/100\n",
            "13/13 - 0s - loss: 1.3813 - accuracy: 0.6566\n",
            "Epoch 48/100\n",
            "13/13 - 0s - loss: 1.3519 - accuracy: 0.6617\n",
            "Epoch 49/100\n",
            "13/13 - 0s - loss: 1.3258 - accuracy: 0.6566\n",
            "Epoch 50/100\n",
            "13/13 - 0s - loss: 1.2985 - accuracy: 0.6591\n",
            "Epoch 51/100\n",
            "13/13 - 0s - loss: 1.2353 - accuracy: 0.7018\n",
            "Epoch 52/100\n",
            "13/13 - 0s - loss: 1.2016 - accuracy: 0.7043\n",
            "Epoch 53/100\n",
            "13/13 - 0s - loss: 1.1379 - accuracy: 0.7343\n",
            "Epoch 54/100\n",
            "13/13 - 0s - loss: 1.0989 - accuracy: 0.7494\n",
            "Epoch 55/100\n",
            "13/13 - 0s - loss: 1.0592 - accuracy: 0.7594\n",
            "Epoch 56/100\n",
            "13/13 - 0s - loss: 1.0308 - accuracy: 0.7619\n",
            "Epoch 57/100\n",
            "13/13 - 0s - loss: 1.0027 - accuracy: 0.7845\n",
            "Epoch 58/100\n",
            "13/13 - 0s - loss: 0.9703 - accuracy: 0.7920\n",
            "Epoch 59/100\n",
            "13/13 - 0s - loss: 0.9363 - accuracy: 0.8045\n",
            "Epoch 60/100\n",
            "13/13 - 0s - loss: 0.8954 - accuracy: 0.8271\n",
            "Epoch 61/100\n",
            "13/13 - 0s - loss: 0.8719 - accuracy: 0.8521\n",
            "Epoch 62/100\n",
            "13/13 - 0s - loss: 0.8322 - accuracy: 0.8521\n",
            "Epoch 63/100\n",
            "13/13 - 0s - loss: 0.7890 - accuracy: 0.8672\n",
            "Epoch 64/100\n",
            "13/13 - 0s - loss: 0.7668 - accuracy: 0.8822\n",
            "Epoch 65/100\n",
            "13/13 - 0s - loss: 0.7494 - accuracy: 0.8747\n",
            "Epoch 66/100\n",
            "13/13 - 0s - loss: 0.7148 - accuracy: 0.9098\n",
            "Epoch 67/100\n",
            "13/13 - 0s - loss: 0.6822 - accuracy: 0.9073\n",
            "Epoch 68/100\n",
            "13/13 - 0s - loss: 0.6544 - accuracy: 0.9198\n",
            "Epoch 69/100\n",
            "13/13 - 0s - loss: 0.6296 - accuracy: 0.9348\n",
            "Epoch 70/100\n",
            "13/13 - 0s - loss: 0.6128 - accuracy: 0.9223\n",
            "Epoch 71/100\n",
            "13/13 - 0s - loss: 0.6081 - accuracy: 0.9298\n",
            "Epoch 72/100\n",
            "13/13 - 0s - loss: 0.5728 - accuracy: 0.9474\n",
            "Epoch 73/100\n",
            "13/13 - 0s - loss: 0.5533 - accuracy: 0.9323\n",
            "Epoch 74/100\n",
            "13/13 - 0s - loss: 0.5267 - accuracy: 0.9474\n",
            "Epoch 75/100\n",
            "13/13 - 0s - loss: 0.5045 - accuracy: 0.9474\n",
            "Epoch 76/100\n",
            "13/13 - 0s - loss: 0.4812 - accuracy: 0.9624\n",
            "Epoch 77/100\n",
            "13/13 - 0s - loss: 0.4549 - accuracy: 0.9674\n",
            "Epoch 78/100\n",
            "13/13 - 0s - loss: 0.4358 - accuracy: 0.9774\n",
            "Epoch 79/100\n",
            "13/13 - 0s - loss: 0.4212 - accuracy: 0.9749\n",
            "Epoch 80/100\n",
            "13/13 - 0s - loss: 0.4101 - accuracy: 0.9674\n",
            "Epoch 81/100\n",
            "13/13 - 0s - loss: 0.3980 - accuracy: 0.9799\n",
            "Epoch 82/100\n",
            "13/13 - 0s - loss: 0.3741 - accuracy: 0.9799\n",
            "Epoch 83/100\n",
            "13/13 - 0s - loss: 0.3634 - accuracy: 0.9799\n",
            "Epoch 84/100\n",
            "13/13 - 0s - loss: 0.3511 - accuracy: 0.9825\n",
            "Epoch 85/100\n",
            "13/13 - 0s - loss: 0.3317 - accuracy: 0.9825\n",
            "Epoch 86/100\n",
            "13/13 - 0s - loss: 0.3158 - accuracy: 0.9875\n",
            "Epoch 87/100\n",
            "13/13 - 0s - loss: 0.3036 - accuracy: 0.9925\n",
            "Epoch 88/100\n",
            "13/13 - 0s - loss: 0.2982 - accuracy: 0.9900\n",
            "Epoch 89/100\n",
            "13/13 - 0s - loss: 0.2799 - accuracy: 0.9900\n",
            "Epoch 90/100\n",
            "13/13 - 0s - loss: 0.2716 - accuracy: 0.9925\n",
            "Epoch 91/100\n",
            "13/13 - 0s - loss: 0.2608 - accuracy: 0.9925\n",
            "Epoch 92/100\n",
            "13/13 - 0s - loss: 0.2522 - accuracy: 0.9925\n",
            "Epoch 93/100\n",
            "13/13 - 0s - loss: 0.2461 - accuracy: 0.9875\n",
            "Epoch 94/100\n",
            "13/13 - 0s - loss: 0.2350 - accuracy: 0.9875\n",
            "Epoch 95/100\n",
            "13/13 - 0s - loss: 0.2250 - accuracy: 0.9900\n",
            "Epoch 96/100\n",
            "13/13 - 0s - loss: 0.2148 - accuracy: 0.9925\n",
            "Epoch 97/100\n",
            "13/13 - 0s - loss: 0.2039 - accuracy: 0.9900\n",
            "Epoch 98/100\n",
            "13/13 - 0s - loss: 0.1952 - accuracy: 0.9950\n",
            "Epoch 99/100\n",
            "13/13 - 0s - loss: 0.1891 - accuracy: 0.9900\n",
            "Epoch 100/100\n",
            "13/13 - 0s - loss: 0.1815 - accuracy: 0.9950\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FE_42ireZAza"
      },
      "source": [
        "##**18.5 Generate Text**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MG86Olfh_vki"
      },
      "source": [
        "###**18.5.1 Load Model**\n",
        "\n",
        "The first step is to load the model saved to the file model.h5. We can use the load model()\n",
        "function from the Keras API."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "57j2LSQGZAdt"
      },
      "source": [
        "# load the model\n",
        "model = load_model('model.h5')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O3vhFeMDAOqQ"
      },
      "source": [
        "We also need to load the pickled dictionary for mapping characters to integers from the file\n",
        "mapping.pkl. We will use the Pickle API to load the object."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "amyrhsvPUkh9"
      },
      "source": [
        "# load the mapping\n",
        "mapping = load(open('mapping.pkl','rb'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OlYuIrBiA4I8"
      },
      "source": [
        "###**18.5.2 Generate Characters**\n",
        "\n",
        "We must provide sequences of 10 characters as input to the model in order to start the generation\n",
        "process. We will pick these manually. A given input sequence will need to be prepared in the\n",
        "same way as preparing the training data for the model. First, the sequence of characters must\n",
        "be integer encoded using the loaded mapping.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j1OPsA5kBsQW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "dd127f62-0c5f-478d-bdde-84b4ce8572fa"
      },
      "source": [
        "# encode the charactersas integers\n",
        "in_text = 'Sing a son'\n",
        "encoded = [mapping[char] for char in in_text]\n",
        "encoded"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[12, 23, 27, 21, 1, 15, 1, 32, 28, 27]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ulGef1VRB8iw"
      },
      "source": [
        "Next, the integers need to be one hot encoded using the to categorical() Keras function.\n",
        "We also need to reshape the sequence to be 3-dimensional, as we only have one sequence and\n",
        "LSTMs require all input to be three dimensional (samples, time steps, features)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qDt18pmNB90P"
      },
      "source": [
        "# one hot encode\n",
        "encoded = to_categorical(encoded, num_classes=len(mapping))\n",
        "encoded = encoded.reshape(1, encoded.shape[0], encoded.shape[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KwcoU61-CevY"
      },
      "source": [
        "We can then use the model to predict the next character in the sequence. We use\n",
        "predict_classes() instead of predict() to directly select the integer for the character with\n",
        "the highest probability instead of getting the full probability distribution across the entire set of\n",
        "characters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C2VR4YchCwvo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "9f7daa49-0a03-4fd7-d72a-c24ebca25123"
      },
      "source": [
        "# predict character\n",
        "yhat = model.predict_classes(encoded, verbose=0)\n",
        "yhat"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([21])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iybL3EwPDNOZ"
      },
      "source": [
        "We can then decode this integer by looking up the mapping to see the character to which it\n",
        "maps"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6ij5Tofl8wdj"
      },
      "source": [
        "out_char = ''\n",
        "for char, index in mapping.items():\n",
        "    if index == yhat:\n",
        "        out_char = char\n",
        "        break"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "653Om6eFlyfO"
      },
      "source": [
        "This character can then be added to the input sequence. We then need to make sure that the\n",
        "input sequence is 10 characters by truncating the first character from the input sequence text.\n",
        "We can use the pad sequences() function from the Keras API that can perform this truncation\n",
        "operation. Putting all of this together, we can define a new function named generate seq()\n",
        "for using the loaded model to generate new sequences of text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hWmIuSdXcCKC"
      },
      "source": [
        "# generate a sequence of characters with a language model\n",
        "def generate_seq(model, mapping, seq_length, seed_text, n_chars):\n",
        "  in_text = seed_text\n",
        "  # generate a fixed number of characters\n",
        "  for _ in range(n_chars):\n",
        "    # encode the characters as integers\n",
        "    encoded = [mapping[char] for char in in_text]\n",
        "    # truncate sequence to a fixed length\n",
        "    encoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')\n",
        "    #one hot encode\n",
        "    encoded = to_categorical(encoded, num_classes=len(mapping))\n",
        "    encoded = encoded.reshape(1, encoded.shape[0], encoded.shape[1])\n",
        "    # predict charater\n",
        "    yhat = model.predict_classes(encoded, verbose=0)\n",
        "    #reverse map integer to character\n",
        "    out_char = ' '\n",
        "    for char, index in mapping.items():\n",
        "      if index == yhat:\n",
        "        out_char = char\n",
        "        break\n",
        "    # append to input\n",
        "    in_text +=char\n",
        "    return in_text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "53zWxPjattcN",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "019517e3-f585-4133-eebc-7d3736875bbb"
      },
      "source": [
        "from keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "mapping = mapping\n",
        "print(len(mapping))\n",
        "seq_length = 10\n",
        "seed_text = 'Sing a son' \n",
        "n_chars = 20\n",
        "in_text = seed_text\n",
        "# generate a fixed number of characters\n",
        "for _ in range(n_chars):\n",
        "  # encode the characters as integers\n",
        "  encoded = [mapping[char] for char in in_text]\n",
        "  # truncate sequence to a fixed length\n",
        "  encoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')\n",
        "  #one hot encode\n",
        "  encoded = to_categorical(encoded, num_classes=len(mapping))\n",
        "  encoded = encoded.reshape(1, encoded.shape[1], encoded.shape[2])\n",
        "  # predict charater\n",
        "  yhat = model.predict_classes(encoded, verbose=0)\n",
        "  #reverse map integer to character\n",
        "  out_char = ' '\n",
        "  for char, index in mapping.items():\n",
        "    if index == yhat:\n",
        "      out_char = char\n",
        "      break\n",
        "  # append to input\n",
        "  in_text +=char"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "38\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cdm182sprOFl"
      },
      "source": [
        "###**18.5.3 Complete Example**\n",
        "Tying all of this together, the complete example for generating text using the fit neural language\n",
        "model is listed below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xb3y7PEEq8SR",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "fe96b4ce-252c-4383-c95f-f8542474c75a"
      },
      "source": [
        "from pickle import load\n",
        "from keras.models import load_model\n",
        "from keras.utils import to_categorical\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "# generate a sequence of characters with a language model\n",
        "def generate_seq(model, mapping, seq_length, seed_text, n_chars):\n",
        "    in_text = seed_text\n",
        "    # generate a fixed number of characters\n",
        "    for _ in range(n_chars):\n",
        "        # encode the characters as integers\n",
        "        encoded = [mapping[char] for char in in_text]\n",
        "        # truncate sequences to a fixed length\n",
        "        encoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')\n",
        "        # one hot encode\n",
        "        encoded = to_categorical(encoded, num_classes=len(mapping))\n",
        "        encoded = encoded.reshape(1, encoded.shape[1], encoded.shape[2])\n",
        "        # predict character\n",
        "        yhat = model.predict_classes(encoded, verbose=0)\n",
        "        # reverse map integer to character\n",
        "        out_char = ''\n",
        "        for char, index in mapping.items():\n",
        "            if index == yhat:\n",
        "                out_char = char\n",
        "                break\n",
        "        # append to input\n",
        "        in_text += out_char\n",
        "    return in_text\n",
        "\n",
        "# load the model\n",
        "model = load_model('model.h5')\n",
        "# load the mapping\n",
        "mapping = load(open('mapping.pkl', 'rb'))\n",
        "# test start of rhyme\n",
        "print(generate_seq(model, mapping, 10, 'Sing a son', 20))\n",
        "# test mid-line\n",
        "print(generate_seq(model, mapping, 10, 'The birds began', 20))\n",
        "# test not in original\n",
        "print(generate_seq(model, mapping, 10, 'When down came', 20))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sing a song of sixpence, A poc\n",
            "The birds began to sing; Wasn't tha\n",
            "When down came a blackbird And pec\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MEzOF_1W-HDd"
      },
      "source": [
        "#**Chapter 19 How to Develop a Word-Based Neural Language Model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8d34uder_kfo"
      },
      "source": [
        "##**19.1 Tutorial Overview**\n",
        "This tutorial is divided into the following parts:\n",
        "1. Framing Language Modeling\n",
        "2. Jack and Jill Nursery Rhyme\n",
        "3. Model 1: One-Word-In, One-Word-Out Sequences\n",
        "4. Model 2: Line-by-Line Sequence\n",
        "5. Model 3: Two-Words-In, One-Word-Out Sequence"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "inKajprb_3Vo"
      },
      "source": [
        "##**19.3 Jack and Jill Nursery Rhyme**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dBtDYIDC_yru",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "e5088258-e70d-45a8-fd66-26497134cb2e"
      },
      "source": [
        "# source text\n",
        "data = \"\"\" Jack and Jill went up the hill\\n\n",
        "To fetch a pail of water\\n\n",
        "Jack fell down and broke his crown\\n\n",
        "And Jill came tumbling after\\n \"\"\"\n",
        "data"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "' Jack and Jill went up the hill\\n\\nTo fetch a pail of water\\n\\nJack fell down and broke his crown\\n\\nAnd Jill came tumbling after\\n '"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8e9KbPFYATEM"
      },
      "source": [
        "##**19.4 Model 1: One-Word-In, One-Word-Out Sequences**\n",
        "\n",
        "We can start with a very simple model. Given one word as input, the model will learn to predict\n",
        "the next word in the sequence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KByCJPWICcng"
      },
      "source": [
        "from numpy import array\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.utils import to_categorical\n",
        "from keras.utils.vis_utils import plot_model\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import LSTM\n",
        "from keras.layers import Embedding\n",
        "# generate a sequence from the model\n",
        "def generate_seq(model, tokenizer, seed_text, n_words):\n",
        "    in_text, result = seed_text, seed_text\n",
        "    # generate a fixed number of words\n",
        "    for _ in range(n_words):\n",
        "        # encode the text as integer\n",
        "        encoded = tokenizer.texts_to_sequences([in_text])[0]\n",
        "        encoded = array(encoded)\n",
        "        # predict a word in the vocabulary\n",
        "        yhat = model.predict_classes(encoded, verbose=0)\n",
        "        # map predicted word index to word\n",
        "        out_word = ''\n",
        "        for word, index in tokenizer.word_index.items():\n",
        "            if index == yhat:\n",
        "                out_word = word\n",
        "                break\n",
        "        # append to input\n",
        "        in_text, result = out_word, result + ' ' + out_word\n",
        "    return result\n",
        "# define the model\n",
        "def define_model(vocab_size):\n",
        "    model = Sequential()\n",
        "    model.add(Embedding(vocab_size, 10, input_length=1))\n",
        "    model.add(LSTM(50))\n",
        "    model.add(Dense(vocab_size, activation='softmax'))\n",
        "    # compile network\n",
        "    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "    # summarize defined model\n",
        "    model.summary()\n",
        "    plot_model(model, to_file='model.png', show_shapes=True)\n",
        "    return model\n",
        "# source text\n",
        "data = \"\"\"Jack and Jill went up the hill\\n\n",
        "          To fetch a pail of water\\n\n",
        "          Jack fell down and broke his crown\\n\n",
        "          And Jill came tumbling after\\n \"\"\"\n",
        "# integer encode text\n",
        "tokenizer = Tokenizer()\n",
        "tokenizer.fit_on_texts([data])\n",
        "encoded = tokenizer.texts_to_sequences([data])[0]\n",
        "# determine the vocabulary size\n",
        "vocab_size = len(tokenizer.word_index) + 1\n",
        "print('Vocabulary Size: %d' % vocab_size)\n",
        "# create word -> word sequences\n",
        "sequences = list()\n",
        "for i in range(1, len(encoded)):\n",
        "    sequence = encoded[i-1:i+1]\n",
        "    sequences.append(sequence)\n",
        "print('Total Sequences: %d' % len(sequences))\n",
        "# split into X and y elements\n",
        "sequences = array(sequences)\n",
        "X, y = sequences[:,0],sequences[:,1]\n",
        "# one hot encode outputs\n",
        "y = to_categorical(y, num_classes=vocab_size)\n",
        "# define model\n",
        "model = define_model(vocab_size)\n",
        "# fit network\n",
        "model.fit(X, y, epochs=500, verbose=2)\n",
        "# evaluate\n",
        "print(generate_seq(model, tokenizer, 'Jack', 6))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9nlWEy1oNZly"
      },
      "source": [
        "##**19.5 Model 2: Line-by-Line Sequence**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jvBewLycokpc"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BPmyQvu37Bhe"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b7BlS-Xu7CIM"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GbRZExCL7CCG"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E2rbGIP57B8r"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OZOHMvWQ7B2c"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SeuEuely7Bwi"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KZ7z5dAh7Bqu"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eLsYTebG7AVK"
      },
      "source": [
        "#**Part VIII   Image Captioning**\n",
        "\n",
        "#**Chapter 21: Neural Image Caption Generation**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ijbhp3I97pLJ"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gaWi_binsZsN"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d5VDgTj6sZoG"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-neF9tzSsZj-"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DssNNKtrsZIt"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2oLdOw09salW"
      },
      "source": [
        "#**Chapter 26 Project: Develop a Neural   Image Caption Generation Model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0-zjba-ZtHfs"
      },
      "source": [
        "##**26.1 Tutorial Overview**\n",
        "This tutorial is divided into the following parts:\n",
        "1. Photo and Caption Dataset\n",
        "2. Prepare Photo Data\n",
        "3. Prepare Text Data\n",
        "4. Develop Deep Learning Model\n",
        "5. Evaluate Model\n",
        "6. Generate New Captions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y7iEPuUZtTMw"
      },
      "source": [
        "##**26.2 Photo and Caption Dataset**\n",
        "\n",
        "You can use the link below to request\n",
        "the dataset:\n",
        "\n",
        "* Dataset Request Form.\n",
        "https://illinois.edu/fb/sec/1713398\n",
        "Within a short time, you will receive an email that contains links to two files:\n",
        "* Flickr8k Dataset.zip (1 Gigabyte) An archive of all photographs.\n",
        "* Flickr8k text.zip (2.2 Megabytes) An archive of all text descriptions for photographs.\n",
        "\n",
        "* Flicker8k Dataset: Contains 8092 photographs in JPEG format (yes the directory name\n",
        "spells it ‘Flicker’ not ‘Flickr’).\n",
        "* Flickr8k text: Contains a number of files containing different sources of descriptions for\n",
        "the photographs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0v6SqdvevaL7"
      },
      "source": [
        "##**26.3 Prepare Photo Data**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "THy6d1posmsL"
      },
      "source": [
        "from os import listdir\n",
        "from os import path\n",
        "from pickle import dump\n",
        "from keras.applications.vgg16 import VGG16\n",
        "from keras.preprocessing.image import load_img\n",
        "from keras.preprocessing.image import img_to_array\n",
        "from keras.applications.vgg16 import preprocess_input\n",
        "from keras.models import Model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oYl2rnq6MYfZ"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}