Embeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Embeddings.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [
        "copyright"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "copyright"
      },
      "source": [
        "#### Copyright 2020 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "24p97VuTvYVT"
      },
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CVmV0M74xwm7"
      },
      "source": [
        "# Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XuFjSsW53I9_"
      },
      "source": [
        "Embeddings are a powerful way to represent data for deep learning models.\n",
        "\n",
        "In this exercise we will work specifically with embeddings of words. This is one of the most common applications of this technique. We will train word embeddings from scratch on a small dataset and visualize those embeddings."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LboFaJ47r6x-"
      },
      "source": [
        "Embeddings can be trained using either supervised or unsupervised learning. For this exercise we will train a supervised sentiment classifier on IMDB movie reviews. We will learn our own word embeddings in the process."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3NSkp7Qw6aJ4"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zu7rUHNXWBue"
      },
      "source": [
        "First we will set random seeds for reproducible results. Controlling which random numbers are generated ensures we get the same results every time we run this lab. **This should almost never be used in production code.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uI4PxR8OcADw"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "np.random.seed(42)\n",
        "tf.random.set_seed(42)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4LloAqop6WeT"
      },
      "source": [
        "## The Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1zoVKdwCZTK5"
      },
      "source": [
        "We will use the [IMDB dataset](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb) that contains the text of 50,000 movie reviews from the [Internet Movie Database](https://www.imdb.com/). This dataset is conveniently packaged in TensorFlow Keras. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xhd_d5vzrc_h"
      },
      "source": [
        "from tensorflow import keras\n",
        "\n",
        "imdb = keras.datasets.imdb\n",
        "\n",
        "(train_data, train_labels), (test_data, test_labels) = imdb.load_data()\n",
        "\n",
        "train_data.shape, train_labels.shape, test_data.shape, test_labels.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vr02kdK2aHnJ"
      },
      "source": [
        "The reviews have been pre-processed, so the text of each review (a list of words) has been converted to a list of integers, and each integer represents a specific word."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KERrbc1Xarhj"
      },
      "source": [
        "print(train_data[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NVQdpZrVDd1u"
      },
      "source": [
        "There are also some special placeholder characters. For instance, each review starts with a `start_char` placeholder that defaults to a value of `1`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BQnnoFoeCsK-"
      },
      "source": [
        "{review[0]: 'start_char' for review in train_data}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HA8VWNfy6j1B"
      },
      "source": [
        "Each sentiment label is an integer value of either 0 or 1, where 0 is a negative review, and 1 is a positive review. Thus this is a binary classification problem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0huU4bUD7O4p"
      },
      "source": [
        "train_labels[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dAQyOxlg9o_v"
      },
      "source": [
        "### Word Index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Goq0Frawyuk"
      },
      "source": [
        "It is useful to be able to convert a list of integers back to text. The `imdb.get_word_index()` returns a dictionary of all of the words in the reviews. The words have a numeric value that represents their frequency in the dataset.\n",
        "\n",
        "For instance, if we wanted to find the `25` most common words in the reviews, we could use the code below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hGGowYyG7T0x"
      },
      "source": [
        "top_25 = ['' for _ in range(25)]\n",
        "for k, v  in imdb.get_word_index().items():\n",
        "  if v <= 25:\n",
        "    top_25[v-1] = k\n",
        "\n",
        "top_25"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ryk0Bdq7-Mqk"
      },
      "source": [
        "You can see that these words probably aren't helpful for predicting sentiment. The `skip_top=n` argument can be passed to [`load_data`](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb/load_data) to skip over the `n` most common words.\n",
        "\n",
        "Let's reload our data using `skip_top`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kbdqmCzt932v"
      },
      "source": [
        "from tensorflow import keras\n",
        "\n",
        "imdb = keras.datasets.imdb\n",
        "\n",
        "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(\n",
        "    skip_top=25,\n",
        ")\n",
        "\n",
        "train_data.shape, train_labels.shape, test_data.shape, test_labels.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HUdkPjm3_mUL"
      },
      "source": [
        "Our data now doesn't include the top `25` words in the reviews. That begs the question: what happens to these words? Are they removed? Are they replaced?\n",
        "\n",
        "It turns out they are replaced by a placeholder value. This value is the `oov_char`, which by default is `2`.\n",
        "\n",
        "If we print out the first training review we can see that the top `25` most common words are now replaced with `2`, including the `start_char`!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yn8jnFELAUgs"
      },
      "source": [
        "train_data[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r9MTktjo95EN"
      },
      "source": [
        "Let's create a function that can convert a numeric review into text so we can read the reviews ourselves.\n",
        "\n",
        "To do this we need to modify the word index a bit. The index from Keras starts at `1`, but we know that the `start_char` is `1` and that the `oov_char` is `2`. We need to shift the indexes by a few values.\n",
        "\n",
        "The `load_data` function has an argument called `index_from` that defaults to `3`. The reviews are indexed with the first word starting at `3`, while the word index has the first word starting at `1`. Let's fix that and add placeholders for start, oov, and a padding character we'll use to make the reviews equal-sized for modeling."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LfcdK_js_dQd"
      },
      "source": [
        "# A dictionary mapping each word to an integer index.\n",
        "word_index = imdb.get_word_index()\n",
        "\n",
        "# Reserve the first 3 indices for special tokens.\n",
        "word_index = {k:(v+3) for k,v in word_index.items()}\n",
        "word_index[\"<PAD>\"] = 0\n",
        "word_index[\"<START>\"] = 1\n",
        "word_index[\"<OOV>\"] = 2\n",
        "\n",
        "vocab_size = len(word_index)\n",
        "print('# unique words: {}'.format(vocab_size))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_1-v7LMQFgPV"
      },
      "source": [
        "And now let's try to decode a review."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y5fVfsydTeCB"
      },
      "source": [
        "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
        "\n",
        "def decode_review(text):\n",
        "    return ' '.join([reverse_word_index.get(i, '?') for i in text])\n",
        "\n",
        "decode_review(train_data[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u617UoBvzI_A"
      },
      "source": [
        "### Preprocessing\n",
        "\n",
        "Movie reviews can be different lengths, but inputs to a neural network must all be the same length. Let's take a look at the distribution of review lengths."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i5oIPHvKMqvt"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(5,5))\n",
        "review_lengths = [len(review) for review in train_data]\n",
        "plt.hist(review_lengths, density=True, cumulative=True)\n",
        "plt.yticks(np.arange(0, 1.1, step=0.1))\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QyUvPXstTup9"
      },
      "source": [
        "Even though the longest review is close to `2500` words long, many of the reviews have fewer than 500 words. \n",
        "\n",
        "We can use the `pad_sequences` function to standardize the lengths of the reviews to `500` words long. Any reviews longer than this will have the extra words truncated, while any reviews shorter than `500` words will have extra `\"<PAD>\"` tokens added to the end. \n",
        "\n",
        "Choosing a standardized length requires balancing efficiency (longer lengths mean slower training) and information loss (shorter lengths may truncate too much valuable information). Aiming for a length that fully covers `90%` of samples is generally reasonable, and you can further tune this as a hyperparameter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nX-sWA3Z2uu3"
      },
      "source": [
        "maxlen = 500\n",
        "\n",
        "train_data = keras.preprocessing.sequence.pad_sequences(\n",
        "    train_data, value=word_index[\"<PAD>\"], padding='post', maxlen=maxlen)\n",
        "\n",
        "test_data = keras.preprocessing.sequence.pad_sequences(\n",
        "    test_data, value=word_index[\"<PAD>\"], padding='post', maxlen=maxlen)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r8Uv3iI725Mb"
      },
      "source": [
        "Let's inspect the first padded review."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O-6VBoda3agD"
      },
      "source": [
        "print(decode_review(train_data[0]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NW1OYV_piA5M"
      },
      "source": [
        "## Build the Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B7Cl6fRyDuDI"
      },
      "source": [
        "### Using the Embedding Layer\n",
        "\n",
        "Keras makes it easy to use embeddings. The [Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding) layer takes at least two arguments: the number of possible words in the vocabulary and the dimensionality of the embeddings. We will start by using a small embedding size of 2 to make visualization easier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bsve7z4TFJyk"
      },
      "source": [
        "embedding_dim = 2\n",
        "\n",
        "embedding_layer = keras.layers.Embedding(vocab_size, embedding_dim)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ucstp6tBFt94"
      },
      "source": [
        "The Embedding layer can be understood as a lookup table that maps from integer indices, which stand for specific words, to dense vectors (their embeddings). The dimensionality, or width, of the embedding is a parameter you can experiment with to see what works well for your problem, much in the same way you would experiment with the number of neurons in a Dense layer.\n",
        "\n",
        "When we create an Embedding layer, the weights for the embedding are randomly initialized just like any other layer. During training, they are gradually adjusted via backpropagation. Once trained, the learned word embeddings will roughly encode similarities between words (as they were learned for the specific problem our model is trained on).\n",
        "\n",
        "As input, the Embedding layer takes a 2D tensor of integers, of shape `(num_samples, sequence_length)`, where each sample is a sequence of integers. As output, the embedding layer returns a 3D floating point tensor, of shape `(num_samples, sequence_length, embedding_dimensionality)`. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w9Clq0KWL2dD"
      },
      "source": [
        "### Configure the Model\n",
        "\n",
        "1. The first layer is an Embedding layer. This layer takes the integer-encoded vocabulary and looks up the embedding vector for each word index. These vectors are learned as the model trains. The vectors add a dimension to the output array. The resulting dimensions are: `(batch size, sequence length, embedding size)`.\n",
        "\n",
        "1. Next, we flatten the output from a 2-d array to a 1-d array.\n",
        "\n",
        "1. The last layer is densely connected with a single output node. Using the sigmoid activation function, this value is a float between 0 and 1, representing a probability, or confidence level, that the review is positive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3OnpePqclffu"
      },
      "source": [
        "model = keras.Sequential([\n",
        "  keras.layers.Embedding(vocab_size, embedding_dim, input_length=maxlen),\n",
        "  keras.layers.Flatten(),\n",
        "  keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f8DaPLm2tki9"
      },
      "source": [
        "### Compile and Train the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Itc2oZQMnz7t"
      },
      "source": [
        "model.compile(\n",
        "  # Calculate loss for a binary classification problem.\n",
        "  loss='binary_crossentropy',\n",
        "\n",
        "  # Adam is one of the most commonly used optimizers.\n",
        "  optimizer=tf.keras.optimizers.Adam(),\n",
        "\n",
        "  # We will only track accuracy for this task.\n",
        "  metrics=['accuracy']\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ysiCSDsVn2SC"
      },
      "source": [
        "The number of epochs and batch size used during training are both hyperparameters, which means you may need to experiment with different values to improve performance. This also means there's no magic answer for which values to choose.\n",
        "\n",
        "If you've implemented your model correctly, you should observe decreasing training loss within a few epochs. With more epochs (more training), the model will experience more overfitting. You will typically train for at least 5 epochs. For this model, we train for 10 epochs as a reasonable tradeoff between learning (as the validation accuracy is still increasing) and overfitting.\n",
        "\n",
        "Since we have a fairly large dataset, we also want to process data in batches instead of the entire dataset at once. If we choose batch sizes that are too small, we will get slower training per epoch, while larger batch sizes may require more epochs to train and even cause out-of-memory errors. It's good practice to start with a small power of two (e.g., 32), and then experiment with continuously doubling your batch size. For this model, we are able to use a batch size of 512 with good performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mPekaK45pdOc"
      },
      "source": [
        "history = model.fit(\n",
        "    train_data,\n",
        "    train_labels,\n",
        "    epochs=10,\n",
        "    batch_size=512,\n",
        "    validation_split=0.2\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUh9kTq5HxoG"
      },
      "source": [
        "Let's visualize the training and validation accuracy over time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VFcv3PYzJwjl"
      },
      "source": [
        "acc = history.history['accuracy']\n",
        "val_acc = history.history['val_accuracy']\n",
        "\n",
        "epochs = range(1, len(acc) + 1)\n",
        "\n",
        "plt.figure(figsize=(12,9))\n",
        "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
        "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.legend(loc='lower right')\n",
        "plt.ylim((0.5,1))\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ODIrlPDiJrDP"
      },
      "source": [
        "With this approach, our model reaches a validation accuracy of around 87%. (Note that the model is already beginning to overfit, as reflected in the diverging training and validation accuracy curves.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uLCSnMlwJy3j"
      },
      "source": [
        "## Visualize Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PMqbq-qMnCR7"
      },
      "source": [
        "### Retrieve the Learned Embeddings\n",
        "\n",
        "Next, let's retrieve the word embeddings learned during training. This will be a matrix of shape `(vocab size, embedding dimension)`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KncdCMmgnHtU"
      },
      "source": [
        "e = model.layers[0]\n",
        "embedding_matrix = e.get_weights()[0]\n",
        "print(embedding_matrix.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ilLX1zdS4PIn"
      },
      "source": [
        "Looking at all 80,000+ at once would be overwhelming, so let's explore the learned word embeddings on a small set of terms. Most of these terms are generally strongly indicative of sentiment, but some are added just for fun."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D3nzbrDtp1VO"
      },
      "source": [
        "informative_terms = [ \"bad\", \"great\", \"best\", \"worst\", \"fun\", \"beautiful\",\n",
        "                      \"excellent\", \"poor\", \"boring\", \"awful\", \"terrible\",\n",
        "                      \"definitely\", \"perfect\", \"liked\", \"worse\", \"waste\",\n",
        "                      \"entertaining\", \"loved\", \"unfortunately\", \"amazing\",\n",
        "                      \"enjoyed\", \"favorite\", \"horrible\", \"brilliant\", \"highly\",\n",
        "                      \"simple\", \"annoying\", \"today\", \"hilarious\", \"enjoyable\",\n",
        "                      \"dull\", \"fantastic\", \"poorly\", \"fails\", \"disappointing\",\n",
        "                      \"disappointment\", \"not\", \"him\", \"her\", \"good\", \"time\",\n",
        "                      \"sad\", \"exciting\", \"slow\", \"movie\", \"film\", \"action\",\n",
        "                      \"comedy\", \"drama\", \"fabulous\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "axURTCgJwOjY"
      },
      "source": [
        "Now we plot each word in `informative_terms` on a two-dimensional graph. Recall that we are using two-dimensional embeddings, so consider the first value as an x-value and the second value as a y-value. \n",
        "\n",
        "*Note: If the displayed plot is too small, try running the cell again.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p2kNso7P6fem"
      },
      "source": [
        "for word in informative_terms:\n",
        "  word_num = word_index[word]\n",
        "  embeddings = embedding_matrix[word_num]\n",
        "  plt.text(embeddings[0], embeddings[1], word)\n",
        "\n",
        "# Do a little set-up to make sure the plot displays nicely.\n",
        "plt.rcParams[\"figure.figsize\"] = (25, 25)\n",
        "plt.xlim(1.2 * embedding_matrix.min(), 1.2 * embedding_matrix.max())\n",
        "plt.ylim(1.2 * embedding_matrix.min(), 1.2 * embedding_matrix.max())\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M89j_dhuIGJ2"
      },
      "source": [
        "We can see that positive words tend to cluster around each, other and negative words tend to cluster around each other."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fxE73zG6IPys"
      },
      "source": [
        "## Final Test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ANB1E92IScT"
      },
      "source": [
        "Let's see how our model performs on the holdout dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K2onqLu0IZgH"
      },
      "source": [
        "predictions = model.predict(test_data)\n",
        "predictions"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rYwpxe1PIhJW"
      },
      "source": [
        "The prediction values that we get are floating point numbers, but we want to convert these to a prediction of `0` for negative and `1` for positive. One way to do this is to simply round."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ucV1VqAEIq1A"
      },
      "source": [
        "predictions = [int(round(prediction)) for prediction in predictions.flatten()]\n",
        "predictions[:10]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "laJ_ZjG6I3zj"
      },
      "source": [
        "And now we can measure our model quality."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vOAso2BBI76X"
      },
      "source": [
        "precision = tf.keras.metrics.Precision()\n",
        "precision.update_state(test_labels, predictions)\n",
        "\n",
        "recall = tf.keras.metrics.Recall()\n",
        "recall.update_state(test_labels, predictions)\n",
        "\n",
        "print('Precision: {}, Recall: {}'.format(\n",
        "    precision.result().numpy(), recall.result().numpy()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gE8uZeJwWsqC"
      },
      "source": [
        "# Exercises"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hIx-sOKz6qLl"
      },
      "source": [
        "## Exercise 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZvHk7PYtkTY6"
      },
      "source": [
        "What structures do you see in the embeddings of the words that were visualized earlier in this lab?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LMeBYRbEW2eB"
      },
      "source": [
        "### **Student Solution**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jUWVyvqq6vj7"
      },
      "source": [
        "*In the clusters of the words we see little circles that are of positive and negative connotation.*\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hCbn3FCcMf0I"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tI9DB-Dn76O-"
      },
      "source": [
        "## Exercise 2: Higher-Dimensional Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qhlN0BGnkVf3"
      },
      "source": [
        "Now we can visualize two-dimensional embeddings, but recall that our original vocabulary had over 80,000 words; we're missing a lot of information if we only use two dimensions!\n",
        "\n",
        "Typical embedding dimensions are 50, 100, 200, 300, and sometimes even larger.\n",
        "\n",
        "Retrain our earlier model with the same hyperparameters, using **50-dimensional** embeddings and retrieve the learned embeddings from this trained model. You should observe a slower training time and much higher training accuracy after 10 epochs.\n",
        "\n",
        "Print out the precision and recall on the testing holdout data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CU2jvjHHXE0t"
      },
      "source": [
        "### **Student Solution**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D5ab4EOm8aYW"
      },
      "source": [
        "\n",
        "\n",
        "embedding_dim = 50\n",
        "embedding_layer = keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "\n",
        "model = keras.Sequential([\n",
        "  keras.layers.Embedding(vocab_size, embedding_dim, input_length=maxlen),\n",
        "  keras.layers.Flatten(),\n",
        "  keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "model.summary()\n",
        "\n",
        "model.compile(\n",
        "  # Calculate loss for a binary classification problem.\n",
        "  loss='binary_crossentropy',\n",
        "\n",
        "  # Adam is one of the most commonly used optimizers.\n",
        "  optimizer=tf.keras.optimizers.Adam(),\n",
        "\n",
        "  # We will only track accuracy for this task.\n",
        "  metrics=['accuracy']\n",
        ")\n",
        "history = model.fit(\n",
        "    train_data,\n",
        "    train_labels,\n",
        "    epochs=10,\n",
        "    batch_size=512,\n",
        "    validation_split=0.2\n",
        ")\n",
        "\n",
        "precision = tf.keras.metrics.Precision()\n",
        "precision.update_state(test_labels, predictions)\n",
        "recall = tf.keras.metrics.Recall()\n",
        "recall.update_state(test_labels, predictions)\n",
        "print('Precision: {}, Recall: {}'.format(\n",
        "    precision.result().numpy(), recall.result().numpy()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V7EkkplFNQX3"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "heyIqwTPQffC"
      },
      "source": [
        "## Exercise 3: Embedding Projector"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q-2TJScF9DEM"
      },
      "source": [
        "In this section we'll introduce the [Embedding Projector](http://projector.tensorflow.org/), a tool to visualize high-dimensional embeddings.\n",
        "\n",
        "We will now write the embeddings to disk. To use the Embedding Projector, we will upload two files in tab separated format: a file of vectors (containing the embeddings) and a file of metadata (containing the words). We will again only analyze our small set of terms from above.\n",
        "\n",
        "Run the code block below to create the files necessary for the Embedding Projector."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ub8R-OiRz1PE"
      },
      "source": [
        "import io\n",
        "\n",
        "e = model.layers[0]\n",
        "embedding_matrix = e.get_weights()[0]\n",
        "\n",
        "out_v = io.open('vecs.tsv', 'w', encoding='utf-8')\n",
        "out_m = io.open('meta.tsv', 'w', encoding='utf-8')\n",
        "for word in informative_terms:\n",
        "  word_num = word_index[word]\n",
        "  embeddings = embedding_matrix[word_num]\n",
        "\n",
        "  out_m.write(word.encode(\"utf-8\").decode(\"utf-8\")  + \"\\n\")\n",
        "  out_v.write('\\t'.join(\n",
        "      [str(x).encode(\"utf-8\").decode(\"utf-8\") for x in embeddings]) + \"\\n\")\n",
        "\n",
        "out_v.close()\n",
        "out_m.close()\n",
        "\n",
        "try:\n",
        "  from google.colab import files\n",
        "except ImportError:\n",
        "  pass\n",
        "else:\n",
        "  files.download('vecs.tsv')\n",
        "  files.download('meta.tsv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UIhhw4L_93aO"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "Finally, open the [Embedding Projector](http://projector.tensorflow.org/):\n",
        "\n",
        "*   Click on \"Load\" in the Data panel.\n",
        "*   Upload the two files we created above: `vecs.tsv` and `meta.tsv`.\n",
        "\n",
        "The embeddings we have trained will now be displayed. You can search for words to find their closest neighbors.\n",
        "\n",
        "*Note: Your results may be a bit different, depending on how weights were randomly initialized before training the embedding layer.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "721P7udZRGSq"
      },
      "source": [
        "### Question 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ZHtRLBwka1x"
      },
      "source": [
        "How do the structures of these embeddings compare to the two-dimensional embeddings we visualized earlier?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYOIudwSRL6O"
      },
      "source": [
        "**Student Solution**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G0wJ755YROXi"
      },
      "source": [
        "> *The clusters seem much less circular in nature and take a more triangular prism shape. Seeing the data points in a 3 dimensional shape gives the appearance that the data points are much more closer than on the 2D graph.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZxYPkeOvRS9C"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nxwdOW0MR1JM"
      },
      "source": [
        "### Question 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NYit1JO0R2uH"
      },
      "source": [
        "List at least `5` words that seem to add little value to the sentiment analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rVSboMQQR8fK"
      },
      "source": [
        "**Student Solution**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a2xFw2EOR-qo"
      },
      "source": [
        "> 1. *film*\n",
        "> 1. *her*\n",
        "> 1. *him*\n",
        "> 1. *comedy*\n",
        "> 1. *movie*"
      ]
    }
  ]
}