Meetup10-DeepLearningTensorFlowinAzure-Part-3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">Deep learning with TensorFlow in Azure PART 3</h1>\n",
    "<h1 align=\"center\">Recurrent Neural Networks</h1>\n",
    "<h1 align=\"center\">Meetup DFW Data & AI - Microsoft</h1>\n",
    "## Setting Up Environment "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Deploy the new Linux Deep Learning VM in Azure "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to complete this notebook, you must deploy a Linux DLVM (Deep Learning VM) in your azure subscription. Click [HERE](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.dsvm-deep-learning?tab=Overview), then click on GET IT NOW and once you are directed to the Azure portal click CREATE.\n",
    "\n",
    "**In Basics blade:**<br>\n",
    "**Name:** meetupdsvmgpu <br>\n",
    "**Select OS:** Linux <br>\n",
    "**Username:** sshuser<br>\n",
    "**Password:** Passw0rd.1!!<br>\n",
    "**Resource Group:** meetupdsvmgpu_rg <br>\n",
    "**Location:** Pick among East US, North Central US, South Central US or West US 2<br>\n",
    "\n",
    "**In Settings blade:**\n",
    "Leave as is\n",
    "\n",
    "The Deep Learning Virtual Machine (DLVM) is a specially configured variant of the Data Science Virtual Machine(DSVM) to make it easier to use GPU-based VM instances for training deep learning models. It is supported on Windows 2016, or the Ubuntu Data Science Virtual Machine and shares the same core VM images (and hence all the rich toolset) as the DSVM. We also provide end-to-end AI samples for image and text understanding. The deep learning virtual machine also makes the rich set of tools and samples on the DSVM more easily discoverable. In terms of the tooling, the Deep Learning Virtual Machine provides several popular deep learning frameworks, tools to acquire and pre-process image, textual data. \n",
    "\n",
    "\n",
    "The DLVM contains several tools for AI including popular GPU editions of deep learning frameworks like Microsoft Cognitive Toolkit, TensorFlow, Keras, Caffe2, Chainer; tools to acquire and pre-process image, textual data, tools for data science modeling and development activities such as Microsoft R Server Developer Edition, Anaconda Python, Jupyter notebooks for Python and R, IDEs for Python and R , SQL database and many other data science and ML tools. \n",
    "\n",
    "\n",
    "The DLVM runs on Azure GPU NC-series VM instances. These GPUs use discrete device assignment, resulting in performance close to bare-metal, and are well-suited to deep learning problems.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) SSH into the VM and git clone the meetup repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "> ssh sshuser@YOUR.VM.IP.ADDRESS\n",
    "\n",
    "> cd notebooks\n",
    "\n",
    "> git clone https://github.com/pablomarin/Meetups-Data-AI-DFW.git\n",
    "\n",
    "> sudo ln -s /anaconda/envs/py35/bin/pip /usr/bin/pip3\n",
    "\n",
    "> sudo pip3 install tqdm\n",
    "\n",
    "> sudo pip3 install livelossplot\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Open the Jupyter notebook from your VM on your local browser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> https://YOUR.VM.IP.ADDRESS:8000 <br>\n",
    "> Login with your VM username and password<br>\n",
    "> Go to the ***Meetups-Data-AI-DFW folder***<br>\n",
    "> Open the Notebook:***Meetup10-DeepLearningTensorFlowinAzure-Part-3.ipynb***<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PART 3 - RNNs Recurrent Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Until now, we have studied the Foundations of Neural Networks (PART 1) and deep stacking of layers with Convolutional Networks (PART 2). We are now going complicate things a little bit more by studiying Networks of Neural Networks, this is the concept behind Recurrent Neural Networks (PART 3)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) What are Recurrent Neural Networks?\n",
    "Source: http://karpathy.github.io/2015/05/21/rnn-effectiveness/<br>\n",
    "**Depending on your background you might be wondering:** What makes Recurrent Networks so special? A glaring limitation of Vanilla Neural Networks (and also Convolutional Networks) is that their API is too constrained: they accept a fixed-sized vector as input (e.g. an image) and produce a fixed-sized vector as output (e.g. probabilities of different classes). Not only that: These models perform this mapping using a fixed amount of computational steps (e.g. the number of layers in the model). The core reason that recurrent nets are more exciting is that they allow us to operate over sequences of vectors: Sequences in the input, the output, or in the most general case both. A few examples may make this more concrete:\n",
    "\n",
    "<img src=\"http://karpathy.github.io/assets/rnn/diags.jpeg\" alt=\"IMAGE\" /><br>\n",
    " \n",
    "Each rectangle is a vector and arrows represent functions (e.g. matrix multiply). Input vectors are in red, output vectors are in blue and green vectors hold the RNN's state (more on this soon).<br> From left to right: (1) Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g. image classification). (2) Sequence output (e.g. image captioning takes an image and outputs a sentence of words). (3) Sequence input (e.g. sentiment analysis where a given sentence is classified as expressing positive or negative sentiment). (4) Sequence input and sequence output (e.g. Machine Translation: an RNN reads a sentence in English and then outputs a sentence in French). (5) Synced sequence input and output (e.g. video classification where we wish to label each frame of the video). Notice that in every case are no pre-specified constraints on the lengths sequences because the recurrent transformation (green) is fixed and can be applied as many times as we like.\n",
    "\n",
    "Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/<br>\n",
    "Recurrent neural networks are networks with loops in them, allowing information to persist.\n",
    "<div style=\"width:11.8%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img alt=\"\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-rolled.png\" data-disqus-identifier=\"/posts/2015-08-Understanding-LSTMs/disqussion-50\">\n",
    "</div>\n",
    "\n",
    "In the above diagram, a chunk of neural network, **A**, looks at some input **xt** and outputs a value **ht**. A loop allows information to be passed from one step of the network to the next.\n",
    "These loops make recurrent neural networks seem kind of mysterious. However, if you think a bit more, it turns out that they aren’t all that different than a normal neural network. A recurrent neural network can be thought of as multiple copies of the same network, each passing a message to a successor. Consider what happens if we unroll the loop:\n",
    "<div style=\"width:70%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img alt=\"An unrolled recurrent neural network.\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png\" data-disqus-identifier=\"/posts/2015-08-Understanding-LSTMs/disqussion-51\">\n",
    "</div>\n",
    "\n",
    "This chain-like nature reveals that recurrent neural networks are intimately related to sequences and lists. They’re the natural architecture of neural network to use for such data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 3 Concepts to understand RNNs:\n",
    "\n",
    "1. **LSTM cell - Long Short Term Memory cell**:<br>\n",
    "Source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/<br><br>\n",
    "Long Short Term Memory networks – usually just called “LSTMs” – are a special kind of RNN, capable of learning long-term dependencies. They were introduced by <a href=\"http://www.bioinf.jku.at/publications/older/2604.pdf\">Hochreiter &amp; Schmidhuber (1997)</a>, and were refined and popularized by many people in following work. They work tremendously well on a large variety of problems, and are now widely used.\n",
    "<br>LSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn!.<br><br>\n",
    "All recurrent neural networks have the form of a chain of repeating modules of neural network. In standard RNNs, this repeating module will have a very simple structure, such as a single tanh layer.<br>\n",
    "<div style=\"width:90%; margin-left:auto; margin-right:auto; margin-bottom:15px; margin-top:15px;\">\n",
    "<img alt=\"\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-SimpleRNN.png\">\n",
    "<div class=\"caption\" style=\"margin-bottom:10px;\">\n",
    "<strong>The repeating module in a standard RNN contains a single layer.</strong>\n",
    "</div>\n",
    "</div>\n",
    "LSTMs also have this chain like structure, but the repeating module has a different structure. Instead of having a single neural network layer, there are four, interacting in a very special way.\n",
    "<div style=\"width:90%; margin-left:auto; margin-right:auto; margin-bottom:15px; margin-top:15px;\">\n",
    "<img alt=\"A LSTM neural network.\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png\">\n",
    "<div class=\"caption\" style=\"margin-bottom:10px;\">\n",
    "<strong>The repeating module in an LSTM contains four interacting neural networks.</strong>\n",
    "</div>\n",
    "</div>\n",
    "Don’t worry about the details of what’s going on. We’ll walk through the LSTM diagram step by step later. For now, let’s just try to get comfortable with the notation we’ll be using.\n",
    "<div style=\"width:70%; margin-left:auto; margin-right:auto; margin-bottom:8px; margin-top:8px;\">\n",
    "<img alt=\"\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM2-notation.png\">\n",
    "</div>\n",
    "In the above diagram, each line carries an entire vector, from the output of one node to the inputs of others. The pink circles represent pointwise operations, like vector addition, while the yellow boxes are learned neural network layers. Lines merging denote concatenation, while a line forking denote its content being copied and the copies going to different locations. <br><br>\n",
    "\n",
    "2. **Batching**: <br>\n",
    "Source: https://www.quora.com/What-are-the-meanings-of-batch-size-mini-batch-iterations-and-epoch-in-neural-networks <br><br>\n",
    "Gradient descent is an iterative algorithm which computes the gradient of a function and uses it to update the parameters of the function in order to find a maximum or minimum value of the function. In case of Neural Networks, the function to be optimized (minimized) is the loss function, and the parameters are the weights and biases in the network.<br><br>\n",
    "Number of **iterations** (n): The number of times the gradient is estimated and the parameters of the neural network are updated using a batch of training instances. The **batch size** B is the number of training instances used in one iteration.\n",
    "When the total number of training instances (N) is large, a small number of training instances (B < < N) which constitute a **mini-batch** can be used in one iteration to estimate the gradient of the loss function and update the parameters of the neural network.\n",
    "<br>It takes n (=N/B) iterations to use the entire training data once. This constitutes an epoch. So, the total number of times the parameters get updated is (N/B) x E, where E is the number of epochs.\n",
    "<br><br>\n",
    "**Three modes of gradient descent**:\n",
    "<br>\n",
    "Batch mode: N=B, one epoch is same as one iteration.<br>\n",
    "Stochastic mode: B=1, one epoch takes N iterations.<br>\n",
    "Mini-batch mode: 1 < B < N, one epoch consists of N/B iterations.<br>\n",
    "<div style=\"width:70%; margin-left:auto; margin-right:auto; margin-bottom:8px; margin-top:8px;\">\n",
    "<img alt=\"\" src=\"https://visualstudiomagazine.com/articles/2015/07/01/~/media/ECG/visualstudiomagazine/Images/2015/07/0715vsm_McCaffreyFig2.ashx\n",
    "\">\n",
    "</div><br>\n",
    "3. **Word Embeddings**: <br>\n",
    "Source: Udacity<br><br>\n",
    "This is a deep neural network method for representing data with a huge number of classes more efficiently. Embeddings greatly improve the ability of networks to learn from data of this sort by representing the data with lower dimensional vectors.\n",
    "<br>Word embeddings in particular are interesting because the networks are able to learn semantic relationships between words. For example, the embeddings will know that the male equivalent of a queen is a king.\n",
    "<img class=\"index--image--1wh9w\" style=\"width: 560px;\" src=\"https://d17h27t6h515a5.cloudfront.net/topher/2017/March/58c0aaf4_linear-relationships/linear-relationships.png\">\n",
    "\n",
    "These word embeddings are learned using a model called Word2vec.\n",
    "Normally, for Language processing tasks,  the input to LSTM cells is the output of an Embedding layer. This Embedding layer is in charge of converting words to vectors of numbers. These vectors are the semanctic representation of the meaning of the words.\n",
    "\n",
    "Source: wikipedia<br>\n",
    "**Word2vec** is a group of related models that are used to produce [word embeddings](https://en.wikipedia.org/wiki/Word2vec). These models are shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words. Word2vec takes as its input a large corpus of text and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a corresponding vector in the space.<br><br>\n",
    "Word vectors are positioned in the vector space such that words that share common contexts in the corpus are located in close proximity to one another in the space.\n",
    "<div style=\"width:70%; margin-left:auto; margin-right:auto; margin-bottom:8px; margin-top:8px;\">\n",
    "<img alt=\"\" src=\"http://i1.wp.com/www.lifestyletrading101.com/wp-content/uploads/2017/03/word2-vec-diagram.png?resize=1218%2C523\n",
    "\">\n",
    "</div>\n",
    "After the Word2Vec NN converts words to their semantic vector representations, we can make calculations like:\n",
    "<br>\n",
    "**KING - MAN + WOMAN = QUEEN**\n",
    "<div style=\"width:40%; margin-left:auto; margin-right:auto; margin-bottom:8px; margin-top:8px;\">\n",
    "<img alt=\"\" src=\"http://www.lifestyletrading101.com/wp-content/uploads/2017/03/word2-vec-king-queen.png\">\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) How RNNs work?:\n",
    "Source: Udacity<br><br>\n",
    "1. Introduction:\n",
    "<a href=\"http://www.youtube.com/watch?feature=player_embedded&v=64HSG6HAfEI\" target=\"_blank\"><img src=\"http://img.youtube.com/vi/64HSG6HAfEI/0.jpg\" alt=\"IMAGE\" width=\"240\" height=\"180\" border=\"10\" /></a>\n",
    "2. LSTM Cells:\n",
    "<a href=\"http://www.youtube.com/watch?feature=player_embedded&v=RYbSHogZetc\" target=\"_blank\"><img src=\"http://img.youtube.com/vi/RYbSHogZetc/0.jpg\" alt=\"IMAGE\" width=\"240\" height=\"180\" border=\"10\" /></a>\n",
    "3. Applications of RNNs:\n",
    "<a href=\"http://www.youtube.com/watch?feature=player_embedded&v=tDJBDwriJYQ\" target=\"_blank\"><img src=\"http://img.youtube.com/vi/tDJBDwriJYQ/0.jpg\" alt=\"IMAGE\" width=\"240\" height=\"180\" border=\"10\" /></a>\n",
    "4. Sequence to Sequence Architecture:\n",
    "<a href=\"http://www.youtube.com/watch?feature=player_embedded&v=dkHdEAJnV_w\" target=\"_blank\"><img src=\"http://img.youtube.com/vi/dkHdEAJnV_w/0.jpg\" alt=\"IMAGE\" width=\"240\" height=\"180\" border=\"10\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is way more on RNNs that I won't cover on this Tutorial. I recommend the following Resources:\n",
    "1. Very easy way to understand RNNs - https://www.youtube.com/watch?v=UNmqTiOnRfg\n",
    "2. RNNs and LSTMs (Brandon Rohrer) -  https://www.youtube.com/watch?v=WCUNPb-5EYI\n",
    "3. Sequence to Sequence architectures - https://www.youtube.com/watch?v=RIR_-Xlbp7s\n",
    "4. RNNs with Attention - https://www.youtube.com/watch?v=QuvRWevJMZ4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) RNNs in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding three more API calls for using RNNs in TensorFlow:\n",
    "\n",
    "11. **LSTM cell**:<br>\n",
    "Here we will create the LSTM cell we'll use in the hidden layer. We'll use this cell as a building block for the RNN. So we aren't actually defining the RNN here, just the type of cell we'll use in the hidden layer.<br>\n",
    "We first create a basic LSTM cell with:\n",
    "```python\n",
    "lstm = tf.contrib.rnn.BasicLSTMCell(num_units)\n",
    "```\n",
    "where `num_units` is the number of units in the hidden layers in the cell.<br><br>\n",
    "12. **Dropout wrapper**:<br>\n",
    "You pass in a LSTM cell and it will automatically add dropout to the inputs or outputs\n",
    "```python\n",
    "tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
    "```\n",
    "13. **MultiRNNCell**:<br>\n",
    "Finally, we can stack up the LSTM cells into layers with [`tf.contrib.rnn.MultiRNNCell`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/contrib/rnn/MultiRNNCell). With this, you pass in a list of cells and it will send the output of one cell into the next cell. Previously with TensorFlow 1.0, you could do this\n",
    "```python\n",
    "tf.contrib.rnn.MultiRNNCell([cell]*num_layers)\n",
    "```\n",
    "This might look a little weird if you know Python well because this will create a list of the same `cell` object. However, TensorFlow 1.0 will create different weight matrices for all `cell` objects. But, starting with TensorFlow 1.1 you actually need to create new cell objects in the list. To get it to work in TensorFlow 1.1, it should look like:<br>\n",
    "```python\n",
    "def build_cell(num_units, keep_prob):\n",
    "    lstm = tf.contrib.rnn.BasicLSTMCell(num_units)\n",
    "    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
    "    return drop\n",
    "tf.contrib.rnn.MultiRNNCell([build_cell(num_units, keep_prob) for _ in range(num_layers)])\n",
    "```\n",
    "Even though this is actually multiple LSTM cells stacked on each other, you can treat the multiple layers as one cell.<br>\n",
    "We also need to create an initial cell state of all zeros. This can be done like so\n",
    "```python\n",
    "initial_state = cell.zero_state(batch_size, tf.float32)\n",
    "```\n",
    "Below, an implementation of the build_lstm function to create these LSTM cells and the initial state.<br><br>\n",
    "\n",
    "14. **Seq2Seq**:<br>\n",
    "Tensorflow has a bunch of APIs that help you build a sequence to sequence model (also called a seq2seq model for short). It’s important to note that these APIs have changed at the end of 2016 (and in a way are still evolving). A lot of the tutorials you’ll find on the web for seq2seq in tensorflow use the now-deprecated tf.contrib.legacy_seq2seq (which was previously called “tf.nn.seq2seq”).<br><br>\n",
    "The modules of note for seq2seq are:\n",
    "```python\n",
    "tf.nn, which allows us to construct different kinds of RNNs\n",
    "tf.contrib.rnn, which defines a number of RNN cells (an RNN cell is a required parameter for the RNNs defined in tf.nn).\n",
    "tf.contrib.seq2seq, which contains seq2seq decoders and loss operations.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_lstm(lstm_size, num_layers, batch_size, keep_prob):\n",
    "    ''' Build LSTM cell.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        keep_prob: Scalar tensor (tf.placeholder) for the dropout keep probability\n",
    "        lstm_size: Size of the hidden layers in the LSTM cells\n",
    "        num_layers: Number of LSTM layers\n",
    "        batch_size: Batch size\n",
    "\n",
    "    '''\n",
    "    ### Build the LSTM Cell\n",
    "    \n",
    "    def build_cell(lstm_size, keep_prob):\n",
    "        # Use a basic LSTM cell\n",
    "        lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)\n",
    "        \n",
    "        # Add dropout to the cell\n",
    "        drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
    "        return drop\n",
    "    \n",
    "    \n",
    "    # Stack up multiple LSTM layers, for deep learning\n",
    "    cell = tf.contrib.rnn.MultiRNNCell([build_cell(lstm_size, keep_prob) for _ in range(num_layers)])\n",
    "    initial_state = cell.zero_state(batch_size, tf.float32)\n",
    "    \n",
    "    return cell, initial_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, enough theory, let's run some code.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 4) LAB 1 - TV Script generation (NLP) using TF API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](Meetup10-Lab-TV-script-generation.ipynb) to open the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5) LAB 2 - Stock Market Prediction (Time Series Regression) using Keras API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](Meetup10-Lab-stockdemo.ipynb) to open the notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}