Meetup11-DeepLearningTensorFlowinAzure-Part-4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">Deep learning with TensorFlow in Azure PART 4</h1>\n",
    "<h1 align=\"center\">Distributed Training</h1>\n",
    "<h1 align=\"center\">Meetup DFW Data & AI - Microsoft</h1>\n",
    "## Setting Up Environment "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Deploy the Linux Deep Learning VM in Azure "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to complete this notebook, you must deploy a Linux DLVM (Deep Learning VM) in your azure subscription. Click [HERE](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.dsvm-deep-learning?tab=Overview), then click on GET IT NOW and once you are directed to the Azure portal click CREATE.\n",
    "\n",
    "**In Basics blade:**<br>\n",
    "**Name:** meetupdsvmgpu <br>\n",
    "**Select OS:** Linux <br>\n",
    "**Username:** sshuser<br>\n",
    "**Password:** Passw0rd.1!!<br>\n",
    "**Resource Group:** meetupdsvmgpu_rg <br>\n",
    "**Location:** Pick among East US or West US 2<br>\n",
    "\n",
    "**In Settings blade:**\n",
    "Leave as is\n",
    "\n",
    "The Deep Learning Virtual Machine (DLVM) is a specially configured variant of the Data Science Virtual Machine(DSVM) to make it easier to use GPU-based VM instances for training deep learning models. It is supported on Windows 2016, or the Ubuntu Data Science Virtual Machine and shares the same core VM images (and hence all the rich toolset) as the DSVM. We also provide end-to-end AI samples for image and text understanding. The deep learning virtual machine also makes the rich set of tools and samples on the DSVM more easily discoverable. In terms of the tooling, the Deep Learning Virtual Machine provides several popular deep learning frameworks, tools to acquire and pre-process image, textual data. \n",
    "\n",
    "\n",
    "The DLVM contains several tools for AI including popular GPU editions of deep learning frameworks like Microsoft Cognitive Toolkit, TensorFlow, Keras, Caffe2, Chainer; tools to acquire and pre-process image, textual data, tools for data science modeling and development activities such as Microsoft R Server Developer Edition, Anaconda Python, Jupyter notebooks for Python and R, IDEs for Python and R , SQL database and many other data science and ML tools. \n",
    "\n",
    "\n",
    "The DLVM runs on Azure GPU NC-series VM instances. These GPUs use discrete device assignment, resulting in performance close to bare-metal, and are well-suited to deep learning problems.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) SSH into the VM and git clone the meetup repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "> ssh sshuser@YOUR.VM.IP.ADDRESS\n",
    "\n",
    "> cd notebooks\n",
    "\n",
    "> git clone https://github.com/pablomarin/Meetups-Data-AI-DFW.git\n",
    "\n",
    "> sudo ln -s /anaconda/envs/py35/bin/pip /usr/bin/pip3\n",
    "\n",
    "> sudo pip3 install tqdm\n",
    "\n",
    "> sudo pip3 install ipython-autotime\n",
    "\n",
    "> az provider register -n Microsoft.BatchAI\n",
    "\n",
    "> az provider register -n Microsoft.Batch\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Open the Jupyter notebook from your VM on your local browser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> https://YOUR.VM.IP.ADDRESS:8000 <br>\n",
    "> Login with your VM username and password<br>\n",
    "> Go to the ***Meetups-Data-AI-DFW folder***<br>\n",
    "> Open the Notebook:***Meetup10-DeepLearningTensorFlowinAzure-Part-4.ipynb***<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PART 4 - Distributed TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Until now, we have studied the Foundations of Neural Networks (PART 1), deep stacking of layers with Convolutional Networks (PART 2), and recurrent Neural Networks for sequence prediction (PART 3). Now in PART 4, we are going to learn how to answer these two questions:\n",
    "* How do I train a model with big data without waiting days/weeks for results?\n",
    "* How do I train a model whose gradients won't fit on my GPU?\n",
    "* How do I try hundreds/thosands of combinations of hyperparameters to achieve the optimun NN tunning?\n",
    "\n",
    "When possible, I recommend training neural networks on a single machine; distributed training code is more complex than single-machine training and slower due to communication overhead. However, you should consider distributed training if your model or your data are too large to fit in memory on a single machine.\n",
    "\n",
    "Let's begin by explaining a few concepts:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) What is Data parallelism and Model parallelism?\n",
    "Source: http://timdettmers.com/2014/10/09/deep-learning-data-parallelism/<br>\n",
    "\n",
    "‘Data parallelism’ and ‘model parallelism’ are different ways of distributing an algorithm. These are often used in the context of machine learning algorithms that use stochastic gradient descent to learn some model parameters, which basically means that:\n",
    "\n",
    "    - The algorithm is trying to estimate some parameters from the given data.\n",
    "    - Parameters are estimated by minimizing the gradient against some loss function.\n",
    "    - Algorithm iterates over data in small batches.\n",
    "\n",
    "Data parallelism is when you use the same model for every thread, but feed it with different parts of the data;<br> \n",
    "Model parallelism is when you use the same data for every thread, but split the model among threads.\n",
    "\n",
    "**Data Parallelism:** in a distributed training setting is when each thread/core trains on the same model but over different data. After every few iterations, all threats synchronize, either with one-another (all-reduce) or via a central server (parameter server). This usually scales up nicely and one can also see some algorithmic speedup due to averaging. Data parallelism uses the same weights but different mini-batches in each thread; the gradients need to be synchronized, i.e. averaged, after each pass through a mini-batch.\n",
    "\n",
    "The idea of data parallelism is simple. If you have, say, 4 GPUs you split a mini-batch into parts for each of them, say, you split a mini-batch with 128 examples into 32 examples for each GPU. Then you feed the respective batch through the net and obtain gradients for each split of the mini-batch. You then use MPI to collect all the gradients and update the parameters with the overall average.\n",
    "\n",
    "<div style=\"width:60%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img src=\"https://i1.wp.com/timdettmers.com/wp-content/uploads/2014/10/datapara1.png?resize=1025%2C626\" alt=\"IMAGE\" /><br>\n",
    "</div>\n",
    "\n",
    "Data parallelism diagram. There is no communication in the forward pass, and during the backward pass you synchronize gradients.\n",
    "\n",
    "**Model Parallelism:** is when each thread/core trains over same data but uses different part of the model. This is trickier since (a) the model needs to be large enough (memory) to justify going over the network (b) the split needs to be careful enough such that the computation/communication is reasonable. Model parallelism splits the weights of the net equally among the threads and all threads work on a single mini-batch; here the generated output after each layer needs to be synchronized, i.e. stacked, to provide the input to the next layer; so each GPU works on a part of the model rather than a part of the data. In deep learning, one approach is to do this by splitting the weights, e.g. a 1000×1000 weight matrix would be split into a 1000×250 matrix if you use four GPUs\n",
    "\n",
    "<div style=\"width:60%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img src=\"https://i0.wp.com/timdettmers.com/wp-content/uploads/2014/11/modelpara1.png?resize=1025%2C626\" alt=\"IMAGE\" /><br>\n",
    "</div>\n",
    "Model parallelism diagram. Synchronizing communication is needed after each dot product with the weight matrix for both forward and backward pass.\n",
    "\n",
    "One advantage of this approach is immediately apparent: If we split the weights among the GPUs we can have very large neural networks which weights would not fit into the memory of a single GPU. Today for common problems in the industry such large neural networks are largely unnecessary. However, for very big unsupervised learning tasks – which will become quite important in the near future – such large networks will be needed in order to learn fine grained features that could learn “intelligent” behavior.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 2) What is Distributed Hyperparameter search?\n",
    "\n",
    "Source: https://docs.microsoft.com/en-us/azure/machine-learning/preview/scenario-distributed-tuning-of-hyperparameters<br><br>\n",
    "\n",
    "Many machine learning algorithms have one or more knobs, called hyperparameters. These knobs allow tuning of algorithms to optimize their performance over future data, measured according to user-specified metrics (for example, accuracy, AUC, RMSE). Data scientist needs to provide values of hyperparameters when building a model over training data and before seeing the future test data. How based on the known training data can we set up the values of hyperparameters so that the model has a good performance over the unknown test data? \n",
    "\n",
    "A popular technique for tuning hyperparameters is a grid search combined with cross-validation. Cross-validation is a technique that assesses how well a model, trained on a training set, predicts over the test set. Using this technique, we first divide the dataset into K folds and then train the algorithm K times in a round-robin fashion. We do this on all but one of the folds called the \"held-out fold\". We compute the average value of the metrics of K models over K held-out folds. This average value, called cross-validated performance estimate, depends on the values of hyperparameters used when creating K models. When tuning hyperparameters, we search through the space of candidate hyperparameter values to find the ones that optimize cross-validation performance estimate. Grid search is a common search technique. In grid search, the space of candidate values of multiple hyperparameters is a cross-product of sets of candidate values of individual hyperparameters.\n",
    "\n",
    "Grid search using cross-validation can be time-consuming. If an algorithm has five hyperparameters each with five candidate values, we use K=5 folds. We then complete a grid search by training 56=15625 models.\n",
    "\n",
    "Picking the right parameters leads to high performance, while bad parameters can lead to prolonged training and bad performance. In practice, machine learning practitioners rerun the same model multiple times with different hyperparameters in order to find the best set. This is a classical technique called hyperparameter tuning.\n",
    "\n",
    "When building a neural network, there are many important hyperparameters to choose carefully. For example:\n",
    "\n",
    "    - Number of neurons in each layer: Too few neurons will reduce the expression power of the network, but too many will substantially increase the running time and return noisy estimates.\n",
    "    - Learning rate: If it is too high, the neural network will only focus on the last few samples seen and disregard all the experience accumulated before. If it is too low, it will take too long to reach a good state.\n",
    "    - Numbers of layers\n",
    "    - Type of network topology\n",
    "\n",
    "The interesting thing here is that even though TensorFlow itself is not distributed, the hyperparameter tuning process is “embarrassingly parallel” and can be distributed using Spark or Batch AI in Azure for example. We can use Spark or Batch AI to broadcast the common elements such as data and model description, and then schedule the individual repetitive computations across a cluster of machines in a fault-tolerant manner.\n",
    "\n",
    "<div style=\"width:90%; margin-left:auto; margin-right:auto; margin-bottom:15px; margin-top:15px;\">\n",
    "<img alt=\"\" src=\"https://databricks.com/wp-content/uploads/2016/01/image04.png\">\n",
    "<div class=\"caption\" style=\"margin-bottom:10px;\">\n",
    "\n",
    "<div style=\"width:90%; margin-left:auto; margin-right:auto; margin-bottom:15px; margin-top:15px;\">\n",
    "<img alt=\"\" src=https://docs.microsoft.com/en-us/azure/machine-learning/preview/media/scenario-distributed-tuning-of-hyperparameters/architecture.png>\n",
    "<div class=\"caption\" style=\"margin-bottom:10px;\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) What tools exist out there to do Distributed TensorFlow?\n",
    "Source: Databricks<br><br>\n",
    "\n",
    "\n",
    "\n",
    "1. **Batch AI:**\n",
    "[Batch AI](https://docs.microsoft.com/en-us/azure/batch-ai/overview) is a managed service that enables data scientists and AI researchers to train AI and other machine learning models on clusters of Azure virtual machines, including VMs with GPU support. You describe the requirements of your job, where to find the inputs and store the outputs, and Batch AI handles the rest.<br>\n",
    "Batch AI provides resource management and job scheduling specialized for AI training and testing. Key capabilities include:\n",
    "    * Running long-running batch jobs, iterative experimentation, and interactive training \n",
    "    * Automatic or manual scaling of VM clusters using GPUs or CPUs \n",
    "    * Configuring SSH communication between VMs and for remote access \n",
    "    * Support for any Deep Learning or machine learning framework, with optimized configuration for popular toolkits such as Microsoft Cognitive Toolkit (CNTK), TensorFlow, and Chainer \n",
    "    * Priority-based job queue to share clusters and take advantage of low-priority VMs and reserved instances \n",
    "    * Flexible storage options including Azure Files and a managed NFS server \n",
    "    * Mounting remote file shares into the VM and optional container \n",
    "    * Providing job status and restarting in case of VM failures \n",
    "    * Access to output logs, stdout, stderr, and models, including streaming from Azure Storage \n",
    "    * Azure command-line interface (CLI), SDKs for Python, C#, and Java, monitoring in the Azure Portal, and integration with Microsoft AI tools \n",
    "<br><br>\n",
    "2. **Spark Deep Learning Pipelines:**\n",
    "[Deep Learning Pipelines](https://github.com/databricks/spark-deep-learning) is a high-level deep learning framework that facilitates common deep learning workflows via the Spark MLlib Pipelines API and scales out deep learning on big data using Spark. It is an open source project and employs the Apache 2.0 License.\n",
    "Deep Learning Pipelines is a high-level API that calls into lower-level deep learning libraries. It currently supports TensorFlow and Keras with the TensorFlow-backend.<br><br>\n",
    "\n",
    "3. **Microsoft Machine Learning for Apache Spark :**\n",
    "[MMLSpark](https://github.com/Azure/mmlspark) provides a number of deep learning and data science tools for Apache Spark, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK) and OpenCV, enabling you to quickly create powerful, highly-scalable predictive and analytical models for large image and text datasets.<br><br>\n",
    "\n",
    "4. **Horovod:**\n",
    "Horovod is an open-source Python framework for launching Tensorflow or Keras-based distributed deep learning workflows (see [the original Uber blog post](https://eng.uber.com/horovod/)). Notably, Horovod also works well for launching single machine, multi-GPU workflows.<br>\n",
    "Horovod simplifies deep learning training by:\n",
    "    - Supporting different parallelism modes (single-machine, single-machine-multi-GPU, multi-machine multi-GPU, etc.) with zero code changes\n",
    "    - Removing the need for separate workers and parameter servers (all model training processes run the same logic)<br>    \n",
    "The changes necessary to make single-GPU programs distributed:<br><br>\n",
    "    - **hvd.init()** initializes Horovod.\n",
    "    - **config.gpu_options.visible_device_list = str(hvd.local_rank())** assigns a GPU to each of the TensorFlow processes.\n",
    "    - **opt=hvd.DistributedOptimizer(opt)** wraps any regular TensorFlow optimizer with Horovod optimizer which takes care of averaging gradients using ring-allreduce.\n",
    "    - **hvd.BroadcastGlobalVariablesHook(0)** broadcasts variables from the first process to all other processes to ensure consistent initialization. If the program does not use **MonitoredTrainingSession**, users can run the **hvd.broadcast_global_variables(0)** operations instead.<br><br>\n",
    "\n",
    "5. **TensorFlowOnSpark:**\n",
    "TensorFlowOnSpark, developed by Yahoo, is an open-source Python framework for launching Tensorflow-based distributed deep learning workflows on Spark clusters ([see the original Yahoo blog post](http://yahoohadoop.tumblr.com/post/157196317141/open-sourcing-tensorflowonspark-distributed-deep)).\n",
    "TensorFlowOnSpark simplifies deep learning training on Spark clusters by:\n",
    "    - Abstracting away the need to manually specify a cluster configuration (mapping from CPUs/GPUs to Tensorflow tasks)\n",
    "    - Providing APIs for feeding data from Spark RDDs to Tensorflow programs\n",
    "    - Leveraging Spark’s built-in fault-tolerance (to recover from failures during TF training)\n",
    "<br><br>\n",
    "6. **dist-keras:**\n",
    "[dist-keras](https://github.com/cerndb/dist-keras) is an open-source framework for distributed training of Keras models (deep neural networks). It leverages Apache Spark to distribute and coordinate the training computation, and runs training directly on data in Spark DataFrames. dist-keras provides a built-in set of optimization strategies, such as Downpour and Dynamic SGD. To learn more about the available optimization strategies, see the dist-keras README.\n",
    "For single-machine training, see the Keras guide. For inference, we recommend that you use Deep Learning Pipelines, which leverages Spark to efficiently perform large-scale batch inference for Keras and TensorFlow models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, enough theory, let's run some code.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 4) LAB 1 - Batch AI for Multi-GPU Tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](Meetup11-Lab1-BatchAI-SingleNode-MultiGPU.ipynb) to open the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5) LAB 2 - Batch AI for Horovord"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](Meetup11-Lab2-BatchAI-MultiNode-Horovod.ipynb) to open the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6) LAB 3 - Batch AI for Keras with CNTK backend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](Meetup11-Lab3-BatchAI-Keras.ipynb) to open the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}